CN111224807B - Distributed log processing method, device, equipment and computer storage medium - Google Patents
Distributed log processing method, device, equipment and computer storage medium Download PDFInfo
- Publication number
- CN111224807B CN111224807B CN201811427214.1A CN201811427214A CN111224807B CN 111224807 B CN111224807 B CN 111224807B CN 201811427214 A CN201811427214 A CN 201811427214A CN 111224807 B CN111224807 B CN 111224807B
- Authority
- CN
- China
- Prior art keywords
- log
- service
- analyzed
- index data
- running
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention discloses a distributed log processing method, a device, equipment and a computer storage medium. The distributed log processing method comprises the following steps: acquiring the running log and the identification information of the running log from the application host through the message middleware; selecting a log to be analyzed from the running logs according to the identification information; and determining index data of the business processing process corresponding to the log to be analyzed according to the log to be analyzed. According to the embodiment of the invention, the real-time performance of data acquisition can be improved, the load of an application host can be reduced, and the index data of the business processing process can be monitored.
Description
Technical Field
The present invention relates to the field of information processing technologies, and in particular, to a method, an apparatus, a device, and a computer storage medium for processing a distributed log.
Background
Operation management for mobile service support systems has evolved over the years and forms a more sophisticated support platform, the china mobile service support network operation management system (Business Operation Management Center, BOMC). When the BOMC monitors each service application system, the process state and middleware queuing of each service application system are generally collected through a collection Agent (Agent), or the service index of each service application system is collected through accessing a database storing service information. However, this method can only monitor the key link performance of the service successfully handled by each service application system, and cannot monitor the service failed to be handled and determine the fault location.
In order to improve the operation and maintenance means of the service application system, the real-time operation condition of the service application system needs to be monitored, wherein one way is to collect and analyze the operation log of the service application system.
Typically, the travel log has the following characteristics:
1. the data size is large: the running log will record all internal interface calls, typically 10 orders of magnitude larger than the access log.
2. The distribution servers are as follows: the running logs of each service application system can be distributed in different servers and cannot be uniformly stored.
3. The running environment is complex: the service application system may fall into operation on a different application hosting environment.
In the prior art, the log information is generally read by directly capturing the running log, namely, after the service application system writes the log information into the running log and outputs the running log to a designated log catalog, the log catalog is monitored through an acquisition agent installed on each application host, the log information in the log catalog in a certain period is read at regular time, and then the log information is uniformly transmitted to a log server. Such an approach has the following problems:
1. the real-time performance is not good: the running log needs to be read by the acquisition agent and sent to the log server after a certain period.
2. Increasing application host load: the log information needs to be written into the running log and then read by the acquisition agent, so that the operations of writing and reading the log information can be increased, and the performance of the service application system can be influenced when the service application system is busy.
3. Increasing deployment complexity: when the application host environment is inconsistent or the log catalog is changed, the configuration of the collection agent and the catalog needs to be correspondingly adjusted.
4. The acquisition cannot be automatically expanded: when a newly extended application host exists, an acquisition agent needs to be manually installed on the newly extended application host, otherwise log information in the application host cannot be acquired.
Disclosure of Invention
The embodiment of the invention provides a distributed log processing method, a device, equipment and a computer storage medium, which can improve the real-time performance of data acquisition, reduce the load of an application host and monitor index data in a business processing process.
In one aspect, an embodiment of the present invention provides a distributed log processing method, including:
acquiring an operation log from an application host through a message middleware and identifying information of the operation log;
selecting a log to be analyzed from the running logs according to the identification information;
and determining index data of a business processing process corresponding to the log to be analyzed according to the log to be analyzed.
Further, before selecting the log to be analyzed from the running log according to the identification information, the method further includes:
and screening the running log conforming to the format information according to the preset format information.
Further, the method further comprises the following steps:
storing a running log which does not conform to the format information.
Further, the identification information includes a service ID and an interface ID, and selecting the log to be analyzed from the running log according to the identification information includes:
and selecting a log to be analyzed corresponding to the service ID or the interface ID from the running log according to the service ID or the interface ID.
Further, selecting a log to be analyzed from the running log according to the identification information further includes:
acquiring the relation data of the operation log and the acquisition time of the operation log;
and selecting a log to be analyzed from the running log according to the relation data and the identification information.
Further, according to the log to be analyzed, determining the index data of the business processing process corresponding to the log to be analyzed includes:
acquiring field information related to the business processing process in the log to be analyzed;
and determining the index data of the business processing process according to the field information.
Further, the index data at least comprises corresponding relation data of the service ID and the interface ID, service index data corresponding to the service processing process and service index data corresponding to the service processing process.
In another aspect, an embodiment of the present invention provides a distributed log processing apparatus, including:
the system comprises an acquisition unit, a message middleware and a control unit, wherein the acquisition unit is configured to acquire an operation log and identification information of the operation log from an application host through the message middleware;
the processing unit is configured to select a log to be analyzed from the running logs according to the identification information;
and the analysis unit is configured to determine index data of a business processing process corresponding to the log to be analyzed according to the log to be analyzed.
In still another aspect, an embodiment of the present invention provides a distributed log processing apparatus, including: a processor and a memory storing computer program instructions;
the processor, when executing the computer program instructions, implements a distributed log processing method as described above.
In yet another aspect, embodiments of the present invention provide a computer storage medium having stored thereon computer program instructions which, when executed by a processor, implement a distributed log processing method as described above.
According to the distributed log processing method, the distributed log processing device, the distributed log processing equipment and the computer storage medium, the running log can be obtained from the application host through the message middleware, so that the real-time performance of data acquisition is improved, and the load of the application host is reduced; meanwhile, the log to be analyzed can be selected from the running log, and then the index data of the business processing process corresponding to the log to be analyzed is determined according to the log to be analyzed, so that the business processing success business and the business processing failure business in the business processing process can be determined by monitoring the index data of the business processing process, the failure reason of the business processing failure business is determined according to the index data, and the fault position is determined.
Drawings
In order to more clearly illustrate the technical solution of the embodiments of the present invention, the drawings that are needed to be used in the embodiments of the present invention will be briefly described, and it is possible for a person skilled in the art to obtain other drawings according to these drawings without inventive effort.
FIG. 1 is a flow chart of a method for distributed log processing according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of an application side interface structure of a BOMC in an embodiment of the present invention;
FIG. 3 is a flow chart of a method for distributed log processing according to another embodiment of the present invention;
FIG. 4 is a detailed flow diagram of one example of step S120 of the distributed log processing method shown in FIG. 1;
FIG. 5 is a flowchart illustrating a step S130 of the distributed log processing method shown in FIG. 1;
FIG. 6 is a schematic diagram of a distributed log processing apparatus according to an embodiment of the present invention;
fig. 7 is a schematic hardware structure of a distributed log processing device according to an embodiment of the present invention.
Detailed Description
Features and exemplary embodiments of various aspects of the present invention will be described in detail below, and in order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in further detail below with reference to the accompanying drawings and the detailed embodiments. It should be understood that the specific embodiments described herein are merely configured to illustrate the invention and are not configured to limit the invention. It will be apparent to one skilled in the art that the present invention may be practiced without some of these specific details. The following description of the embodiments is merely intended to provide a better understanding of the invention by showing examples of the invention.
It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.
In order to solve the problems in the prior art, the embodiment of the invention provides a distributed log processing method, a device, equipment and a computer storage medium. The following first describes a distributed log processing method provided by an embodiment of the present invention.
Fig. 1 is a schematic flow chart of a distributed log processing method according to an embodiment of the present invention. As shown in fig. 1, the distributed log processing method includes:
s110, acquiring an operation log and identification information of the operation log from an application host through a message middleware;
s120, selecting a log to be analyzed from the running logs according to the identification information;
s130, determining index data of a business processing process corresponding to the log to be analyzed according to the log to be analyzed.
According to the embodiment of the invention, the running log can be obtained from the application host in real time through the message middleware, so that the real-time performance of data acquisition is improved, and the application host is not required to write in and read the running log, so that the load of the application host is reduced. Meanwhile, the embodiment of the invention can select the log to be analyzed from the running log, and then determine the index data of the business processing process corresponding to the log to be analyzed according to the log to be analyzed, so that the business processing success business and the business processing failure business in the business processing process can be determined by monitoring the index data of the business processing process, the failure reason of the business processing failure business is determined according to the index data, and the fault position is determined. In addition, in the embodiment of the invention, the problem of consistency of the application host environment and the problem of new extension application main bodies are not required to be considered, so that the deployment complexity can be reduced, and the monitoring intelligence and automation can be improved.
FIG. 2 is a schematic diagram of an application side interface of a BOMC according to an embodiment of the present invention. The embodiment of the invention can be applied to a method for processing distributed operation logs in the process of monitoring the operation state of an application host by using the BOMC with the application side interface structure shown in fig. 2. The application host 210 may be a terminal that performs mobile service processing using, for example, a Web client, an application program, or the like, and when the application host 210 performs mobile service processing, a running log may be generated, the terminal may call the client API interface 221 in the application interface structure 220, directly output the running log to the message middleware 230 of the BOMC through the proxy 222 in the application interface structure, and the message middleware 230 may format the running log, so that the BOMC obtains the formatted running log and identification information of the running log through the message middleware 230, so that the BOMC performs subsequent processing of the running log.
In the embodiment of the invention, the message middleware is preferably kafka message middleware, and the kafka message middleware has higher throughput and is special for log collection and transmission. However, any message middleware having a function of collecting and transmitting a running log may be applied to the embodiment of the present invention.
Fig. 3 is a schematic flow chart of a distributed log processing method according to another embodiment of the present invention. As shown in fig. 3, step S340 of the distributed log processing method, before selecting a log to be analyzed from the running log according to the identification information, further includes:
s320, screening the operation log conforming to the format information according to the preset format information.
In the embodiment of the invention, the running log transmitted by the message middleware can be received in real time through the store program, and the format verification is carried out on the received running log according to the preset format information, so that the running log which accords with the preset format information is screened out, and the subsequent analysis and processing are carried out.
After the running logs conforming to the format information are screened out, the running logs can be stored in the HBase database, so that data corresponding to the running logs can be searched and extracted when the running logs are analyzed and processed. The running logs can be stored in respective detail tables according to different log types. For example, the table name of the detail table may be a log type, and the detail table may store data corresponding to each running log and identification information corresponding to the running log.
As shown in fig. 3, the distributed log processing method further includes:
s330, storing a running log which does not accord with the format information.
Specifically, in the embodiment of the invention, the running log which does not accord with the format information can be saved as an abnormal log to the ES database, so that the abnormal log can be conveniently searched later, and the abnormal reason of the abnormal log is analyzed.
In an embodiment of the present invention, the identification information may include a service ID and an interface ID. In the service processing process, each step of calling a background interface through a foreground corresponds to the same interface ID, and each operation log has a unique interface ID so as to correlate the operation logs of the same service and facilitate the subsequent analysis and processing of the operation logs. In the service processing process, the same service is processed, and only the same service ID is provided, and each operation log also only has a unique service ID, so that the operation logs of the same service are associated, and the operation logs are convenient to analyze and process subsequently.
Therefore, in one example, in step S120, according to the identification information, a specific method for selecting the log to be analyzed from the running log may be:
and selecting a log to be analyzed corresponding to the service ID or the interface ID from the running log according to the service ID or the interface ID.
That is, in the embodiment of the present invention, a time interval for selecting a log to be analyzed corresponding to a service ID or an interface ID from a running log may be preset, for example, it may be set that the log to be analyzed is selected from the running log every minute, so that analysis processing is performed on the log to be analyzed subsequently.
Fig. 4 shows a specific flowchart of an example of step S120 of the distributed log processing method shown in fig. 1. In the example shown in fig. 4, in step S120, selecting the log to be analyzed from the running log according to the identification information may further include:
s121, acquiring relation data of the operation log and the acquisition time of the operation log;
s122, selecting a log to be analyzed from the running log according to the relation data and the identification information.
Specifically, the relationship data may be parsed in the log to be analyzed. The relationship data may specifically be a relationship between the acquisition time and the interface ID: < time, list < traceId >. The relationship data may be stored in a Redis database for retrieval for use in subsequent analysis processes.
When the preset time interval is reached, whether the mobile service is processed is determined according to the relation data, if so, all the running logs related to the mobile service are acquired according to the sequence of the acquisition time and the interface ID as the logs to be analyzed.
Fig. 5 is a specific flowchart of step S130 of the distributed log processing method shown in fig. 1. As shown in fig. 5, step S130, determining, according to the log to be analyzed, index data of a service processing procedure corresponding to the log to be analyzed includes:
s131, acquiring field information related to a business processing process in a log to be analyzed;
s132, determining index data of the service processing process according to the field information.
Firstly, the log to be analyzed is disassembled in step S131, the field information related to performance statistics in the single business processing process is analyzed, then, in step S132, the multi-line log information of the same call in the field information is combined into one call log information, so that index data related to performance statistics in the single business processing process is obtained, and finally, the index data can be sent to the corresponding cache area of the message middleware for determining the business handling success business and the business handling failure business in the business processing process and determining the failure reason of the business handling failure.
Specifically, in the embodiment of the present invention, the index data at least includes corresponding relation data between the service ID and the interface ID, service index data corresponding to the service processing procedure, and service index data corresponding to the service processing procedure.
The business index data comprises business handling success or failure handling results and time from the front stage to the back stage for completing the business. The service index data comprises background response time of different services and response results of success or failure of response in a business processing process.
According to the service index data and the service index data, the failure reason for handling the failure service can be determined, and then the failure position can be determined according to the failure reason through the corresponding relation data of the service ID and the interface ID.
In summary, in the embodiment of the invention, the running logs are directly obtained from the application host through the message middleware, so that the pressure of reading and writing the running logs when the application host generates a large number of running logs can be reduced, the acquisition process of the running logs in the cloud environment is simplified, the log acquisition agents are not required to be deployed in each application host, and the management cost is saved. Meanwhile, the embodiment of the invention adopts the storm program to process the running log, fully utilizes the data processing capability of the cluster environment, and can also distribute a large amount of logs to be analyzed which need to be analyzed and processed to a plurality of application hosts for processing respectively, namely a plurality of processes are distributed to process the logs to be analyzed, wherein each application host analyzes and processes all the logs to be analyzed corresponding to the same business processing process, thereby improving the efficiency of analysis and processing and being easy to expand the capability of data processing.
Fig. 6 is a schematic structural diagram of a distributed log processing apparatus according to an embodiment of the present invention. As shown in fig. 6, the distributed log processing apparatus includes:
an obtaining unit 410 configured to obtain the running log and the identification information of the running log from the application host through the message middleware;
the processing unit 420 is configured to select a log to be analyzed from the running logs according to the identification information;
and the parsing unit 430 is configured to determine index data of the business process corresponding to the log to be analyzed according to the log to be analyzed.
According to the embodiment of the invention, the running log can be obtained from the application host in real time through the message middleware, so that the real-time performance of data acquisition is improved, and the application host is not required to write in and read the running log, so that the load of the application host is reduced. Meanwhile, the embodiment of the invention can select the log to be analyzed from the running log, and then determine the index data of the business processing process corresponding to the log to be analyzed according to the log to be analyzed, so that the business processing success business and the business processing failure business in the business processing process can be determined by monitoring the index data of the business processing process, the failure reason of the business processing failure business is determined according to the index data, and the fault position is determined. In addition, in the embodiment of the invention, the problem of consistency of the application host environment and the problem of new extension application main bodies are not required to be considered, so that the deployment complexity can be reduced, and the monitoring intelligence and automation can be improved. In an embodiment of the present invention, the system may further include a cleaning unit configured to filter the running logs according with the format information according to the preset format information, so that the processing unit 420 selects the log to be analyzed from the running logs filtered by the cleaning unit.
In one example of an embodiment of the present invention, the processing unit 420 may be further configured to select a log to be analyzed corresponding to the service ID or the interface ID from the running log according to the service ID or the interface ID.
In another example of the embodiment of the present invention, the processing unit 420 may be further configured to obtain the relationship data between the running log and the obtaining time of the running log, and select the log to be analyzed from the running log according to the relationship data and the identification information.
In the embodiment of the invention, the system further comprises a storage unit, and the storage unit can be formed by different databases, such as an ES database, an HBase database and a Redis database. The HBase database is used for storing the running logs into respective detail tables according to different log types, the ES database is used for storing the running logs which do not accord with the format information, and the Redis database is used for storing the relation data.
In the embodiment of the present invention, the parsing unit 430 may be further configured to obtain field information related to a service processing procedure in the log to be analyzed; and determining index data of the service processing process according to the field information.
Specifically, the index data may at least include corresponding relation data of the service ID and the interface ID, service index data corresponding to the service processing procedure, and service index data corresponding to the service processing procedure.
Fig. 7 is a schematic diagram of a hardware structure of a distributed log processing device according to an embodiment of the present invention.
A distributed log processing device may include a processor 501 and a memory 502 storing computer program instructions.
In particular, the processor 501 may include a Central Processing Unit (CPU), or an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), or may be configured as one or more integrated circuits that implement embodiments of the present invention.
Memory 502 may include mass storage for data or instructions. By way of example, and not limitation, memory 502 may comprise a Hard Disk Drive (HDD), floppy Disk Drive, flash memory, optical Disk, magneto-optical Disk, magnetic tape, or universal serial bus (Universal Serial Bus, USB) Drive, or a combination of two or more of the foregoing. Memory 502 may include removable or non-removable (or fixed) media, where appropriate. Memory 502 may be internal or external to the integrated gateway disaster recovery device, where appropriate. In a particular embodiment, the memory 502 is a non-volatile solid state memory. In a particular embodiment, the memory 502 includes Read Only Memory (ROM). The ROM may be mask programmed ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically Erasable PROM (EEPROM), electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these, where appropriate.
The processor 501 implements any of the distributed log processing methods of the above embodiments by reading and executing computer program instructions stored in the memory 502.
In one example, the distributed log processing device may also include a communication interface 503 and a bus 510. As shown in fig. 7, the processor 501, the memory 502, and the communication interface 503 are connected to each other via a bus 510 and perform communication with each other.
The communication interface 503 is mainly used to implement communication between each module, apparatus, unit and/or device in the embodiments of the present invention.
Bus 510 includes hardware, software, or both that couple the components of the online data flow billing device to each other. By way of example, and not limitation, the buses may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an infiniband interconnect, a Low Pin Count (LPC) bus, a memory bus, a micro channel architecture (MCa) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a video electronics standards association local (VLB) bus, or other suitable bus, or a combination of two or more of the above. Bus 510 may include one or more buses, where appropriate. Although embodiments of the invention have been described and illustrated with respect to a particular bus, the invention contemplates any suitable bus or interconnect.
The distributed log processing equipment can acquire the running log from the application host in real time based on the message middleware and analyze and process the running log, so that the distributed log processing method and the device are realized.
In addition, in combination with the distributed log processing method in the above embodiment, the embodiment of the present invention may be implemented by providing a computer storage medium. The computer storage medium has stored thereon computer program instructions; the computer program instructions, when executed by a processor, implement any of the distributed log processing methods of the above embodiments.
It should be understood that the invention is not limited to the particular arrangements and instrumentality described above and shown in the drawings. For the sake of brevity, a detailed description of known methods is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method processes of the present invention are not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between steps, after appreciating the spirit of the present invention.
The functional blocks shown in the above-described structural block diagrams may be implemented in hardware, software, firmware, or a combination thereof. When implemented in hardware, it may be, for example, an electronic circuit, an Application Specific Integrated Circuit (ASIC), suitable firmware, a plug-in, a function card, or the like. When implemented in software, the elements of the invention are the programs or code segments used to perform the required tasks. The program or code segments may be stored in a machine readable medium or transmitted over transmission media or communication links by a data signal carried in a carrier wave. A "machine-readable medium" may include any medium that can store or transfer information. Examples of machine-readable media include electronic circuitry, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio Frequency (RF) links, and the like. The code segments may be downloaded via computer networks such as the internet, intranets, etc.
It should also be noted that the exemplary embodiments mentioned in this disclosure describe some methods or systems based on a series of steps or devices. However, the present invention is not limited to the order of the above-described steps, that is, the steps may be performed in the order mentioned in the embodiments, or may be performed in a different order from the order in the embodiments, or several steps may be performed simultaneously.
In the foregoing, only the specific embodiments of the present invention are described, and it will be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the systems, modules and units described above may refer to the corresponding processes in the foregoing method embodiments, which are not repeated herein. It should be understood that the scope of the present invention is not limited thereto, and any equivalent modifications or substitutions can be easily made by those skilled in the art within the technical scope of the present invention, and they should be included in the scope of the present invention.
Claims (8)
1. A distributed log processing method, comprising:
acquiring an operation log from an application host through a message middleware and identifying information of the operation log;
selecting a log to be analyzed from the running logs according to the identification information;
determining index data of a business processing process corresponding to the log to be analyzed according to the log to be analyzed;
the selecting a log to be analyzed from the running log according to the identification information further comprises:
acquiring the relation data of the operation log and the acquisition time of the operation log;
selecting a log to be analyzed from the running logs according to the relation data and the identification information;
the relation data specifically refers to the relation between the acquisition time and the interface ID;
and selecting a log to be analyzed from the running logs according to the relation data and the identification information, wherein the log to be analyzed comprises the following steps:
when a preset time interval is reached, determining whether a service with the processed end exists according to the relation data;
under the condition that the target service with the processing end is determined, selecting an operation log related to the target service from the operation logs according to the sequence of the interface ID and the acquisition time as the log to be analyzed;
the index data at least comprises corresponding relation data of a service ID and an interface ID, service index data corresponding to the service processing process and service index data corresponding to the service processing process;
the business index data comprises business handling results of successful or failed business handling and time from a foreground to a background to finish the business; the service index data comprises background response time of different services and response results of success or failure of response in a service processing process;
the service index data and the service index data are used for determining failure reasons for handling failure services; and the corresponding relation data of the service ID and the interface ID is used for determining the fault position according to the failure reason.
2. The distributed log processing method according to claim 1, wherein before selecting a log to be analyzed from the running logs according to the identification information, the method further comprises:
and screening the running log conforming to the format information according to the preset format information.
3. The distributed log processing method of claim 2, further comprising:
storing a running log which does not conform to the format information.
4. The distributed log processing method according to claim 1, wherein the identification information includes a service ID and an interface ID, and selecting a log to be analyzed from the running log according to the identification information includes:
and selecting a log to be analyzed corresponding to the service ID or the interface ID from the running log according to the service ID or the interface ID.
5. The distributed log processing method according to claim 1, wherein determining, according to the log to be analyzed, index data of a business process corresponding to the log to be analyzed includes:
acquiring field information related to the business processing process in the log to be analyzed;
and determining the index data of the business processing process according to the field information.
6. A distributed log processing apparatus, the apparatus comprising:
the system comprises an acquisition unit, a message middleware and a control unit, wherein the acquisition unit is configured to acquire an operation log and identification information of the operation log from an application host through the message middleware;
the processing unit is configured to select a log to be analyzed from the running logs according to the identification information;
the analysis unit is configured to determine index data of a business processing process corresponding to the log to be analyzed according to the log to be analyzed;
the selecting a log to be analyzed from the running log according to the identification information further comprises:
acquiring the relation data of the operation log and the acquisition time of the operation log;
selecting a log to be analyzed from the running logs according to the relation data and the identification information;
the relation data specifically refers to the relation between the acquisition time and the interface ID;
and selecting a log to be analyzed from the running logs according to the relation data and the identification information, wherein the log to be analyzed comprises the following steps:
when a preset time interval is reached, determining whether a service with the processed end exists according to the relation data;
under the condition that the target service with the processing end is determined, selecting an operation log related to the target service from the operation logs according to the sequence of the interface ID and the acquisition time as the log to be analyzed;
the index data at least comprises corresponding relation data of a service ID and an interface ID, service index data corresponding to the service processing process and service index data corresponding to the service processing process;
the business index data comprises business handling results of successful or failed business handling and time from a foreground to a background to finish the business; the service index data comprises background response time of different services and response results of success or failure of response in a service processing process;
the service index data and the service index data are used for determining failure reasons for handling failure services; and the corresponding relation data of the service ID and the interface ID is used for determining the fault position according to the failure reason.
7. A distributed log processing apparatus, the apparatus comprising: a processor and a memory storing computer program instructions;
the processor, when executing the computer program instructions, implements a distributed log processing method as claimed in any one of claims 1 to 5.
8. A computer storage medium having stored thereon computer program instructions which when executed by a processor implement the distributed log processing method of any of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811427214.1A CN111224807B (en) | 2018-11-27 | 2018-11-27 | Distributed log processing method, device, equipment and computer storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811427214.1A CN111224807B (en) | 2018-11-27 | 2018-11-27 | Distributed log processing method, device, equipment and computer storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111224807A CN111224807A (en) | 2020-06-02 |
| CN111224807B true CN111224807B (en) | 2023-08-01 |
Family
ID=70828830
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811427214.1A Active CN111224807B (en) | 2018-11-27 | 2018-11-27 | Distributed log processing method, device, equipment and computer storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111224807B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111639138B (en) * | 2020-06-03 | 2023-04-25 | 中国联合网络通信集团有限公司 | Data processing method, device, equipment and storage medium |
| CN111897834A (en) * | 2020-08-12 | 2020-11-06 | 网易（杭州）网络有限公司 | Log searching method and device and server |
| CN112416887B (en) * | 2020-11-18 | 2024-01-30 | 脸萌有限公司 | Information interaction method and device and electronic equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101938373A (en) * | 2010-08-16 | 2011-01-05 | 北京全路通信信号研究设计院 | Analysis and treatment method of RBC (Recording Buffer Controller) log record and RBC record analyzer |
| CN102385549A (en) * | 2010-09-02 | 2012-03-21 | 北京无限立通通讯技术有限责任公司 | Log processing system, log processing method and log storage sub-system |
| CN103023693A (en) * | 2012-11-27 | 2013-04-03 | 北京小米科技有限责任公司 | Behaviour log data management system and behaviour log data management method |
| CN103178982A (en) * | 2011-12-23 | 2013-06-26 | 阿里巴巴集团控股有限公司 | Method and device for analyzing log |
| KR20180065400A (en) * | 2016-12-07 | 2018-06-18 | 동국대학교 산학협력단 | System and method for analyzing log data |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9965624B2 (en) * | 2014-02-12 | 2018-05-08 | Mitsubishi Electric Corporation | Log analysis device, unauthorized access auditing system, computer readable medium storing log analysis program, and log analysis method |
-
2018
- 2018-11-27 CN CN201811427214.1A patent/CN111224807B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101938373A (en) * | 2010-08-16 | 2011-01-05 | 北京全路通信信号研究设计院 | Analysis and treatment method of RBC (Recording Buffer Controller) log record and RBC record analyzer |
| CN102385549A (en) * | 2010-09-02 | 2012-03-21 | 北京无限立通通讯技术有限责任公司 | Log processing system, log processing method and log storage sub-system |
| CN103178982A (en) * | 2011-12-23 | 2013-06-26 | 阿里巴巴集团控股有限公司 | Method and device for analyzing log |
| CN103023693A (en) * | 2012-11-27 | 2013-04-03 | 北京小米科技有限责任公司 | Behaviour log data management system and behaviour log data management method |
| KR20180065400A (en) * | 2016-12-07 | 2018-06-18 | 동국대학교 산학협력단 | System and method for analyzing log data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111224807A (en) | 2020-06-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111224807B (en) | Distributed log processing method, device, equipment and computer storage medium | |
| CN112422344A (en) | Log abnormity warning method and device, storage medium and electronic device | |
| CN109995555B (en) | Monitoring method, device, equipment and medium | |
| CN113918376B (en) | Fault detection method, device, equipment and computer readable storage medium | |
| CN106874135B (en) | Method, device and equipment for detecting machine room fault | |
| CN114077525A (en) | Abnormal log processing method and device, terminal equipment, cloud server and system | |
| CN110083575A (en) | Fulfilling monitoring method, device, equipment and computer readable storage medium | |
| CN113225339B (en) | Network security monitoring method and device, computer equipment and storage medium | |
| CN109409948B (en) | Transaction abnormity detection method, device, equipment and computer readable storage medium | |
| CN116719750B (en) | Software testing method and device, server equipment and storage medium | |
| CN110609761B (en) | Method and device for determining fault source, storage medium and electronic equipment | |
| CN111385157B (en) | Server abnormity detection method and device | |
| CN111240923A (en) | Automatic test method and device for recurring problems of vehicle navigation system and storage medium | |
| CN111654405A (en) | Method, device, equipment and storage medium for fault node of communication link | |
| CN107342917B (en) | Method and apparatus for detecting network device performance | |
| CN116302989A (en) | Pressure testing method and system, storage medium and computer equipment | |
| CN112269879B (en) | Method and equipment for analyzing middle station log based on k-means algorithm | |
| US20230004478A1 (en) | Systems and methods of continuous stack trace collection to monitor an application on a server and resolve an application incident | |
| CN116415045A (en) | Data acquisition method and device, electronic equipment and storage medium | |
| CN113238911A (en) | Alarm processing method and device | |
| CN113014675A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN110430093B (en) | Data processing method and device and computer readable storage medium | |
| CN113260045B (en) | Method, device, equipment and storage medium for determining geographic position of router | |
| CN116074439A (en) | Fault monitoring method, device, equipment and computer storage medium | |
| CN116028362A (en) | Data acquisition method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |