CN118012832A - Log processing method and device, electronic equipment and readable storage medium - Google Patents

Log processing method and device, electronic equipment and readable storage medium Download PDF

Info

Publication number
CN118012832A
CN118012832A CN202410317392.8A CN202410317392A CN118012832A CN 118012832 A CN118012832 A CN 118012832A CN 202410317392 A CN202410317392 A CN 202410317392A CN 118012832 A CN118012832 A CN 118012832A
Authority
CN
China
Prior art keywords
log
data
log data
call
target
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202410317392.8A
Other languages
Chinese (zh)
Inventor
邢统坤
王荻
刘晓辉
陈福荣
李婷
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China Travelsky Technology Co Ltd
Original Assignee
China Travelsky Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China Travelsky Technology Co Ltd filed Critical China Travelsky Technology Co Ltd
Priority to CN202410317392.8A priority Critical patent/CN118012832A/en
Publication of CN118012832A publication Critical patent/CN118012832A/en
Pending legal-status Critical Current

Links

Landscapes

  • Debugging And Monitoring (AREA)

Abstract

The invention relates to a log processing method, a device, an electronic device and a readable storage medium, which can combine airport equipment and a background system through a trace band log mark, aggregate log data generated by the system according to the trace band log mark to obtain call link data of each trace band, and realize clear link drawing of interface call relations among various applications of a departure system, so that maintainers can clearly know call links where the generated log data is located, and realize full link tracking of the log data from airport front-end equipment to the background system. The method is convenient for maintenance personnel to quickly locate the call link where the problem log data is located, and can assist in realizing automatic test of service call while reducing problem locating difficulty.

Description

Log processing method and device, electronic equipment and readable storage medium
Technical Field
The present invention relates to the field of data monitoring technologies, and in particular, to a log processing method, a log processing device, an electronic device, and a readable storage medium.
Background
In the process of opening the departure system, as the scale of the production business software increases and the flow increases, the log flow of the system is higher and higher, and the related content comprises the whole process from airport front-end equipment to the background. Therefore, maintenance personnel can not realize full-link tracking from front-end equipment to the background of the airport in the face of the condition that the log data are not available when the airport system problem is examined, and the difficulty of the airport system problem examination and tracking is increased.
Disclosure of Invention
This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
The invention provides a log processing method, a log processing device, electronic equipment and a readable storage medium, which can realize the tracking of a link for generating log data and effectively reduce the difficulty of checking and tracking problems of an airport system.
In order to achieve the above purpose, the present invention provides the following technical solutions:
according to the embodiment of the invention, a log processing method comprises the following steps:
Acquiring original log data of each tracking band based on the message queue;
Screening target log data containing corresponding tracking band log identification marks from each original log data;
Aggregating each target log data based on a tracking band log identification mark of each target log data to obtain calling link data;
And after correlating the obtained call link data with the corresponding tracking band log identification, storing the call link data into a preset relational database.
According to the technical scheme, the log processing method is disclosed, the airport equipment and the background system can be combined through the trace band log mark, the log data generated by the system are aggregated according to the trace band log mark, call link data of each trace band are obtained, clear link drawing is made on interface call relations among various applications of the departure system, so that maintenance personnel can clearly know call links where the generated log data are located, and full link tracking of the log data from the airport front-end equipment to the background system is realized. The method is convenient for maintenance personnel to quickly locate the call link where the problem log data is located, and can assist in realizing automatic test of service call while reducing problem locating difficulty.
Drawings
The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent by reference to the following detailed description when taken in conjunction with the accompanying drawings. The same or similar reference numbers will be used throughout the drawings to refer to the same or like elements. It should be understood that the figures are schematic and that elements and components are not necessarily drawn to scale.
FIG. 1 is a flowchart of a log processing method according to an embodiment of the present invention;
FIG. 2 is a block diagram of a log processing device according to an embodiment of the present invention;
fig. 3 is a block diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure have been shown in the accompanying drawings, it is to be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustration purposes only and are not intended to limit the scope of the present disclosure.
The term "including" and variations thereof as used herein are intended to be open-ended, i.e., including, but not limited to. The term "based on" is based at least in part on. The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments. Related definitions of other terms will be given in the description below.
It should be noted that the terms "first," "second," and the like in this disclosure are merely used to distinguish between different devices, modules, or units and are not used to define an order or interdependence of functions performed by the devices, modules, or units.
It should be noted that references to "one", "a plurality" and "a plurality" in this disclosure are intended to be illustrative rather than limiting, and those of ordinary skill in the art will appreciate that "one or more" is intended to be understood as "one or more" unless the context clearly indicates otherwise.
Referring to fig. 1, the present invention provides a log processing method, which may include the steps of:
101. the original log data of each tracking band is obtained based on the message queue.
Specifically, the tracking band is a link formed from the airport front-end equipment to the airport background system for tracking each departure service. The collected raw log data may be transferred to a Kafka queue (a high throughput distributed publish-subscribe messaging system) as a data source.
102. And screening target log data containing corresponding tracking band log identification marks from the original log data.
Specifically, all open software trace bands received may be filtered according to log identification (globalid) of the departure trace band, only trace band log data containing globalid is retained, where the filtered log data is stored in HDFS (a distributed file system) for problem investigation.
103. And aggregating each target log data based on the tracking zone log identification mark of each target log data to obtain calling link data.
Specifically, software contained in the call link data may be determined by first determining interaction log data of a request type input in each target log data. And then determining the calling sequence relation among the software according to the response time of the response log data output in each target log data. The format of the tracking tape log data obtained is as follows, for example:
The message header contains information such as a software code, a server name, a transaction ID and the like, the extension area contains identification information (PS: different Protocol type identification calling party/called party identification method and depending fields, in the example, protocol=5) such as a departure transaction global log identification mark (Globalid), an interface name (Action), a calling party (OriAppCode), a called party (DesAppCode) and the like, and the representation of the loss of the calling party information is called from an avionic external system (recorded as exesys).
The application server is used as a peripheral system of a large host system and comprises all input and output of the host system. According to the division of the call in the system and the call out of the system, the log is divided into DAUDIT types and STAT types, wherein DAUDIT types represent the call of the module in the system, and STAT types represent the interaction of the system with other systems. Each type of log includes IN and OUT directions, and meanwhile, the log is divided into two types of request and response, which are represented by MsgType (the value of which is represented by 1: synchronous request message, 2: asynchronous request message, 3: synchronous response message, 5: asynchronous response message, 6: no response message), since the inter-deployment call is also recorded as STAT type, only stat_in log of MsgType type 1/2/reuqest can be obtained when analysis is performed, then response time (timecost field value) IN stat_out type log is utilized to assist IN determining the sequence of the request call, because timecost represents the total time from the request to the last to the return, the longest response time should be the software that initiates the request first, and so on, and the interface of the calling party and the called party software information and the called party IN each stat_in log is added, so that a similar "software 1:interface 1-software 2:2-interface 3" of the log can be obtained according to the known information:
if the raw data obtained is (interface name is indicated in brackets):
exesys- > A (A1); a- > B (B1), A- > B (B2); B-C (C1), the record of the interface call relationship is:
it will be appreciated that the person skilled in the art may adjust the formation of the call links according to the needs of the actual application, and is not limited herein.
104. And after correlating the obtained call link data with the corresponding tracking band log identification, storing the call link data into a preset relational database.
Specifically, on the basis of the HDFS data stored after being aggregated according to globalid, each globalid is subjected to a call link series analysis, and the analyzed link data is stored in an EDB (enterprise-level database) database base table for later manual confirmation or automatic batch confirmation. Two major functions can be provided on the query interface, one is a business monitoring class and the other is a departure interface calling relation class. The service monitoring class page provides pages such as a monitoring center, a query center, software access quantity, compliance query, software non-compliance log quantity statistics and the like. The departure interface call relationship class page provides departure interface call relationship (design), departure interface call relationship (actual), departure interface call relationship (comparison), departure interface management, departure interface call relationship graph confirmation and management, non-compliance data statistics, child node missing statistics and mail notification functions and the like. The log processing method solves the defect that the full-link tracking, automatic testing and monitoring from airport front-end equipment to the background cannot be realized in the departure field. And the airport equipment and the background system can be combined through a global log identification mark to inquire the log of the full-link tracking zone, so that the difficulty of the problem investigation and tracking of the airport system is reduced.
In another embodiment of the present invention, in order to improve the reliability of the call link determination, after target log data including the corresponding trace band log identification mark is screened from each original log data, compliance verification is performed on each target log data, and the target log data that is not verified is stored in the distributed file system for problem investigation.
Specifically, after the basic compliance verification is performed on the log, the following 4 logs can be filtered out:
1) globalid is incorrectly formatted.
2) Lack of source or target.
3) Transaction type (MsgType) protocol= 1/2/3/4/5 is absent.
4) The Protocol is absent.
The filtered logs are stored to the HDFS for troubleshooting.
Storing the checked target log data into a distributed database, and aggregating the target log data in the distributed database at intervals of a first preset duration. And storing the aggregated target log data into a distributed file system based on the tracking band log identification mark.
Specifically, log data stored in hbase (a distributed database) is aggregated once every 10 minutes according to globalid by using spark-streaming technology, the aggregated data is stored in HDFS, and different globalid are stored in the HDFS in a file according to a hash table. And processing the data stored in the HDFS, so that the query center can query the log according to globaid and provide the interface calling relation of globalid. And providing the interface form for an automatic test call to realize the automatic test target of opening all software in the departure port. The automatic test is based on the fact that a design interface call relation diagram is provided, meanwhile, an actual interface call relation diagram is provided, then the design and the actual call relation diagram are compared to obtain whether the actual call relation diagram accords with the design, the test is considered to pass if the actual call relation diagram accords with the design, and the test is considered to not pass if the actual call relation diagram does not accord with the design. For example, the following 9 interfaces may be provided:
interface one: the interface list of all software is queried.
Interface II: query the link list of the interface (support batch queries).
And an interface III: and inquiring the link verification result.
And an interface IV: the page links of the link map are queried.
Interface five: and returning global design calling relation data of the appointed interface.
And the interface six provides a query interface for actually calling the relation of two pairs. (by using spark-streaming technology to perform offline link data analysis on all logs of the previous day, forming individual call relations every two, storing the call relations into an EDB database for the call relations of the departure port interface to compare with the page query call, and forming an actual call link and a design call link comparison diagram)
Interface seven: a comparison query page link is provided.
Interface eight: and newly making a page to realize the effect of the departure interface calling relationship compared with the actual interface calling relationship in the page.
Interface nine: providing coverage for automated test inquiry smoke test.
The implementation of the function of automatically confirming the call relation of the design interface in batch is to perform link series analysis on each globalid based on the stored HDFS data after being aggregated according to globalid, store the analyzed link data into the base table of the EDB database, and simultaneously perform a synchronization operation on the link data in the base table at each whole point, where the synchronization operation can take all globalid link map data generated in the previous hour to check whether globalid of the same call link data exists in the EDB database to be confirmed, if yes, synchronization to the database to be confirmed is not needed, and if no, the globalid link map data needs to be synchronized to the database to be confirmed for later manual confirmation or automatic batch confirmation.
Of course, instead of using spark-streaming, flink may be used to process, flink can actually achieve the effect of real-time streaming, while spark-streaming is just to simulate real-time streaming, and in the future, when the log amount reaches a certain amount, the analysis data performance may be considered to be processed instead of flink.
In some embodiments of the present invention, globalid is a global service ID recorded when initiated from airport front-end software, where more transactions are involved, and then other software generates a different transaction ID according to different transactions.
Although operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order. In certain circumstances, multitasking and parallel processing may be advantageous.
It should be understood that the various steps recited in the method embodiments of the present disclosure may be performed in a different order, or in parallel. Furthermore, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the present disclosure is not limited in this respect.
Computer program code for carrying out operations of the present disclosure may be written in one or more programming languages, including, but not limited to, an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).
Referring to fig. 2, an embodiment of the present invention further provides a log processing apparatus, where the log processing apparatus may implement each step of the log processing method, where the apparatus may include:
the log data obtaining module 201 is configured to obtain original log data of each trace band based on the message queue.
The log data screening module 202 is configured to screen target log data including a corresponding tracking band log identification identifier from each original log data.
The log data aggregation module 203 is configured to aggregate each target log data based on the tracking band log identification identifier of the target log data, to obtain call link data. And
The call link generation module 204 is configured to store the obtained call link data and the corresponding tracking area log identifier in a preset relational database after associating the call link data and the corresponding tracking area log identifier.
Further, the log data screening module 202 is further configured to:
And carrying out compliance verification on each target log data, and storing the target log data which is not verified to a distributed file system for problem investigation.
Further, the log data screening module 202 is further configured to: storing the checked target log data into a distributed database, and aggregating the target log data in the distributed database at intervals of a first preset duration;
and storing the aggregated target log data into the distributed file system based on the tracking band log identification mark.
Further, the log data aggregation module 203 is further configured to:
Comparing the current calling link data in the relational database with the calling link data in the database to be confirmed at intervals of a second preset time length;
and adding call link data which is different from the call link data in the database to be confirmed in the current call link data into the database to be confirmed.
Further, the call link generation module 204 is further configured to:
and generating an interface for external use based on the calling relation among all software in the calling link data of the relational database.
Further, the log data aggregation module 203 is specifically configured to:
Determining software contained in the call link data based on the interactive log data of the request type input in each target log data;
And determining the calling sequence relation among the software based on the response time of the response log data output in each target log data.
Further, the call link generation module 204 is further configured to: the call link data is adjusted based on a transaction identification generated on the call link.
The units involved in the embodiments of the present disclosure may be implemented by means of software, or may be implemented by means of hardware. The name of the unit does not in any way constitute a limitation of the unit itself, for example the first acquisition unit may also be described as "unit acquiring at least two internet protocol addresses".
The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), an Application Specific Standard Product (ASSP), a system on a chip (SOC), a Complex Programmable Logic Device (CPLD), and the like.
Referring to fig. 3, an embodiment of the present invention further provides an electronic device, which may include: a memory 301 and a processor 302.
A memory 301 for storing a program.
A processor 302 for executing the program, and implementing the respective steps of the log processing method as described in the above embodiment.
Embodiments of the present invention also provide a readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the log processing method described in the above embodiments.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
It should be noted that the computer readable medium described in the present disclosure may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present disclosure, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, fiber optic cables, RF (radio frequency), and the like, or any suitable combination of the foregoing.

Claims (10)

1. A log processing method, comprising:
Acquiring original log data of each tracking band based on the message queue;
Screening target log data containing corresponding tracking band log identification marks from each original log data;
Aggregating each target log data based on a tracking band log identification mark of each target log data to obtain calling link data;
And after correlating the obtained call link data with the corresponding tracking band log identification, storing the call link data into a preset relational database.
2. The log processing method as defined in claim 1, further comprising, after screening out target log data containing a corresponding trace strip log identification mark from each original log data:
And carrying out compliance verification on each target log data, and storing the target log data which is not verified to a distributed file system for problem investigation.
3. The log processing method as defined in claim 2, further comprising:
storing the checked target log data into a distributed database, and aggregating the target log data in the distributed database at intervals of a first preset duration;
and storing the aggregated target log data into the distributed file system based on the tracking band log identification mark.
4. The log processing method as defined in claim 1, further comprising:
Comparing the current calling link data in the relational database with the calling link data in the database to be confirmed at intervals of a second preset time length;
and adding call link data which is different from the call link data in the database to be confirmed in the current call link data into the database to be confirmed.
5. The log processing method as defined in claim 1, further comprising:
and generating an interface for external use based on the calling relation among all software in the calling link data of the relational database.
6. The log processing method according to claim 1, wherein the aggregating the target log data based on the trace band log identification identifier of the target log data to obtain the call link data includes:
Determining software contained in the call link data based on the interactive log data of the request type input in each target log data;
And determining the calling sequence relation among the software based on the response time of the response log data output in each target log data.
7. The log processing method as defined in claim 1, further comprising:
The call link data is adjusted based on a transaction identification generated on the call link.
8. A log processing apparatus, comprising:
the log data acquisition module is used for acquiring the original log data of each tracking band based on the message queue;
The log data screening module is used for screening target log data containing corresponding tracking band log identification marks from all original log data;
The log data aggregation module is used for aggregating each target log data based on the tracking band log identification mark of each target log data to obtain calling link data; and
And the call link generation module is used for storing the obtained call link data and the corresponding tracking band log identification into a preset relational database after being correlated.
9. An electronic device, comprising: a memory and a processor;
The memory is used for storing programs;
the processor is configured to execute the program to implement the respective steps of the log processing method according to any one of claims 1 to 7.
10. A readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the log processing method according to any one of claims 1 to 7.
CN202410317392.8A 2024-03-19 2024-03-19 Log processing method and device, electronic equipment and readable storage medium Pending CN118012832A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202410317392.8A CN118012832A (en) 2024-03-19 2024-03-19 Log processing method and device, electronic equipment and readable storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202410317392.8A CN118012832A (en) 2024-03-19 2024-03-19 Log processing method and device, electronic equipment and readable storage medium

Publications (1)

Publication Number Publication Date
CN118012832A true CN118012832A (en) 2024-05-10

Family

ID=90952235

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202410317392.8A Pending CN118012832A (en) 2024-03-19 2024-03-19 Log processing method and device, electronic equipment and readable storage medium

Country Status (1)

Country Link
CN (1) CN118012832A (en)

Similar Documents

Publication Publication Date Title
CN109684052B (en) Transaction analysis method, device, equipment and storage medium
CN111752799A (en) Service link tracking method, device, equipment and storage medium
CN111611100B (en) Transaction fault detection method, device, computing equipment and medium
CN110647447B (en) Abnormal instance detection method, device, equipment and medium for distributed system
CN112583644B (en) Alarm processing method, device, equipment and readable storage medium
CN112948492A (en) Data processing system, method and device, electronic equipment and storage medium
CN112711496A (en) Log information full link tracking method and device, computer equipment and storage medium
Zhang et al. A survey on quality assurance techniques for big data applications
CN113704077B (en) Test case generation method and device
CN111897806A (en) Big data offline data quality inspection method and device
CN113360376A (en) Buried point testing method and device
CN111061696A (en) Method and device for analyzing transaction message log
CN113596078A (en) Service problem positioning method and device
CN106648722A (en) Flume receiving side data processing method and device based on big data
CN113094269A (en) Application program test exception analysis method and device
CN116303380B (en) Data quality checking method, equipment and medium in monitoring service
CN118012832A (en) Log processing method and device, electronic equipment and readable storage medium
CN111291127B (en) Data synchronization method, device, server and storage medium
CN114996080A (en) Data processing method, device, equipment and storage medium
CN114218173A (en) Batch processing system, processing method, medium and equipment for account-transfer transaction files
CN111026616A (en) Information processing method and device, electronic equipment and storage medium
CN115168297A (en) Bypassing log auditing method and device
CN113128848A (en) Data quality monitoring method for full-service index, electronic equipment and storage medium
CN112948341A (en) Method and apparatus for identifying abnormal network device logs
CN112286792A (en) Interface testing method, device, equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination