CN115766856A - Dump and retrieval method for service bus message stream log - Google Patents

Abstract

The invention relates to the technical field of network information, and discloses a method for dumping and retrieving a service bus message flow log, which comprises the following steps: s1, setting a security interception component and a log interception component in an input pipeline of a service bus; s2, the security interception component verifies the request resource authority and acquires Http Header information or Token information, and the log interception component acquires request message information and request instance information; s3, the request is forwarded to the original service through the service; s4, receiving a result after the original service processing request and returning the result to an output pipeline of the service bus, and capturing returned message information and response example information through a log interception component of the output pipeline of the service bus; s5, asynchronously transmitting the message flow information and the message logs acquired by the log intercepting components of the input pipeline and the output pipeline to a JMS message queue for queuing; and S6, capturing the message from the JMS message queue through an asynchronous message receiving API, and storing the message into a database.

Description

Dump and retrieval method for service bus message stream log

Technical Field

The invention relates to the technical field of network information, in particular to a method for dumping and retrieving a service bus message flow log.

Background

Under a centralized framework realized by a service bus or an API gateway, the capabilities of safety control, current-limiting fusing, load balancing, routing and the like are realized in order to realize the control of the operation of an interface on the service bus. The common scheme is that message log information consumed by interface service is intercepted in an intrusive mode, the interface service operation logs are collected and stored, the interface service is further analyzed and optimized, the service management and control level is improved, and message stream logs are synchronously accessed to a local file or a database in the service operation process, so that message persistence is realized.

The method for synchronizing the persistent messages in the scheme has the advantages that the implementation method is direct, and the messages are acquired and stored as structured data in an immersion type mode. The service bus which is common at present supports the scheme, but the implementation mode of the scheme usually has the following disadvantages:

1. in consideration of the reliability problem of persistent storage, if a local persistent file is damaged or a database fails to cause that the file cannot be written, the interface call is failed;

2. the real-time synchronous writing library can cause huge performance pressure on the database, and simultaneously directly influences the performance of real-time calling of the interface service;

3. the common real-time synchronous writing library directly stores interface instance information and message information in the same table, and when the concurrency of data volume is large, the writing and query performance is extremely poor;

4. certain immersion transformation work needs to be carried out on the service, and the development difficulty is increased.

Disclosure of Invention

In order to solve the problems in the prior art, the invention provides a method for dumping and retrieving a service bus message stream log.

The technical scheme adopted by the invention is as follows:

a method for dumping and searching service bus message flow logs comprises the following steps:

s1, calling an agent service of a service bus to initiate a request, setting a security interception component and a log interception component in an input pipeline of the service bus, setting the log interception component in an output pipeline of the service bus, and capturing request data;

s2, verifying the request resource authority through a security interception component, acquiring Http Header information or Token information, and acquiring request message information and request instance information through a log interception component;

s3, forwarding the request to the original service by using the service through an input pipeline of the service bus;

s4, receiving a result after the original service processing request and returning the result to an output pipeline of the service bus, and capturing returned message information and response example information through a log interception component of the output pipeline of the service bus;

s5, asynchronously transmitting the message flow information and the message logs acquired by the log interception components of the input pipeline and the output pipeline of the service bus to a JMS message queue for queuing;

and S6, grabbing the message from the JMS message queue through an asynchronous message receiving API, and storing the message into a database to finish the persistence operation.

Preferably, in step S1, a security interception component and a log interception component are set in an input pipe of the service bus by XQuery.

Preferably, in step S3, the original service is a wsdl, restful or jms type service.

Preferably, in step S6, the JMS message is first stored in the local file, and then the message in the local file is read through the asynchronous message receiving API and stored in the database.

Preferably, in step S5, if the service bus is abnormal when writing data into the JMS message middleware, the log information is directly stored in the local file, and the writing is retried after the JMS message middleware is recovered.

Preferably, in step S6, the database stores the message log table and the message table in a time-sharing manner.

The beneficial effects of the invention are:

1. a set of complete design flow and method are provided, complete safety control and log interception functions are realized, and the functions of interface operation management and control and service management are realized, so that the development difficulty is reduced;

2. the JMS message queue is used for realizing asynchronous message access, the influence of an interception plug-in on interface operation is reduced to the maximum extent, and the table partition is used for improving the read-write performance on a database level.

Drawings

FIG. 1 is a schematic flow chart of an embodiment of the present invention;

FIG. 2 is a schematic storage diagram of a database according to an embodiment of the present invention;

Detailed Description

Embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

Examples

As shown in FIG. 1, a method for dumping and retrieving a service bus message flow log includes the following steps:

s1, an external interface calls an agent service of a service bus to initiate a request, a security interception component and a log interception component are arranged in an input pipeline of the service bus, a log interception component is arranged in an output pipeline of the service bus, and request data are captured;

s2, verifying the request resource authority through a security interception component, acquiring Http Header information or Token information, and acquiring request message information and request instance information through a log interception component;

s3, the request is forwarded to the original service by using the service through an input pipeline of the service bus;

s4, receiving a result after the original service processing request and returning the result to an output pipeline of the service bus, and capturing returned message information and response instance information through a log interception component of the output pipeline of the service bus;

s5, asynchronously transmitting the message flow information and the message logs acquired by the log intercepting components of the input pipeline and the output pipeline of the service bus to a JMS message queue for queuing;

and S6, capturing the message from the JMS message queue through an asynchronous message receiving API, and storing the message into a database to finish the persistence operation.

Steps S1-S4 are independent interface real-time call and return processes.

The first key point of the invention is that log interception components are arranged on an input pipeline and an output pipeline based on the same interception mechanism and concept aiming at different service buses, and a safety interception component is arranged on the input pipeline to carry out functions of Token verification, http Header interception, IP address verification and the like aiming at URI access. The log interception component acquires an interface calling message, generates a calling instance number, and records calling starting and ending time, running total time, input and output message size, communication condition, error reporting information and the like.

The second key point of the invention is that an Active MQ or a RabbitMQ is used for deploying a complete JMS message queue cluster, or a Weblogic JMS component is used for constructing the message queue cluster for receiving the message log.

The third key point of the present invention is an asynchronous message receiving API, which is used to asynchronously read the message in the message queue or read the message from the persistent storage (e.g. local file) of the message queue and write the message into the database.

Specifically, in step S1, a security interception component and a log interception component are set in an input pipe of the service bus through XQuery.

Specifically, in step S3, the original service is a wsdl, restful, or jms type service.

Specifically, in step S6, in order to improve the performance of the database, the JMS message may be stored in the local file, and then the message in the local file is read through the asynchronous message receiving API and stored in the database, where the local file persistent message is used in consideration that the performance of the sequential writing of the local file is higher than that of the database.

If the JMS message middleware is abnormal, in order to ensure that the log information is not lost, the invention further carries out reliability and redundancy design. Specifically, in step S5, if the service bus is abnormal when writing data into the JMS message middleware, the log information is directly stored in the local file, and the writing is retried after the JMS message middleware is recovered.

Specifically, in step S6, the database segments and stores the message log table and the message table according to time, and as shown in fig. 2, the table building statement may use the following codes:

creation of partition Table create _ Table Log _ Instance by day

(INSTANCE_ID NUMBER(20)not null,

SERVICE_NAME VARCHAR2(100),

REQ_APP_NAME VARCHAR2(50),

REQ_APP_IP VARCHAR2(50),

REMARK VARCHAR2(1000),

START_DATE DATE,

END_DATE DATE

)

// PARTITION information PARTITION BY RANGE (START _ DATE) INTERVAL

(NUMTODSINTERVAL(1,'day'))

(partition part_t01 values less than(to_date('2019-01-01','yyyy-mm-dd')))；

Creating a primary key, which is the entry _ Instance and constraint test _ pk primary key (INSTANCE _ ID) using INDEX;

the above-mentioned embodiments only express the specific embodiments of the present invention, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present invention. It should be noted that various changes and modifications can be made by those skilled in the art without departing from the spirit of the invention, and these changes and modifications are all within the scope of the invention.

Claims (6)

1. A method for dumping and searching service bus message flow logs is characterized by comprising the following steps:

s1, calling an agent service of a service bus to initiate a request, setting a security interception component and a log interception component in an input pipeline of the service bus, setting the log interception component in an output pipeline of the service bus, and capturing request data;

s2, verifying the request resource authority through a security interception component, acquiring Http Header information or Token information, and acquiring request message information and request instance information through a log interception component;

s3, the request is forwarded to the original service by using the service through an input pipeline of the service bus;

s4, receiving a result after the original service processing request and returning the result to an output pipeline of the service bus, and capturing returned message information and response example information through a log interception component of the output pipeline of the service bus;

s5, asynchronously transmitting the message flow information and the message logs acquired by the log interception components of the input pipeline and the output pipeline of the service bus to a JMS message queue for queuing;

and S6, grabbing the message from the JMS message queue through an asynchronous message receiving API, and storing the message into a database to finish the persistence operation.

2. The method for dumping and retrieving the service bus message stream log according to claim 1, wherein in step S1, a security interception component and a log interception component are set in the input pipe of the service bus by XQuery.

3. The method for dumping and retrieving the service bus message flow log according to claim 2, wherein in step S3, the original service is a wsdl, restful or jms type service.

4. The method for dumping and retrieving the service bus message flow log according to claim 3, wherein in step S6, the JMS message is stored in a local file, and then the message in the local file is read and stored in the database through an asynchronous message receiving API.

5. The method for dumping and retrieving the service bus message flow log according to claim 4, wherein in step S5, if the service bus is abnormal when writing data into the JMS message middleware, the log information is directly stored in the local file, and the writing is retried after the JMS message middleware is recovered.

6. The method for dumping and retrieving message stream logs of claim 5, wherein in step S6, the database stores the message log and the message table in time slices.

Images