CN114168596A

CN114168596A - Kibana-based account following data preparation method and system

Info

Publication number: CN114168596A
Application number: CN202111500680.XA
Authority: CN
Inventors: 栾博
Original assignee: China Construction Bank Corp
Current assignee: China Construction Bank Corp
Priority date: 2021-12-09
Filing date: 2021-12-09
Publication date: 2022-03-11

Abstract

The invention relates to a Kibana-based account following data preparation method and system, which comprises the steps of obtaining first log information and second log information, setting a crawler rule, firstly screening out specific warehouse transaction running water information according to the crawler rule and the second log information, then crawling the first log information according to the warehouse transaction running water information, and obtaining a request message and a response message of transaction, wherein the request message is mainly used for test input of account following, and the response message is used for comparing with the response message obtained by the account following test so as to verify the correctness of new system functions. By adopting the method and the system, the acquisition of various financial transaction type test messages based on the Kibana platform is realized, meanwhile, the acquisition efficiency of the account following test data is improved, and the high coverage rate of the account following test is ensured.

Description

Kibana-based account following data preparation method and system

Technical Field

The invention relates to the technical field of information processing, in particular to a method and a system for preparing account following data based on Kibana.

Background

Kibana is an open source analysis and visualization platform designed for use with an elastic search, through which system log data stored in an elastic search index can be conveniently queried. The interaction mode of Kibana and the elastic search is various different charts, tables, maps and the like, and data are visually displayed, so that the purposes of high-level data analysis and visualization are achieved. At present, Kibana is mainly applied to log analysis and display scenes of a system.

The crawler is also called a web spider, which is a program or script for automatically crawling the world wide web information according to certain rules. The system can replace human beings to automatically collect and arrange data information in the Internet.

The account following refers to a process of performing 'reappearance' on the new system in a certain mode on the transaction generated by the old system after the new system simulates online operation, and checking an operation result with the old system. The account tracking usually includes the processes of preparation of account tracking data, transaction messages of the account tracking data, database comparison and the like.

When a conventional financial core accounting system is subject to switching between a new system and an old system, a comprehensive functional test is often required to verify the correctness of the new system, so the financial core accounting system usually introduces a tracking test to perform comprehensive functional verification on the functions of the new system. However, the conventional preparation of the account following test data usually requires manual production of the test data through simulation of business transactions, which usually consumes a lot of labor cost, and meanwhile, the coverage of the account following test data is ensured, which may not be comprehensive due to the manual factor. Therefore, a full-automatic method for preparing the account following test data is urgently needed.

Disclosure of Invention

The patent provides a preparation method and system of account following data based on Kibana, and the crawler technology is combined with the characteristics of a Kibana system to automatically acquire transaction messages of a financial core accounting system of a database and table type, and the crawled transaction messages can provide input data of an account following test and output data for comparison in the account following process, so that automatic acquisition of the account following test data is realized.

In order to achieve the above purpose, the technical scheme adopted by the invention comprises the following steps:

a Kibana-based account following data preparation method is characterized by comprising the following steps:

s1, acquiring first log information and second log information of Kibana, wherein the first log information is performance condition information of the transaction, and the second log information is content information of the transaction;

s2, setting a crawler rule, wherein the crawler rule comprises transaction information and a quantity threshold value of a transaction message to be crawled, and the transaction information comprises a transaction code range, sub-base information and a transaction processing timestamp range;

s3, searching and matching in the first log information according to the crawler rule;

s4, acquiring a serial number of the transaction information crawled in the first log information, and searching second log information by taking the serial number as a query condition;

s5, storing the transaction message in the searched second log information into a database as a request message and a response message required by the account following test;

s6, judging whether the number of the searched transaction messages reaches the threshold value,

if the number of the searched transaction messages reaches a threshold value, stopping searching;

and if the number of the searched transaction messages does not reach the threshold value, repeating S3 to S5.

Further, the performance condition information comprises a transaction serial number, database processing time, internal call transaction processing time and sub-base information of the transaction processing unit.

Furthermore, the content information comprises a transaction serial number, a transaction message, a transaction processing timestamp, a transaction processing unit and a transaction code.

Further, the first log information and the second log information are regularly matched with the transaction information field through a script program embedded in the system and are stored in the real-time search engine.

Further, the request message in S5 is an input of the account following test, and the response message is used for comparing with the response message obtained by the account following test.

Further, the database includes an Oracle database and a Mysql database.

The invention also relates to a system for preparing account following data based on Kibana, which is characterized by comprising the following components:

the acquisition module is used for acquiring first log information and second log information of Kibana and crawled transaction information;

the crawler rule management module is used for setting crawler rules;

a processing module for crawling transaction information using crawler rules;

the storage module is used for storing the first log information, the second log information, the crawler rule and the crawled transaction information;

the information searching module is used for searching second log information by taking the serial number as a query condition;

and the judging module is used for judging whether the number of the searched transaction messages reaches a threshold value.

The invention also relates to a computer-readable storage medium, characterized in that the storage medium has stored thereon a computer program which, when being executed by a processor, carries out the above-mentioned method.

The invention also relates to an electronic device, characterized in that it comprises a processor and a memory;

the memory is used for storing the first log information, the second log information, the crawler rule and the crawled transaction information;

the processor is used for executing the method according to the log information and the crawler rule stored in the memory.

The invention also relates to a computer program product comprising a computer program and/or instructions, characterized in that the computer program and/or instructions, when executed by a processor, implement the steps of the above-mentioned method.

The invention has the beneficial effects that:

by adopting the Kibana-based account following data preparation method and system, the crawler technology is combined with the characteristics of the Kibana system, the transaction messages of the financial core financial system of a database and table type are automatically acquired, the acquired transaction messages can provide input data of an account following test and output data for comparison in the account following process, the acquisition of various financial transaction test messages based on a Kibana platform is realized, meanwhile, the automatic acquisition of the account following test data also improves the acquisition efficiency of the account following test data, and the high coverage rate of the account following test is ensured.

Drawings

Fig. 1 is a flow chart of a preparation method of account following data based on Kibana according to the present invention.

FIG. 2 is a schematic diagram of a Kibana-based accounting data preparation system according to the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

The invention relates to a Kibana-based account following data preparation method with a step flow shown in FIG. 1, which comprises the following steps:

kibana is an elastic search front-end display tool based on a browser page and is also an open-source and free tool, and Kibana can provide a log analysis friendly Web interface for Logstash and elastic search and can help to summarize, analyze and search important data logs. The ElasticSearch is an open source distributed search server based on Lucene. It is characterized in that: distributed, zero configuration, automatic discovery, index automatic fragmentation, index copy mechanism, restful style interface, multiple data sources, automatic search load, etc. It provides a distributed multi-user capable full-text search engine based on RESTful web interface. The Elasticsearch was developed in Java and published as open source under the Apache licensing terms, being the second popular enterprise search engine. The design is used in cloud computing, can achieve real-time search, and is stable, reliable, quick, and convenient to install and use.

Specifically, in the financial field, the Kibana platform can search transaction log information of financial core accounts, and system operation and maintenance personnel can conveniently monitor and analyze transaction processing capacity and system performance of the financial system. Two types of important log information, a first log information (log _ probability) and a second log information (log _ message), can be searched. Specifically, after log information is acquired from an elastic search tool, the Kibana tool is used for performing visual retrieval processing on the log information.

The first log information (log _ performance) mainly records the detailed performance condition of each transaction, and specifically comprises a financial transaction flow number EVT _ TRACE _ ID, a database processing time DB _ COSTS, an internal call transaction processing time INNERCALL _ COSTS, and a sub-bank grouping SPUNAME to which a transaction processing unit belongs;

the second log information (log _ message) mainly records information such as a request and a response message of a transaction, transaction flow and the like, and specifically includes a financial transaction flow number EVT _ TRACE _ ID, a transaction message UserMessage, a transaction processing timestamp, a transaction processing unit UnitName and transaction CODE TX _ CODE information.

The first log information (log _ probability) and the second log information (log _ message) are both realized by regularly matching a script program embedded in the system to keywords such as serial numbers, transaction messages and the like, and the financial transaction information fields are structurally stored in the elastic search.

Since the second log information (log _ message) does not have information of the sub-base and sub-table, but the account following test data needs to prepare test data according to the sub-base and sub-table, specific sub-base and sub-table transaction flow information needs to be screened out first by combining the first log information (log _ transfer), and then a transaction request and a transaction response message are obtained according to the flow information. The request message is mainly used for test input of account tracking, and the obtained response message is used for comparing with a response message obtained by account tracking test transaction so as to verify the correctness of the new system function.

Firstly, collecting test transaction data based on a first log information (log _ probability) and a second log information (log _ message) financial transaction log type on a Kibana platform, analyzing and matching search results of the first log information (log _ probability) and the second log information (log _ message), and filtering out account following test data.

Specifically, step S1: acquiring first log information (log _ probability) and second log information (log _ message) of Kibana, specifically, after acquiring the log information from an elastic search tool, performing visual retrieval processing on the log information by using the Kibana tool. The first log information (log _ performance) is the performance condition information of the transaction, and specifically comprises a financial transaction flow number EVT _ TRACE _ ID, a database processing time DB _ COSTS, an internal call transaction processing time INNERCALL _ COSTS, and a sub-bank group SPUNAME to which a transaction processing unit belongs; the second log information (log _ message) is transaction content information, and specifically comprises a financial transaction serial number EVT _ TRACE _ ID, a transaction message UserMessage, a transaction processing timestamp, a transaction processing unit UnitName and transaction CODE TX _ CODE information; the first log information (log _ probability) and the second log information (log _ message) are regularly matched with the transaction information field through a script program embedded in the system and are stored in a real-time search engine

Step S2: setting a crawler rule, wherein the crawler rule comprises transaction information and a quantity threshold value of a transaction message to be crawled, and the transaction information comprises a transaction CODE TX _ CODE range, sub-base information SPUNAME and a transaction processing timestamp range;

step S3: searching and matching in first log information (log _ probability) according to the crawler rule;

step S4: searching for second log information (log _ message) using the serial number EVT _ TRACE _ ID of the transaction information crawled in the first log information (log _ transfer) in step S3 as a query;

step S5: and storing the transaction message UserMessage in the second log information (log _ message) searched in the step S4 into a database as a request message and a response message required by the account following test, wherein the transaction message UserMessage includes the request message and the response message.

Step S6: judging whether the number of the searched transaction messages UserMessages reaches a threshold value, and stopping searching if the number of the searched transaction messages UserMessages reaches the threshold value; if the number of the found transaction messages UserMessages does not reach the threshold value, repeating the steps S3 to S5 until the number of the transaction messages UserMessages reaches the threshold value, and thus, completing the data preparation work in the account following test.

In the account tracking test, the request message in the second log information (log _ message) found in step S4 is used as an input of the account tracking test, and the response message obtained in the account tracking test is compared with the response message in the second log information (log _ message) found in step S4, so as to verify the correctness of the new system function.

By applying the method, the collection of various financial transaction type test messages based on the Kibana platform can be realized, meanwhile, the collection efficiency of the account following test data is improved, and the high coverage rate of the account following test is ensured.

Another aspect of the present invention also relates to a Kibana-based tracking data preparation system, whose structure is shown in fig. 2, including:

the acquisition module is used for acquiring first log information (log _ probability) and second log information (log _ message) of Kibana and transaction information crawled according to a crawler rule; (ii) a

The crawler rule management module is used for setting crawler rules;

a processing module for crawling transaction information using crawler rules;

the storage module is used for storing first log information (log _ probability), second log information (log _ message), crawler rules and crawled transaction information;

By using this system, the above-described arithmetic processing method can be executed and a corresponding technical effect can be achieved.

An embodiment of the present invention further provides a computer-readable storage medium capable of implementing all the steps in the Kibana-based ledger data preparation method in the above embodiment, where the computer-readable storage medium stores thereon a computer program, and when the computer program is executed by a processor, the computer program implements all the steps of the Kibana-based ledger data preparation method in the above embodiment.

Embodiments of the present invention also provide an electronic device for executing the above method, as an implementation apparatus of the method, the electronic device at least has a processor and a memory, in particular, the memory stores data and related computer programs, such as log information, crawler rules, etc., required for executing the method, and all steps of implementing the method are executed by calling the data and programs in the memory by the processor, and corresponding technical effects are obtained.

Preferably, the electronic device may comprise a bus architecture, which may include any number of interconnected buses and bridges linking together various circuits including one or more processors and memory. The bus may also link various other circuits such as peripherals, voltage regulators, power management circuits, and the like, which are well known in the art, and therefore, will not be described any further herein. A bus interface provides an interface between the bus and the receiver and transmitter. The receiver and transmitter may be the same element, i.e., a transceiver, providing a means for communicating with various other systems over a transmission medium. The processor is responsible for managing the bus and general processing, while the memory may be used for storing data used by the processor in performing operations.

Additionally, the electronic device may further include a communication module, an input unit, an audio processor, a display, a power source, and the like. The processor (or controller, operational controls) employed may include a microprocessor or other processor device and/or logic device that receives input and controls the operation of various components of the electronic device; the memory may be one or more of a buffer, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices, and may store the above-mentioned related data information, and may also store a program for executing the related information, and the processor may execute the program stored in the memory to realize information storage or processing, etc.; the input unit is used for providing input to the processor, and can be a key or a touch input device; the power supply is used for supplying power to the electronic equipment; the display is used for displaying display objects such as images and characters, and may be an LCD display, for example. The communication module is a transmitter/receiver that transmits and receives signals via an antenna. The communication module (transmitter/receiver) is coupled to the processor to provide an input signal and receive an output signal, which may be the same as in the case of a conventional mobile communication terminal. Based on different communication technologies, a plurality of communication modules, such as a cellular network module, a bluetooth module, and/or a wireless local area network module, may be disposed in the same electronic device. The communication module (transmitter/receiver) is also coupled to a speaker and a microphone via an audio processor to provide audio output via the speaker and receive audio input from the microphone to implement the usual telecommunication functions. The audio processor may include any suitable buffers, decoders, amplifiers and so forth. In addition, the audio processor is also coupled to the central processor, so that recording on the local machine can be realized through the microphone, and sound stored on the local machine can be played through the loudspeaker.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a system for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction system which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks. While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all such alterations and modifications as fall within the scope of the invention.

The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A Kibana-based account following data preparation method is characterized by comprising the following steps:

s2, setting a crawler rule, wherein the crawler rule comprises transaction information and a quantity threshold value of a transaction message to be crawled;

2. The method of claim 1, wherein the performance status information includes a transaction serial number, a database processing time, an internal call transaction processing time, and sub-base information to which the transaction processing unit belongs.

3. The method of claim 2, wherein the content information includes a transaction serial number, a transaction message, a transaction processing timestamp, a transaction processing unit, and a transaction code.

4. The method of claim 3, wherein the transaction information includes a transaction code range, banking information, and a transaction processing timestamp range.

5. The method of claim 4, wherein the first log information and the second log information are regularly matched to transaction information fields by a script embedded in the system and stored in a real-time search engine.

6. The method according to claim 1, wherein the request message in S5 is an input of an account-following test, and the response message is used for comparison with a response message obtained by the account-following test.

7. The method of claim 1, wherein the database comprises an Oracle database and a Mysql database.

8. A Kibana-based ledger data preparation system, comprising:

the crawler rule management module is used for setting crawler rules;

a processing module for crawling transaction information using crawler rules;

9. A computer-readable storage medium, characterized in that the storage medium has stored thereon a computer program which, when being executed by a processor, carries out the method of any one of claims 1 to 7.

10. An electronic device comprising a processor and a memory;

the processor is used for executing the method of any one of claims 1 to 7 according to the log information and the crawler rules stored in the memory.

11. A computer program product comprising a computer program and/or instructions, characterized in that the computer program and/or instructions, when executed by a processor, implement the steps of the method of any one of claims 1 to 7.