CN109818805A - A kind of gateway log recording and search method - Google Patents

A kind of gateway log recording and search method Download PDF

Info

Publication number
CN109818805A
CN109818805A CN201910145409.5A CN201910145409A CN109818805A CN 109818805 A CN109818805 A CN 109818805A CN 201910145409 A CN201910145409 A CN 201910145409A CN 109818805 A CN109818805 A CN 109818805A
Authority
CN
China
Prior art keywords
log
gateway
data
log recording
search method
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201910145409.5A
Other languages
Chinese (zh)
Inventor
潘震
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shandong Inspur Cloud Information Technology Co Ltd
Original Assignee
Shandong Inspur Cloud Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shandong Inspur Cloud Information Technology Co Ltd filed Critical Shandong Inspur Cloud Information Technology Co Ltd
Priority to CN201910145409.5A priority Critical patent/CN109818805A/en
Publication of CN109818805A publication Critical patent/CN109818805A/en
Pending legal-status Critical Current

Links

Landscapes

  • Debugging And Monitoring (AREA)
  • Computer And Data Communications (AREA)

Abstract

The present invention is more particularly directed to a kind of gateway log recording and search methods.The gateway log recording and search method, firstly, the information generated in comprehensive record service invocation procedure, reasonable definition is log-structured, lays the foundation for log index;Then log is indexed and carries out reasonable definition and cutting, realization rationally indexes daily record data, so as to Accurate Analysis log information;Last creation analysis model, analyzes and counts log recording, to realize the log recording to the behavior for sending request data by gateway.The gateway log recording and search method realize the distributed recording for calling log to gateway, and then realize the real time monitoring of gateway service condition, improve the controllability in system use by the big datas technology such as Kafka, Elasticsearch.

Description

A kind of gateway log recording and search method
Technical field
The present invention relates to data retrieval analysis technical field, in particular to a kind of service network based on Elasticsearch Close log recording and search method.
Background technique
Heterogeneous database system is the set of relevant multiple Database Systems, and the shared and transparent visit of data may be implemented It asks, each Database Systems inherently have existed before heterogeneous database system is added, and possess the DBMS of oneself.Isomery number There is the autonomy of itself according to each component part in library, while realizing data sharing, each Database Systems are still possessed certainly Oneself application characteristic, integrity control and safety control.
The isomerism of heterogeneous database system is mainly reflected in the following aspects:
(1) isomery of Computer Architecture
The database of each participation can be separately operable in mainframe, minicomputer, work station, PC or embedded system.
(2) isomery of underlying Operating System
The underlying Operating System of each Database Systems can be Unix, Windows NT, Linux etc..
(3) isomery of DBMS itself
Can be is all Oracle, SQL Server of system R etc., is also possible to different data model Database, such as relationship, mode, level, network, object-oriented, function type database collectively constitutes a heterogeneous database system System.
Heterogeneous database system aims at data information resource, the hardware device resources realized between disparate databases Merging with human resources and shared.Wherein crucial is a little exactly to establish global number based on local data bank mode According to mode or global external view.This global schema is particularly important for establishing advanced DSS.
Large corporation suffers from the Information Number of oneself in many place Dou You branch, the database of each clamp mechanism According to, and decision work out personnel be generally only concerned macroscopic view, for information described in global schema.It establishes in data warehouse technology base The description of heterogeneous database global schema on plinth is a kind of good solution.Data warehouse can be from heterogeneous database system In multiple databases in collect information, and establish unified global schema, the data being collected simultaneously also are supported to historical data Access, user carries out the inquiry of decision support by the unified data-interface that data warehouse provides.
For heterogeneous database system, realize that data sharing should reach two o'clock: first is that realizing database conversion;Second is that real The transparent access of existing data.But there is presently no a kind of widely used data definition model and data query languages, realize The transparent access of data can be using technologies such as many-one conversion, two-way middlewares.Open Database Connectivity (Open DataBaseconnectivity, abbreviation ODBC) it is that one kind is used to access in related or incoherent data base management system The standard application program interface (API) of data.ODBC provides a set of high-rise calling interface specification for application program and is based on dynamic The operation support environment of state chained library.
During Information System configuration, the interface docking being constantly present between heterogeneous system, two-by-two between system Frequently docking not only brings heavy overlapping development work, also becomes that data are trans-regional, and inter-trade, trans-departmental circulation is altogether The obstacle enjoyed.
Then, unified data distribution platform is established, the reticular structure of original system joint mode is become star structure, Guarantee only to dock primary joint mode with me.To greatly reduce the workload in system docking, and become a kind of trend.For Meet the needs of this business scenario, building unified service management platform becomes inevitable, unified service call gateway is constructed, It realizes the record to gateway requests log, retrieval, polymerization and analysis, becomes the important means for ensureing service invariant operation.
For comprehensive retrieval, the service call situation of monitoring and analysis gateway requests data finds system fortune in time Potential problem in row, the value of mining data keep the stabilization of system operation.The invention proposes one kind to be based on The gateway log recording and search method of Elasticsearch.
Summary of the invention
In order to compensate for the shortcomings of the prior art, the present invention provides it is a kind of be simple and efficient based on Elasticsearch's Gateway log recording and search method.
The present invention is achieved through the following technical solutions:
A kind of gateway log recording and search method, it is characterised in that: firstly, comprehensive record service call mistake The information generated in journey, reasonable definition is log-structured, lays the foundation for log index;Then log is indexed and carries out reasonable definition And cutting, realization rationally indexes daily record data, so as to Accurate Analysis log information;Last creation analysis model, to day Will record analyzes and counts, to realize the log recording to the behavior for sending request data by gateway.
The reasonable definition is log-structured to be referred to for comprehensive record interface calling process log, service call it Preceding clear log recording content defines log information.
Log recording is the process continued to increase, and the mode of clear log storage, institute are needed before record log It states and indexes that carry out reasonable definition and cutting and assisted defining log with template version and store using the date as cutting foundation to log Rule.
Described analyze and count to log recording is to call situation with web form monitoring service, in service invocation procedure In, with the generation of log, gateway service situation is monitored in real time by visual interface.
Gateway log recording of the present invention and search method are defined the full-range log of service call, in line with The principle of information redundancy records the full spectrum information of service call.
Gateway log recording of the present invention and search method realize the hair generated to log by Kafka message queue Send with consumer record process, guarantee the completion of log recording;Rationally estimate the order of magnitude that log generates, defines log recording Index templates rely on the real-time of Elasticsearch, stability and ability extending transversely, cut to log according to monthly Point, realize the reasonable storage to daily record data.
Gateway log recording of the present invention and search method reasonably model daily record data, construct abundant Statistical model calls the case where servicing to carry out in real time accurately reaction gateway.
Gateway log recording of the present invention and search method, the specific steps are as follows:
(1) request data, simultaneously record log is sent in Kafka for gateway response, and consumption terminal pulls number in batches from Kafka According to being sent in Elasticsearch and local data base and save;
(2) data give over to backup in local data base, and data are based on passing through unified indices modulo in Elasticsearch Plate carries out cutting index with date (yyyyMM) and saves, the storage of reasonable distribution data, thus realize unified data retrieval, number According to analysis.
The beneficial effects of the present invention are: the gateway log recording and search method, by Kafka, The big datas technology such as Elasticsearch, comprehensive retrieval, the service call situation of monitoring and analysis gateway requests data, The distributed recording for calling log to gateway is realized, and then realizes the real time monitoring of gateway service condition, can be sent out in time Potential problem, the value of mining data keep the stabilization of system operation in existing system operation, improve in system use can Control property.
Detailed description of the invention
Attached drawing 1 is gateway log recording of the present invention and search method schematic diagram.
Specific embodiment
In order to which technical problems, technical solutions and advantages to be solved are more clearly understood, tie below Drawings and examples are closed, the present invention will be described in detail.It should be noted that specific embodiment described herein is only used To explain the present invention, it is not intended to limit the present invention.
The gateway log recording and search method, firstly, the letter generated in comprehensive record service invocation procedure Breath, reasonable definition is log-structured, lays the foundation for log index;Then log is indexed and carries out reasonable definition and cutting, realized Daily record data is rationally indexed, so as to Accurate Analysis log information;Last creation analysis model, divides log recording Analysis statistics, to realize the log recording to the behavior for sending request data by gateway.
The reasonable definition is log-structured to be referred to for comprehensive record interface calling process log, service call it Preceding clear log recording content defines log information.
Log recording is the process continued to increase, and the mode of clear log storage, institute are needed before record log It states and indexes that carry out reasonable definition and cutting and assisted defining log with template version and store using the date as cutting foundation to log Rule.
Described analyze and count to log recording is to call situation with web form monitoring service, in service invocation procedure In, with the generation of log, gateway service situation is monitored in real time by visual interface.
The gateway log recording and search method are defined the full-range log of service call, in line with information The principle of redundancy records the full spectrum information of service call.
The gateway log recording and search method, by Kafka message queue, realize transmission that log is generated with Consumer record process guarantees the completion of log recording;Rationally estimate the order of magnitude that log generates, defines the index of log recording Template relies on the real-time of Elasticsearch, stability and ability extending transversely, real to log according to monthly carry out cutting Now to the reasonable storage of daily record data.
The gateway log recording and search method reasonably model daily record data, construct statistics abundant Model calls the case where servicing to carry out in real time accurately reaction gateway.
The gateway log recording and search method, the specific steps are as follows:
(1) request data, simultaneously record log is sent in Kafka for gateway response, and consumption terminal pulls number in batches from Kafka According to being sent in Elasticsearch and local data base and save;
(2) data give over to backup in local data base, and data are based on passing through unified indices modulo in Elasticsearch Plate carries out cutting index with date (yyyyMM) and saves, the storage of reasonable distribution data, thus realize unified data retrieval, number According to analysis.
The gateway log recording and search method, by the big datas technology such as Kafka, Elasticsearch, Quan Fang The retrieval of position, the service call situation of monitoring and analysis gateway requests data realize the distributed note that log is called to gateway Record, and then the real time monitoring of gateway service condition is realized, problem potential in system operation, mining data can be found in time Value, keep system operation stabilization, improve the controllability in system use.

Claims (8)

1. a kind of gateway log recording and search method, it is characterised in that: firstly, comprehensive record service invocation procedure The information of middle generation, reasonable definition is log-structured, lays the foundation for log index;Then to log index carry out reasonable definition and Cutting, realization rationally indexes daily record data, so as to Accurate Analysis log information;Last creation analysis model, to log Record analyzes and counts, to realize the log recording to the behavior for sending request data by gateway.
2. gateway log recording according to claim 1 and search method, it is characterised in that: the reasonable definition day Will structure refers to that for comprehensive record interface calling process log, the clear log recording content before service call is fixed Adopted log information.
3. gateway log recording according to claim 1 and search method, it is characterised in that: log recording is one The process continued to increase, needs the mode of clear log storage before record log, and described index to log carries out Rational Determination Justice and cutting are using the date as cutting foundation, and auxiliary with template version defines log storage rule.
4. gateway log recording according to claim 1 and search method, it is characterised in that: described to log recording Analyze and count is to call situation, with the generation of log, to pass through in service invocation procedure with web form monitoring service Visual interface monitors gateway service situation in real time.
5. gateway log recording according to claim 2 and search method, it is characterised in that: flowed entirely to service call The log of journey is defined, and in line with the principle of information redundancy, is recorded to the full spectrum information of service call.
6. gateway log recording according to claim 3 and search method, it is characterised in that: by Kafka message Queue realizes the transmission and consumer record process generated to log, guarantees the completion of log recording;Rationally estimate log to generate The order of magnitude, define the index templates of log recording, rely on the real-time of Elasticsearch, stability and energy extending transversely Power realizes the reasonable storage to daily record data to log according to monthly carry out cutting.
7. gateway log recording according to claim 4 and search method, it is characterised in that: carried out to daily record data Reasonable modeling, constructs statistical model abundant, calls the case where servicing to carry out in real time accurately reaction gateway.
8. the gateway log recording according to any one of claim 5,6 or 7 and search method, feature exist In, comprising the following steps:
(1) request data, simultaneously record log is sent in Kafka for gateway response, consumption terminal batch pulling data from Kafka, It is sent in Elasticsearch and local data base and saves;
(2) data give over to backup in local data base, in Elasticsearch data based on by unified index templates, with Date carries out cutting index and saves, the storage of reasonable distribution data, thus realize unified data retrieval, data analysis.
CN201910145409.5A 2019-02-27 2019-02-27 A kind of gateway log recording and search method Pending CN109818805A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910145409.5A CN109818805A (en) 2019-02-27 2019-02-27 A kind of gateway log recording and search method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910145409.5A CN109818805A (en) 2019-02-27 2019-02-27 A kind of gateway log recording and search method

Publications (1)

Publication Number Publication Date
CN109818805A true CN109818805A (en) 2019-05-28

Family

ID=66607541

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910145409.5A Pending CN109818805A (en) 2019-02-27 2019-02-27 A kind of gateway log recording and search method

Country Status (1)

Country Link
CN (1) CN109818805A (en)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111049673A (en) * 2019-11-21 2020-04-21 山东健康医疗大数据有限公司 Method and system for counting and monitoring API call in service gateway
CN111654410A (en) * 2020-04-28 2020-09-11 长沙证通云计算有限公司 Gateway request monitoring method, device, equipment and medium
CN112073408A (en) * 2020-09-04 2020-12-11 南京星邺汇捷网络科技有限公司 Monitoring method based on Syslog protocol and Kafka queue
CN113472808A (en) * 2021-07-16 2021-10-01 浙江大华技术股份有限公司 Log processing method and device, storage medium and electronic device
CN114116402A (en) * 2021-11-30 2022-03-01 山东浪潮云缦智能科技有限公司 Embedded linux-based loosely-coupled and high-expansibility non-buried point data acquisition method

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180307576A1 (en) * 2017-04-21 2018-10-25 Nec Laboratories America, Inc. Field content based pattern generation for heterogeneous logs
CN108712329A (en) * 2018-05-02 2018-10-26 山东汇贸电子口岸有限公司 A kind of gateway and log recording retrieval device based on Elasticsearch
CN108874614A (en) * 2017-05-11 2018-11-23 上海宏时数据系统有限公司 A kind of big data log intelligent analysis system and method

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180307576A1 (en) * 2017-04-21 2018-10-25 Nec Laboratories America, Inc. Field content based pattern generation for heterogeneous logs
CN108874614A (en) * 2017-05-11 2018-11-23 上海宏时数据系统有限公司 A kind of big data log intelligent analysis system and method
CN108712329A (en) * 2018-05-02 2018-10-26 山东汇贸电子口岸有限公司 A kind of gateway and log recording retrieval device based on Elasticsearch

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
许长福: "日志数据分析系统的设计与实现", 《中国优秀硕士学位论文全文数据库•信息科技辑》 *

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111049673A (en) * 2019-11-21 2020-04-21 山东健康医疗大数据有限公司 Method and system for counting and monitoring API call in service gateway
CN111654410A (en) * 2020-04-28 2020-09-11 长沙证通云计算有限公司 Gateway request monitoring method, device, equipment and medium
CN111654410B (en) * 2020-04-28 2021-12-24 长沙证通云计算有限公司 Gateway request monitoring method, device, equipment and medium
CN112073408A (en) * 2020-09-04 2020-12-11 南京星邺汇捷网络科技有限公司 Monitoring method based on Syslog protocol and Kafka queue
CN112073408B (en) * 2020-09-04 2023-12-01 南京星邺汇捷网络科技有限公司 Monitoring method based on Syslog protocol and Kafka queue
CN113472808A (en) * 2021-07-16 2021-10-01 浙江大华技术股份有限公司 Log processing method and device, storage medium and electronic device
CN114116402A (en) * 2021-11-30 2022-03-01 山东浪潮云缦智能科技有限公司 Embedded linux-based loosely-coupled and high-expansibility non-buried point data acquisition method
CN114116402B (en) * 2021-11-30 2024-09-17 山东浪潮超高清视频产业有限公司 Loosely coupled and high-expansibility buried point free data acquisition method based on embedded linux

Similar Documents

Publication Publication Date Title
CN109818805A (en) A kind of gateway log recording and search method
US7412451B2 (en) Method for persisting a schedule and database schema
CN111045656A (en) Method and system for constructing platform system infrastructure in government affair service
CN109656963B (en) Metadata acquisition method, apparatus, device and computer readable storage medium
US6845392B2 (en) Remote systems management via DBMS stored procedures and one communication line
CN109344207B (en) Big data platform of integrative frequency spectrum all over the sky based on big dipper scanning
CN106951552A (en) A kind of user behavior data processing method based on Hadoop
CN112419129A (en) Method and system for constructing platform system infrastructure in government affair service
WO2023202497A1 (en) Method for full-link tracking of transaction, and native distributed database
CN104504014A (en) Data processing method and device based on large data platform
CN112883001A (en) Data processing method, device and medium based on marketing and distribution through data visualization platform
CN112100227A (en) Big data processing method based on multilevel heterogeneous data storage
CN103810203A (en) Connection multiplexing method and connection multiplexing device for database management system
CN111984301A (en) Micro-service data management framework based on spring close and kubernets
Sridhar et al. Optimizing database load and extract for big data era
CN115689788A (en) Financial data analysis method
Tang et al. Research and application of improving the field service ability of electric power marketing measuring mobile operating based on cloud computing technology
CN110928938B (en) Interface middleware system
Zhang The enterprise personnel management system based on B/S Design
CN110716976A (en) Distributed data access and storage system based on two interactive scenes
CN105550143A (en) Interface system for acquiring working condition of information equipment
Shi et al. HEDC++: an extended histogram estimator for data in the cloud
CN111459965B (en) Information monitoring system for edge calculation
Xiao et al. The research of teradata data warehouse technology
Zhao et al. Architecture Design of CTC Log Module Based on Web Service

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20190528