CN109818805A - A kind of gateway log recording and search method - Google Patents
A kind of gateway log recording and search method Download PDFInfo
- Publication number
- CN109818805A CN109818805A CN201910145409.5A CN201910145409A CN109818805A CN 109818805 A CN109818805 A CN 109818805A CN 201910145409 A CN201910145409 A CN 201910145409A CN 109818805 A CN109818805 A CN 109818805A
- Authority
- CN
- China
- Prior art keywords
- log
- gateway
- data
- log recording
- search method
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Debugging And Monitoring (AREA)
- Computer And Data Communications (AREA)
Abstract
The present invention is more particularly directed to a kind of gateway log recording and search methods.The gateway log recording and search method, firstly, the information generated in comprehensive record service invocation procedure, reasonable definition is log-structured, lays the foundation for log index；Then log is indexed and carries out reasonable definition and cutting, realization rationally indexes daily record data, so as to Accurate Analysis log information；Last creation analysis model, analyzes and counts log recording, to realize the log recording to the behavior for sending request data by gateway.The gateway log recording and search method realize the distributed recording for calling log to gateway, and then realize the real time monitoring of gateway service condition, improve the controllability in system use by the big datas technology such as Kafka, Elasticsearch.
Description
Technical field
The present invention relates to data retrieval analysis technical field, in particular to a kind of service network based on Elasticsearch
Close log recording and search method.
Background technique
Heterogeneous database system is the set of relevant multiple Database Systems, and the shared and transparent visit of data may be implemented
It asks, each Database Systems inherently have existed before heterogeneous database system is added, and possess the DBMS of oneself.Isomery number
There is the autonomy of itself according to each component part in library, while realizing data sharing, each Database Systems are still possessed certainly
Oneself application characteristic, integrity control and safety control.
The isomerism of heterogeneous database system is mainly reflected in the following aspects:
(1) isomery of Computer Architecture
The database of each participation can be separately operable in mainframe, minicomputer, work station, PC or embedded system.
(2) isomery of underlying Operating System
The underlying Operating System of each Database Systems can be Unix, Windows NT, Linux etc..
(3) isomery of DBMS itself
Can be is all Oracle, SQL Server of system R etc., is also possible to different data model
Database, such as relationship, mode, level, network, object-oriented, function type database collectively constitutes a heterogeneous database system
System.
Heterogeneous database system aims at data information resource, the hardware device resources realized between disparate databases
Merging with human resources and shared.Wherein crucial is a little exactly to establish global number based on local data bank mode
According to mode or global external view.This global schema is particularly important for establishing advanced DSS.
Large corporation suffers from the Information Number of oneself in many place Dou You branch, the database of each clamp mechanism
According to, and decision work out personnel be generally only concerned macroscopic view, for information described in global schema.It establishes in data warehouse technology base
The description of heterogeneous database global schema on plinth is a kind of good solution.Data warehouse can be from heterogeneous database system
In multiple databases in collect information, and establish unified global schema, the data being collected simultaneously also are supported to historical data
Access, user carries out the inquiry of decision support by the unified data-interface that data warehouse provides.
For heterogeneous database system, realize that data sharing should reach two o'clock: first is that realizing database conversion；Second is that real
The transparent access of existing data.But there is presently no a kind of widely used data definition model and data query languages, realize
The transparent access of data can be using technologies such as many-one conversion, two-way middlewares.Open Database Connectivity (Open
DataBaseconnectivity, abbreviation ODBC) it is that one kind is used to access in related or incoherent data base management system
The standard application program interface (API) of data.ODBC provides a set of high-rise calling interface specification for application program and is based on dynamic
The operation support environment of state chained library.
During Information System configuration, the interface docking being constantly present between heterogeneous system, two-by-two between system
Frequently docking not only brings heavy overlapping development work, also becomes that data are trans-regional, and inter-trade, trans-departmental circulation is altogether
The obstacle enjoyed.
Then, unified data distribution platform is established, the reticular structure of original system joint mode is become star structure,
Guarantee only to dock primary joint mode with me.To greatly reduce the workload in system docking, and become a kind of trend.For
Meet the needs of this business scenario, building unified service management platform becomes inevitable, unified service call gateway is constructed,
It realizes the record to gateway requests log, retrieval, polymerization and analysis, becomes the important means for ensureing service invariant operation.
For comprehensive retrieval, the service call situation of monitoring and analysis gateway requests data finds system fortune in time
Potential problem in row, the value of mining data keep the stabilization of system operation.The invention proposes one kind to be based on
The gateway log recording and search method of Elasticsearch.
Summary of the invention
In order to compensate for the shortcomings of the prior art, the present invention provides it is a kind of be simple and efficient based on Elasticsearch's
Gateway log recording and search method.
The present invention is achieved through the following technical solutions:
A kind of gateway log recording and search method, it is characterised in that: firstly, comprehensive record service call mistake
The information generated in journey, reasonable definition is log-structured, lays the foundation for log index；Then log is indexed and carries out reasonable definition
And cutting, realization rationally indexes daily record data, so as to Accurate Analysis log information；Last creation analysis model, to day
Will record analyzes and counts, to realize the log recording to the behavior for sending request data by gateway.
The reasonable definition is log-structured to be referred to for comprehensive record interface calling process log, service call it
Preceding clear log recording content defines log information.
Log recording is the process continued to increase, and the mode of clear log storage, institute are needed before record log
It states and indexes that carry out reasonable definition and cutting and assisted defining log with template version and store using the date as cutting foundation to log
Rule.
Described analyze and count to log recording is to call situation with web form monitoring service, in service invocation procedure
In, with the generation of log, gateway service situation is monitored in real time by visual interface.
Gateway log recording of the present invention and search method are defined the full-range log of service call, in line with
The principle of information redundancy records the full spectrum information of service call.
Gateway log recording of the present invention and search method realize the hair generated to log by Kafka message queue
Send with consumer record process, guarantee the completion of log recording；Rationally estimate the order of magnitude that log generates, defines log recording
Index templates rely on the real-time of Elasticsearch, stability and ability extending transversely, cut to log according to monthly
Point, realize the reasonable storage to daily record data.
Gateway log recording of the present invention and search method reasonably model daily record data, construct abundant
Statistical model calls the case where servicing to carry out in real time accurately reaction gateway.
Gateway log recording of the present invention and search method, the specific steps are as follows:
(1) request data, simultaneously record log is sent in Kafka for gateway response, and consumption terminal pulls number in batches from Kafka
According to being sent in Elasticsearch and local data base and save；
(2) data give over to backup in local data base, and data are based on passing through unified indices modulo in Elasticsearch
Plate carries out cutting index with date (yyyyMM) and saves, the storage of reasonable distribution data, thus realize unified data retrieval, number
According to analysis.
The beneficial effects of the present invention are: the gateway log recording and search method, by Kafka,
The big datas technology such as Elasticsearch, comprehensive retrieval, the service call situation of monitoring and analysis gateway requests data,
The distributed recording for calling log to gateway is realized, and then realizes the real time monitoring of gateway service condition, can be sent out in time
Potential problem, the value of mining data keep the stabilization of system operation in existing system operation, improve in system use can
Control property.
Detailed description of the invention
Attached drawing 1 is gateway log recording of the present invention and search method schematic diagram.
Specific embodiment
In order to which technical problems, technical solutions and advantages to be solved are more clearly understood, tie below
Drawings and examples are closed, the present invention will be described in detail.It should be noted that specific embodiment described herein is only used
To explain the present invention, it is not intended to limit the present invention.
The gateway log recording and search method, firstly, the letter generated in comprehensive record service invocation procedure
Breath, reasonable definition is log-structured, lays the foundation for log index；Then log is indexed and carries out reasonable definition and cutting, realized
Daily record data is rationally indexed, so as to Accurate Analysis log information；Last creation analysis model, divides log recording
Analysis statistics, to realize the log recording to the behavior for sending request data by gateway.
The reasonable definition is log-structured to be referred to for comprehensive record interface calling process log, service call it
Preceding clear log recording content defines log information.
Log recording is the process continued to increase, and the mode of clear log storage, institute are needed before record log
It states and indexes that carry out reasonable definition and cutting and assisted defining log with template version and store using the date as cutting foundation to log
Rule.
Described analyze and count to log recording is to call situation with web form monitoring service, in service invocation procedure
In, with the generation of log, gateway service situation is monitored in real time by visual interface.
The gateway log recording and search method are defined the full-range log of service call, in line with information
The principle of redundancy records the full spectrum information of service call.
The gateway log recording and search method, by Kafka message queue, realize transmission that log is generated with
Consumer record process guarantees the completion of log recording；Rationally estimate the order of magnitude that log generates, defines the index of log recording
Template relies on the real-time of Elasticsearch, stability and ability extending transversely, real to log according to monthly carry out cutting
Now to the reasonable storage of daily record data.
The gateway log recording and search method reasonably model daily record data, construct statistics abundant
Model calls the case where servicing to carry out in real time accurately reaction gateway.
The gateway log recording and search method, the specific steps are as follows:
(1) request data, simultaneously record log is sent in Kafka for gateway response, and consumption terminal pulls number in batches from Kafka
According to being sent in Elasticsearch and local data base and save；
(2) data give over to backup in local data base, and data are based on passing through unified indices modulo in Elasticsearch
Plate carries out cutting index with date (yyyyMM) and saves, the storage of reasonable distribution data, thus realize unified data retrieval, number
According to analysis.
The gateway log recording and search method, by the big datas technology such as Kafka, Elasticsearch, Quan Fang
The retrieval of position, the service call situation of monitoring and analysis gateway requests data realize the distributed note that log is called to gateway
Record, and then the real time monitoring of gateway service condition is realized, problem potential in system operation, mining data can be found in time
Value, keep system operation stabilization, improve the controllability in system use.
Claims (8)
1. a kind of gateway log recording and search method, it is characterised in that: firstly, comprehensive record service invocation procedure
The information of middle generation, reasonable definition is log-structured, lays the foundation for log index；Then to log index carry out reasonable definition and
Cutting, realization rationally indexes daily record data, so as to Accurate Analysis log information；Last creation analysis model, to log
Record analyzes and counts, to realize the log recording to the behavior for sending request data by gateway.
2. gateway log recording according to claim 1 and search method, it is characterised in that: the reasonable definition day
Will structure refers to that for comprehensive record interface calling process log, the clear log recording content before service call is fixed
Adopted log information.
3. gateway log recording according to claim 1 and search method, it is characterised in that: log recording is one
The process continued to increase, needs the mode of clear log storage before record log, and described index to log carries out Rational Determination
Justice and cutting are using the date as cutting foundation, and auxiliary with template version defines log storage rule.
4. gateway log recording according to claim 1 and search method, it is characterised in that: described to log recording
Analyze and count is to call situation, with the generation of log, to pass through in service invocation procedure with web form monitoring service
Visual interface monitors gateway service situation in real time.
5. gateway log recording according to claim 2 and search method, it is characterised in that: flowed entirely to service call
The log of journey is defined, and in line with the principle of information redundancy, is recorded to the full spectrum information of service call.
6. gateway log recording according to claim 3 and search method, it is characterised in that: by Kafka message
Queue realizes the transmission and consumer record process generated to log, guarantees the completion of log recording；Rationally estimate log to generate
The order of magnitude, define the index templates of log recording, rely on the real-time of Elasticsearch, stability and energy extending transversely
Power realizes the reasonable storage to daily record data to log according to monthly carry out cutting.
7. gateway log recording according to claim 4 and search method, it is characterised in that: carried out to daily record data
Reasonable modeling, constructs statistical model abundant, calls the case where servicing to carry out in real time accurately reaction gateway.
8. the gateway log recording according to any one of claim 5,6 or 7 and search method, feature exist
In, comprising the following steps:
(1) request data, simultaneously record log is sent in Kafka for gateway response, consumption terminal batch pulling data from Kafka,
It is sent in Elasticsearch and local data base and saves；
(2) data give over to backup in local data base, in Elasticsearch data based on by unified index templates, with
Date carries out cutting index and saves, the storage of reasonable distribution data, thus realize unified data retrieval, data analysis.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910145409.5A CN109818805A (en) | 2019-02-27 | 2019-02-27 | A kind of gateway log recording and search method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910145409.5A CN109818805A (en) | 2019-02-27 | 2019-02-27 | A kind of gateway log recording and search method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109818805A true CN109818805A (en) | 2019-05-28 |
Family
ID=66607541
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910145409.5A Pending CN109818805A (en) | 2019-02-27 | 2019-02-27 | A kind of gateway log recording and search method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109818805A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111049673A (en) * | 2019-11-21 | 2020-04-21 | 山东健康医疗大数据有限公司 | Method and system for counting and monitoring API call in service gateway |
| CN111654410A (en) * | 2020-04-28 | 2020-09-11 | 长沙证通云计算有限公司 | Gateway request monitoring method, device, equipment and medium |
| CN112073408A (en) * | 2020-09-04 | 2020-12-11 | 南京星邺汇捷网络科技有限公司 | Monitoring method based on Syslog protocol and Kafka queue |
| CN113472808A (en) * | 2021-07-16 | 2021-10-01 | 浙江大华技术股份有限公司 | Log processing method and device, storage medium and electronic device |
| CN114116402A (en) * | 2021-11-30 | 2022-03-01 | 山东浪潮云缦智能科技有限公司 | Embedded linux-based loosely-coupled and high-expansibility non-buried point data acquisition method |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180307576A1 (en) * | 2017-04-21 | 2018-10-25 | Nec Laboratories America, Inc. | Field content based pattern generation for heterogeneous logs |
| CN108712329A (en) * | 2018-05-02 | 2018-10-26 | 山东汇贸电子口岸有限公司 | A kind of gateway and log recording retrieval device based on Elasticsearch |
| CN108874614A (en) * | 2017-05-11 | 2018-11-23 | 上海宏时数据系统有限公司 | A kind of big data log intelligent analysis system and method |
-
2019
- 2019-02-27 CN CN201910145409.5A patent/CN109818805A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180307576A1 (en) * | 2017-04-21 | 2018-10-25 | Nec Laboratories America, Inc. | Field content based pattern generation for heterogeneous logs |
| CN108874614A (en) * | 2017-05-11 | 2018-11-23 | 上海宏时数据系统有限公司 | A kind of big data log intelligent analysis system and method |
| CN108712329A (en) * | 2018-05-02 | 2018-10-26 | 山东汇贸电子口岸有限公司 | A kind of gateway and log recording retrieval device based on Elasticsearch |
Non-Patent Citations (1)
| Title |
|---|
| 许长福: "日志数据分析系统的设计与实现", 《中国优秀硕士学位论文全文数据库•信息科技辑》 * |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111049673A (en) * | 2019-11-21 | 2020-04-21 | 山东健康医疗大数据有限公司 | Method and system for counting and monitoring API call in service gateway |
| CN111654410A (en) * | 2020-04-28 | 2020-09-11 | 长沙证通云计算有限公司 | Gateway request monitoring method, device, equipment and medium |
| CN111654410B (en) * | 2020-04-28 | 2021-12-24 | 长沙证通云计算有限公司 | Gateway request monitoring method, device, equipment and medium |
| CN112073408A (en) * | 2020-09-04 | 2020-12-11 | 南京星邺汇捷网络科技有限公司 | Monitoring method based on Syslog protocol and Kafka queue |
| CN112073408B (en) * | 2020-09-04 | 2023-12-01 | 南京星邺汇捷网络科技有限公司 | Monitoring method based on Syslog protocol and Kafka queue |
| CN113472808A (en) * | 2021-07-16 | 2021-10-01 | 浙江大华技术股份有限公司 | Log processing method and device, storage medium and electronic device |
| CN114116402A (en) * | 2021-11-30 | 2022-03-01 | 山东浪潮云缦智能科技有限公司 | Embedded linux-based loosely-coupled and high-expansibility non-buried point data acquisition method |
| CN114116402B (en) * | 2021-11-30 | 2024-09-17 | 山东浪潮超高清视频产业有限公司 | Loosely coupled and high-expansibility buried point free data acquisition method based on embedded linux |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109818805A (en) | A kind of gateway log recording and search method | |
| US7412451B2 (en) | Method for persisting a schedule and database schema | |
| CN111045656A (en) | Method and system for constructing platform system infrastructure in government affair service | |
| CN109656963B (en) | Metadata acquisition method, apparatus, device and computer readable storage medium | |
| US6845392B2 (en) | Remote systems management via DBMS stored procedures and one communication line | |
| CN109344207B (en) | Big data platform of integrative frequency spectrum all over the sky based on big dipper scanning | |
| CN106951552A (en) | A kind of user behavior data processing method based on Hadoop | |
| CN112419129A (en) | Method and system for constructing platform system infrastructure in government affair service | |
| WO2023202497A1 (en) | Method for full-link tracking of transaction, and native distributed database | |
| CN104504014A (en) | Data processing method and device based on large data platform | |
| CN112883001A (en) | Data processing method, device and medium based on marketing and distribution through data visualization platform | |
| CN112100227A (en) | Big data processing method based on multilevel heterogeneous data storage | |
| CN103810203A (en) | Connection multiplexing method and connection multiplexing device for database management system | |
| CN111984301A (en) | Micro-service data management framework based on spring close and kubernets | |
| Sridhar et al. | Optimizing database load and extract for big data era | |
| CN115689788A (en) | Financial data analysis method | |
| Tang et al. | Research and application of improving the field service ability of electric power marketing measuring mobile operating based on cloud computing technology | |
| CN110928938B (en) | Interface middleware system | |
| Zhang | The enterprise personnel management system based on B/S Design | |
| CN110716976A (en) | Distributed data access and storage system based on two interactive scenes | |
| CN105550143A (en) | Interface system for acquiring working condition of information equipment | |
| Shi et al. | HEDC++: an extended histogram estimator for data in the cloud | |
| CN111459965B (en) | Information monitoring system for edge calculation | |
| Xiao et al. | The research of teradata data warehouse technology | |
| Zhao et al. | Architecture Design of CTC Log Module Based on Web Service |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190528 |