CN106202509A - A kind of processing method of log information - Google Patents

A kind of processing method of log information Download PDF

Info

Publication number
CN106202509A
CN106202509A CN201610577827.8A CN201610577827A CN106202509A CN 106202509 A CN106202509 A CN 106202509A CN 201610577827 A CN201610577827 A CN 201610577827A CN 106202509 A CN106202509 A CN 106202509A
Authority
CN
China
Prior art keywords
log information
log
alarm
time
server
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201610577827.8A
Other languages
Chinese (zh)
Inventor
黎健生
梁远鸿
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Liuzhou Longhui Science & Technology Co Ltd
Original Assignee
Liuzhou Longhui Science & Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Liuzhou Longhui Science & Technology Co Ltd filed Critical Liuzhou Longhui Science & Technology Co Ltd
Priority to CN201610577827.8A priority Critical patent/CN106202509A/en
Publication of CN106202509A publication Critical patent/CN106202509A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/245Query processing
    • G06F16/2458Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
    • G06F16/2462Approximate or statistical queries
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/14Details of searching files based on file metadata
    • G06F16/148File search processing
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/18File system types
    • G06F16/1805Append-only file systems, e.g. using logs or journals to store data
    • G06F16/1815Journaling file systems
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/28Databases characterised by their database models, e.g. relational or object models
    • G06F16/284Relational databases
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90Details of database functions independent of the retrieved data types
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90Details of database functions independent of the retrieved data types
    • G06F16/903Querying

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Computational Linguistics (AREA)
  • Probability & Statistics with Applications (AREA)
  • Fuzzy Systems (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Library & Information Science (AREA)
  • Debugging And Monitoring (AREA)

Abstract

nullThe invention discloses the processing method of a kind of log information,The newly-increased log information collected is filtered by client,The transmission bandwidth shared by invalid log information when transmitting log information can be reduced,Log information high for real-time is sent to server in real time,Log information low for real-time is delayed and is sent to server,Can effectively shorten the cycle that log information gathers,Alarm log information is timely transmitted to alarm treatment device by log server,Disclosure satisfy that the requirement that the warning information in log information is exported in time,And non-alarm log information is respectively stored in different storage positions,The classification of log information is corresponding with the class of operation carrying out operating for log information,Make journalizing platform during log information is operated,The storage position of log information can be judged according to class of operation,Can fast and effeciently shorten the process time that daily record data is operated、Improve treatment effeciency.

Description

Log information processing method
Technical Field
The invention relates to a data processing technology, in particular to a log information processing method.
Background
For the operation management system of the service support network, as the complexity and diversity of the support network and the service are gradually strengthened, the granularity of service monitoring is increasingly finer, so that the generated service log information volume is more and more, and particularly, the data volume of the service log is larger. A communication operator may have more than 100G of service logs per day, and the service logs are distributed over several tens of servers. Meanwhile, the service monitoring center has higher and higher requirements on the real-time performance of the alarm information. How to quickly complete the collection, extraction, processing, storage and efficient query of the logs becomes the key point of the operation management system of the service support network.
At present, there are two main ways to realize the collection method for log information:
the acquisition method comprises the following steps: a processing program on a centralized processing server (or called server) collects log information distributed on each log server through Telnet (namely remote control command collection), or collects the log information to the centralized processing server through an agent program on the log server; the processing program on the centralized processing server performs the refining processing of the log according to the log refining rule or the refining logic.
The second acquisition method comprises the following steps: each log server (or called client) purifies the log data, wherein the purification comprises the operation processes of extraction, combination, storage and the like of the log information, the purified log data is uploaded to the centralized processing server, and the centralized processing server combines the received log data files into one file according to a time period and refines the file.
The prior art addresses the storage and operation of log information in two ways:
the storage and operation method comprises the following steps: after the log information is refined, keeping the log information in a disk relational database according to required elements, performing statistical analysis and alarm information filtering and collection through SQL statements, and providing external query through standard SQL;
a second storage and operation method: after the log information is abstracted, the log information is saved in a file, a related general interface is provided, and external statistical processing service is provided through scanning the whole amount of the file.
It can be seen from the above acquisition method that the existing log information acquisition method has the following defects:
the acquisition method has obvious bottleneck of log processing performance, the log information on each log server acquired by the centralized processing server comprises a large amount of invalid log information, and the transmission of the invalid log information occupies a large amount of network transmission bandwidth, so that the acquisition speed and the acquisition period of the log information are greatly influenced; more importantly, as the service logs reach a certain scale, the processing pressure of invalid log information on the centralized server becomes more obvious, so that the processing logic on the centralized processing server becomes more and more complex, and the timeliness of processing the log information and outputting alarm information in the log information cannot be guaranteed;
although the log server is firstly used for purifying the log, the purification treatment comprises the operation processes of extraction, combination, storage and the like, so that the purification treatment prolongs the period of log information acquisition, and the acquisition speed of the log information is greatly influenced; and in the subsequent storage and query processing of the log information by the centralized server, the timeliness of the alarm information in the output log information cannot be guaranteed.
As can be seen from the above storage query method, the existing storage operation method has the following disadvantages:
the storage and operation method comprises the following steps: the refined log information is stored in a disk relational database (such as Oracle), massive data in the refined log information is retrieved and inquired through SQL statements, and for a complex data analysis scene, multiple large tables are often associated and inquired after Cartesian sets are made, so that the time for waiting for response is too long, and the requirement for timely outputting alarm information in the log information cannot be met;
a second storage and operation method: the processed data is saved in files, typically organized according to time period. Therefore, subsequent analysis processing generally scans the whole file, obviously, the efficiency of query and analysis processing is low, and the requirement for timely outputting the alarm information in the log information cannot be met.
Therefore, the existing acquisition processing and storage query processing method for massive log information has the problems of too long acquisition period and too much transmission bandwidth occupied by invalid log information, and the problems of long processing time and low efficiency of query and statistical processing of massive log information and incapability of meeting the requirement of timely outputting alarm information in the log information.
Disclosure of Invention
The embodiment of the invention provides a method for processing massive log information, which comprises the following steps: the log server receives log information from a log client; according to a preset alarm rule, sending alarm log information to an alarm processing device under the condition that the received log information is judged to be the alarm log information; according to the type of the non-alarm log information and the time for receiving the log information, the non-alarm log information is respectively stored in a relational database, a memory database of a log server or a file system, wherein the type of the log information corresponds to the type of operation for operating the log information, the log information stored in the relational database provides a data basis for statistical operation, the log information stored in the memory database of the log server provides a data basis for real-time query operation, and the log information stored in the file system provides a data basis for non-real-time query operation.
According to the technical scheme of the embodiment of the invention, in the processing method for the massive log information acquired by the log information provided by the embodiment of the invention, the log client filters the newly-added acquired log information, can filter invalid log information in the acquired log information, reduces transmission bandwidth occupied by the invalid log information when the log information is transmitted, carries real-time identification in the filtered log information according to the corresponding relation between the preset log information and the real-time level, transmits the log information with high real-time to the server in real time in the process of transmitting the filtered log information to the server, delays the log information with low real-time to the server, and can effectively shorten the period of acquiring the log information; in the processing method for massive log information stored in log information provided by the embodiment of the invention, the log server sends the alarm log information to the alarm processing device in time, can quickly send the alarm information to the alarm processing device, can meet the requirement of outputting the alarm information in the log information in time, respectively stores the non-alarm log information to a relational database, a memory database or a file system of the server according to the type of the non-alarm log information and the time for receiving the log information, and the type of the log information corresponds to the operation type for operating the log information, so that a log operation platform can judge the storage position of the log information according to the operation type in the process of operating the log information, and the reading speed of data in the relational database and the memory database is higher, the method can quickly and effectively shorten the processing time of log data operation and improve the processing efficiency, thereby solving the problems that the acquisition period is too long, invalid log information occupies too much transmission bandwidth, the operation processing time of the mass log information is long, the efficiency is low, and the requirement for timely outputting the alarm information in the log information cannot be met in the acquisition processing and storage query processing method of the mass log information in the prior art.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
Drawings
Fig. 1 is a flowchart of a method for processing massive log information according to an embodiment of the present invention;
Detailed Description
The embodiments of the present invention will be described in conjunction with the accompanying drawings, and it should be understood that the embodiments described herein are only for the purpose of illustrating and explaining the present invention, and are not intended to limit the present invention.
Fig. 1 shows a work flow diagram of a processing method for massive log information, which is provided by an embodiment of the present invention and is applied to a log server to store log information from a log client, and the method includes:
step 101, a log server receives log information from a log client;
specifically, a log server receives log information sent in real time from a log client;
after receiving a request from the log client, returning a waiting response to the log client under the condition that the data volume of the log information currently received by the log server is greater than or equal to a preset data volume threshold; returning a sending response to the log client under the condition that the data volume of the log information currently received by the log server is smaller than a preset data volume threshold;
furthermore, the log server also establishes an index table for the received log information, wherein the index table at least comprises the identification of the log information in the log information, the generation time of the log information, the end time of the log information and the category of the log information, and the index table is used for providing indexes for query conditions in query operation, namely any keyword of the log information in the index table can be used as the query conditions in the query operation; furthermore, the log server can also establish an index table for log information containing preset keywords or keyword values;
102, according to a preset alarm rule, sending alarm log information to an alarm processing device under the condition that the received log information is judged to be the alarm log information;
the preset alarm rule specifically comprises the following steps: the log information including the predetermined key value is alarm log information; or, a time length value between the end time and the generation time of the log information included in the log information is greater than or equal to a predetermined time length value;
103, respectively storing the non-alarm log information into a relational database, a memory database of the log server or a file system according to the type of the non-alarm log information and the time for receiving the log information;
specifically, under the condition that the category of the non-alarm log information is statistical log information, storing the log information into a relational database; under the condition that the type of the non-alarm log information is non-statistical log information, storing the log information into a memory database of the log server within a preset storage period from the time of receiving the log information, and storing the log information into the file system after the log information is stored in the memory database of the log server for more than the preset storage period;
it can be seen that the log information stored in the relational database provides a data basis for statistical operations, the log information stored in the memory database of the log server provides a data basis for real-time query operations, and the log information stored in the file system provides a data basis for non-real-time query operations.
According to the method shown in fig. 1, the log server sends the alarm log information to the alarm processing device in time when recognizing that the log information is the alarm log information, can quickly send the alarm information to the alarm processing device, can meet the requirement of outputting the alarm information in the log information in time, respectively stores the non-alarm log information into a relational database, a memory database of the server or a file system according to the type of the non-alarm log information and the time for receiving the log information, and the type of the log information corresponds to the operation type for operating the log information, the reading speed of data in the relational database and the memory database is high, so that a quick and effective query path can be provided for the operation processing of the log information, and the problems of low query operation speed, low operation speed of mass log information, low data reading speed of mass log information in the prior art, and the like are solved, The efficiency is low.
It will be understood by those skilled in the art that all or part of the steps carried by the method for implementing the above embodiments may be implemented by hardware related to instructions of a program, which may be stored in a computer readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.
In addition, each functional unit in the embodiments of the present invention may be integrated into one processing module, or each unit may exist alone physically, or two or more units are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may also be stored in a computer readable storage medium.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
While embodiments of the invention have been shown and described, it will be understood by those of ordinary skill in the art that: various changes, modifications, substitutions and alterations can be made to the embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims (5)

1. A method for processing massive log information is characterized by comprising the following steps:
the log server receives log information from a log client;
according to a preset alarm rule, sending alarm log information to an alarm processing device under the condition that the received log information is judged to be the alarm log information;
according to the type of the non-alarm log information and the time for receiving the log information, the non-alarm log information is respectively stored in a relational database, a memory database of the log server or a file system, wherein the type of the log information corresponds to the type of operation for operating the log information, the log information stored in the relational database provides a data basis for statistical operation, the log information stored in the memory database of the log server provides a data basis for real-time query operation, and the log information stored in the file system provides a data basis for non-real-time query operation.
2. The method according to claim 1, wherein the log server receives log information from the log client, and specifically comprises:
the log server receives log information sent by the log client in real time;
after receiving a request from the log client, returning a waiting response to the log client under the condition that the data volume of the log information currently received by the log server is greater than or equal to a preset data volume threshold; and returning a sending response to the log client under the condition that the data volume of the log information currently received by the log server is smaller than the preset data volume threshold.
3. The method of claim 1, further comprising:
and establishing an index table for the received log information, wherein the index table at least comprises the identification of the log information in the log information, the generation time of the log information, the end time of the log information and the category of the log information, and the index table is used for providing indexes for query conditions in query operation.
4. The method according to claim 1, wherein the predetermined alarm rule specifically comprises:
the log information including the predetermined key value is alarm log information; or,
a time length value between the end time and the generation time of the log information included in the log information is greater than or equal to a predetermined time length value.
5. The method according to claim 1, wherein the step of storing the non-alarm log information in a relational database, a memory database of the log server, or a file system according to the type of the non-alarm log information and the time of receiving the log information comprises:
under the condition that the type of the non-alarm log information is statistical log information, storing the log information into a relational database;
and if the type of the non-alarm log information is non-statistical log information, storing the log information into the memory database of the log server within a preset storage period from the time when the log information is received, and storing the log information which is stored for exceeding the preset storage period into the file system after the log information in the memory database of the log server is stored for exceeding the preset storage period.
CN201610577827.8A 2016-07-21 2016-07-21 A kind of processing method of log information Pending CN106202509A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201610577827.8A CN106202509A (en) 2016-07-21 2016-07-21 A kind of processing method of log information

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201610577827.8A CN106202509A (en) 2016-07-21 2016-07-21 A kind of processing method of log information

Publications (1)

Publication Number Publication Date
CN106202509A true CN106202509A (en) 2016-12-07

Family

ID=57492024

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201610577827.8A Pending CN106202509A (en) 2016-07-21 2016-07-21 A kind of processing method of log information

Country Status (1)

Country Link
CN (1) CN106202509A (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107422986A (en) * 2017-05-10 2017-12-01 郑州云海信息技术有限公司 A kind of control device and method of cloud storage system concurrent reading and writing request
CN107612740A (en) * 2017-09-30 2018-01-19 武汉光谷信息技术股份有限公司 A kind of daily record monitoring system and method under distributed environment
CN107678922A (en) * 2017-09-29 2018-02-09 郑州云海信息技术有限公司 Time-consuming management method and relevant apparatus applied to distributed file system
CN112434063A (en) * 2020-11-03 2021-03-02 中国南方电网有限责任公司 Monitoring data processing method based on time sequence database

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103399887A (en) * 2013-07-19 2013-11-20 蓝盾信息安全技术股份有限公司 Query and statistical analysis system for mass logs
CN103942210A (en) * 2013-01-21 2014-07-23 中国移动通信集团上海有限公司 Processing method, device and system of mass log information
CN104391781A (en) * 2014-10-24 2015-03-04 苏州阔地网络科技有限公司 Processing method and system for log information
CN105005528A (en) * 2015-06-26 2015-10-28 浪潮(北京)电子信息产业有限公司 Log information extraction method and apparatus

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103942210A (en) * 2013-01-21 2014-07-23 中国移动通信集团上海有限公司 Processing method, device and system of mass log information
CN103399887A (en) * 2013-07-19 2013-11-20 蓝盾信息安全技术股份有限公司 Query and statistical analysis system for mass logs
CN104391781A (en) * 2014-10-24 2015-03-04 苏州阔地网络科技有限公司 Processing method and system for log information
CN105005528A (en) * 2015-06-26 2015-10-28 浪潮(北京)电子信息产业有限公司 Log information extraction method and apparatus

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107422986A (en) * 2017-05-10 2017-12-01 郑州云海信息技术有限公司 A kind of control device and method of cloud storage system concurrent reading and writing request
CN107678922A (en) * 2017-09-29 2018-02-09 郑州云海信息技术有限公司 Time-consuming management method and relevant apparatus applied to distributed file system
CN107612740A (en) * 2017-09-30 2018-01-19 武汉光谷信息技术股份有限公司 A kind of daily record monitoring system and method under distributed environment
CN112434063A (en) * 2020-11-03 2021-03-02 中国南方电网有限责任公司 Monitoring data processing method based on time sequence database

Similar Documents

Publication Publication Date Title
CN103942210B (en) Processing method, device and the system of massive logs information
CN106250287A (en) A kind of log information processing means
CN113360554B (en) Method and equipment for extracting, converting and loading ETL (extract transform load) data
CN106202509A (en) A kind of processing method of log information
CN107390650A (en) A kind of data collecting system based on Internet of Things and the data compression method based on the system
CN103731298A (en) Large-scale distributed network safety data acquisition method and system
CN102902752A (en) Method and system for monitoring log
CN106169959A (en) A kind of log processing device
CN103997532A (en) Agriculture internet-of-things edge middleware system
CN108334557B (en) Aggregated data analysis method and device, storage medium and electronic equipment
CN109542750A (en) Distributed information log system
CN103023693A (en) Behaviour log data management system and behaviour log data management method
CN113391973B (en) Internet of things cloud container log collection method and device
CN110413478A (en) A kind of method, equipment and medium monitoring log processing
CN108123840A (en) Log processing method and system
CN114238388A (en) Heterogeneous data collection and retrieval system based on multiple protocols
CN106484595A (en) A kind of event-handling method and device
CN114338746A (en) Analysis early warning method and system for data collection of Internet of things equipment
CN112965979A (en) User behavior analysis method and device and electronic equipment
Ferry et al. Towards a big data platform for managing machine generated data in the cloud
CN109800221A (en) A kind of mass data association relationship analysis method, apparatus and system
CN106250406A (en) A kind of log processing method
CN106227644A (en) A kind of magnanimity information processing device
CN106250405A (en) A kind of magnanimity information processing system
CN102722521A (en) Method and system for monitoring data comparison

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
CB02 Change of applicant information

Address after: No. 52 Kwun Tong Road in Liuzhou city of the Guangxi Zhuang Autonomous Region in 545005

Applicant after: LIUZHOU LONGHUI SCIENCE & TECHNOLOGY CO., LTD.

Address before: 545005 the Guangxi Zhuang Autonomous Region Liuzhou Liunan District City Station Road No. 94, a new era of commercial port logistics warehousing center No. 5 Floor 4 No. 022

Applicant before: LIUZHOU LONGHUI SCIENCE & TECHNOLOGY CO., LTD.

CB02 Change of applicant information
WD01 Invention patent application deemed withdrawn after publication

Application publication date: 20161207

WD01 Invention patent application deemed withdrawn after publication