CN102750326A - Log management optimization method of cluster system based on downsizing strategy - Google Patents
Log management optimization method of cluster system based on downsizing strategy Download PDFInfo
- Publication number
- CN102750326A CN102750326A CN2012101721732A CN201210172173A CN102750326A CN 102750326 A CN102750326 A CN 102750326A CN 2012101721732 A CN2012101721732 A CN 2012101721732A CN 201210172173 A CN201210172173 A CN 201210172173A CN 102750326 A CN102750326 A CN 102750326A
- Authority
- CN
- China
- Prior art keywords
- log
- daily record
- strategy
- information
- optimization method
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a log management optimization method of a cluster system based on a downsizing strategy and belongs to the field of cluster memory management. A log concentration strategy, a log downsizing strategy, a log storage strategy and a log display strategy are included. The log management optimization method includes concentrating logs distributed on cluster nodes to a log server through the log concentration strategy, performing classification extraction on collected logs on the log sever with the log downsizing strategy, deleting redundant information to reduce log space, storing log analysis results with the log storage strategy, building log storage files and corresponding log database index tables, and extracting contents of the log database index tables to check log file information through a log check strategy. According to the log management optimization method, logs distributed on cluster nodes are concentrated, analyzed and summarized, log storage methods are optimized, and the log check strategy is provided, so that a system administrator can position error log information of cluster systems rapidly and check whole log information of a certain application stored in cluster.
Description
Technical field
The present invention relates to a kind of cluster storage administration field, specifically a kind of log management optimization method of the group system based on refinement strategy.
Background technology
In the epoch that the information explosion formula increases, march toward TB level even PB level of file size and data scale become reality, and the cluster node number has reached 64 node number of clusters, the severe challenge that management huge group system like this has become data center and faced.In time follow the tracks of the clustered node running status, accurately the location node error message becomes particularly important.But along with increasing of clustered node number, cluster system management become to become increasingly complex, accurately the error log information of the location set group node more and more difficult that becomes.
Cluster storage is through the storage mode with DATA DISTRIBUTION each node in the cluster, and single use interface and interface is provided, and makes the user unify to use and management to all data easily.The cluster storage administration is that the clustered node that constitutes group system is carried out Software deployment configuration and environmental monitoring; Make it reach the deployment requirements of group system; When clustered node breaks down, group system must be made a response rapidly, and the storage space that is distributed in this node is transferred to other nodes.Detection node failure exception information detects the cluster system resource transfer case and becomes particularly important.
Traditional blog management method is timing or transmitting system daily record in real time; Realized the concentration of transmissions of daily record, but daily record has not been analyzed and put in order, along with the node number increases; Application module increases; The log information amount is huge, and the ruuning situation of the whole group system of understanding that can not be overall can not navigate to error message fast.
Summary of the invention
Technical assignment of the present invention provides a kind ofly to be concentrated, analyzes and gather the daily record that is dispersed in each clustered node; Optimize log storing method; Provide the daily record Review Policies to make system operator can locate the error log information of group system fast, check a kind of log management optimization method of the group system based on refinement strategy of whole log information of certain application of cluster storage.
Technical assignment of the present invention is realized by following mode; Comprise daily record centralized policy, daily record refinement strategy, log store strategy and journal displaying strategy; The daily record that at first will be dispersed in each clustered node through the daily record centralized policy focuses on log server, and utilization daily record refinement strategy is to the log information of collecting extractions of classifying, deletion redundant information minimizing log space on log server; Utilization log store strategy is stored the log analysis result; Set up log store file and corresponding log database concordance list, extract log database concordance list content through the daily record Review Policies at last, check journal file information.
The daily record centralized policy is to send to the daily record of clustered node on the log server to manage concentratedly; When sending daily record, this module continues to use classic method; Clustered node sends to log server with system journal in real time; Log server is accepted daily record, and log analysis is carried out in the daily record of accepting handle.
The daily record refinement strategy is that analyzing and processing is carried out in the daily record of clustered node, carries out filtration treatment according to the application module of setting (file system, high available, monitoring, network etc.) sign, makes the daily record can classification and storage; Daily record to clustered node is identical is filtered, and takies log space to prevent same log information; Extract the daily record time, nodal information makes things convenient for log store and index.
The log store strategy is divided into journal file storage and log record database storing; The journal file storage is each clustered node journal file of classification and storage in the cluster log services; The log record database storing is that the daily record of respectively classifying that the daily record refinement strategy obtains is called index with time and node; Quote the log store file, set up the daily record concordance list, check to make things convenient for the user; The log store strategy also compresses daily record, takes up room to reduce journal file; The journal file term of validity is limited, and expired daily record does not need to preserve, so to expired journal file poll deletion.
The daily record Review Policies is in the web system, log viewer to be set; Reader is classified by the daily record type and is checked, the daily record concordance list in the correspondence database, and quote according to daily record URL in the concordance list; Obtain real-time log information, thus realize to the classification of log information with gather; The user selects the corresponding time, and Log Types just can be checked the log information of all clustered nodes of cluster storage application module.
The log management optimization method of a kind of group system based on refinement strategy of the present invention has the following advantages: the daily record to being dispersed in each clustered node is concentrated, is analyzed and gather; Optimize log storing method; Provide the daily record Review Policies to make system operator can locate the error log information of group system fast; Check the whole log information of certain application of cluster storage, thereby, have good value for applications.
Description of drawings
Below in conjunction with accompanying drawing the present invention is further specified.
Accompanying drawing 1 is a kind of group system log management Organization Chart of log management optimization method of the group system based on refinement strategy;
Accompanying drawing 2 is a kind of log information management structural drawing of log management optimization method of the group system based on refinement strategy.
Embodiment
Explanation at length below with reference to Figure of description and specific embodiment the log management optimization method of a kind of group system based on refinement strategy of the present invention being done.
Embodiment:
The log management optimization method of a kind of group system based on refinement strategy of the present invention; Its structure comprises daily record centralized policy, daily record refinement strategy, log store strategy and journal displaying strategy; The daily record that at first will be dispersed in each clustered node through the daily record centralized policy focuses on log server; Utilization daily record refinement strategy is to the log information of the collecting extraction of classifying on log server; The deletion redundant information reduces log space, and utilization log store strategy is stored the log analysis result, sets up log store file and corresponding log database concordance list; Extract log database concordance list content through the daily record Review Policies at last, check journal file information.
The daily record centralized policy is to send to the daily record of clustered node on the log server to manage concentratedly; When sending daily record, this module continues to use classic method; Clustered node sends to log server with system journal in real time; Log server is accepted daily record, and log analysis is carried out in the daily record of accepting handle.
The daily record refinement strategy is that analyzing and processing is carried out in the daily record of clustered node, carries out filtration treatment according to the application module of setting (file system, high available, monitoring, network etc.) sign, makes the daily record can classification and storage; Daily record to clustered node is identical is filtered, and takies log space to prevent same log information; Extract the daily record time, nodal information makes things convenient for log store and index.
The log store strategy is divided into journal file storage and log record database storing; The journal file storage is each clustered node journal file of classification and storage in the cluster log services; The log record database storing is that the daily record of respectively classifying that the daily record refinement strategy obtains is called index with time and node; Quote the log store file, set up the daily record concordance list, check to make things convenient for the user; The log store strategy also compresses daily record, takes up room to reduce journal file; The journal file term of validity is limited, and expired daily record does not need to preserve, so to expired journal file poll deletion.
The daily record Review Policies is in the web system, log viewer to be set; Reader is classified by the daily record type and is checked, the daily record concordance list in the correspondence database, and quote according to daily record URL in the concordance list; Obtain real-time log information, thus realize to the classification of log information with gather; The user selects the corresponding time, and Log Types just can be checked the log information of all clustered nodes of cluster storage application module.
Write the daily record concentration function, can adopt traditional SYSLOG technology, realize clustered node daily record transmission;
Write the log analysis function, confirm the application module key word, extraction is analyzed in each application module daily record, adopt the regular expression searching method to realize.
Write the log store database functions, set up the daily record concordance list, can adopt database realizations such as mysql, information extraction is stored in the database by indexed format;
Write the journal file storage function,, the file that daily record is concentrated is pressed the daily record classification of type according to the log analysis function;
Write daily record and check function, set up the web page, inquiry mechanism is set up in classification according to the daily record application module, is connected to the database index table, obtains the inquiry log information of wanting fast according to concordance list.
The log management optimization method of a kind of group system based on refinement strategy of the present invention except that the described technical characterictic of instructions, is the known technology of those skilled in the art.
Claims (5)
1. log management optimization method based on the group system of refinement strategy; It is characterized in that comprising daily record centralized policy, daily record refinement strategy, log store strategy and journal displaying strategy; The daily record that at first will be dispersed in each clustered node through the daily record centralized policy focuses on log server; Utilization daily record refinement strategy is to the log information of the collecting extraction of classifying on log server; The deletion redundant information reduces log space, and utilization log store strategy is stored the log analysis result, sets up log store file and corresponding log database concordance list; Extract log database concordance list content through the daily record Review Policies at last, check journal file information.
2. the log management optimization method of a kind of group system based on refinement strategy according to claim 1; It is characterized in that the daily record centralized policy is to send to the daily record of clustered node on the log server to manage concentratedly; When sending daily record, this module continues to use classic method; Clustered node sends to log server with system journal in real time, and log server is accepted daily record, and log analysis is carried out in the daily record of accepting handle.
3. the log management optimization method of a kind of group system based on refinement strategy according to claim 1; It is characterized in that the daily record refinement strategy is that analyzing and processing is carried out in the daily record of clustered node; Application module sign according to setting is carried out filtration treatment, makes the daily record can classification and storage; Daily record to clustered node is identical is filtered, and takies log space to prevent same log information; Extract the daily record time, nodal information makes things convenient for log store and index.
4. the log management optimization method of a kind of group system based on refinement strategy according to claim 1; It is characterized in that the log store strategy is divided into journal file storage and log record database storing; The journal file storage is each clustered node journal file of classification and storage in the cluster log services; The log record database storing is that the daily record of respectively classifying that the daily record refinement strategy obtains is called index with time and node; Quote the log store file, set up the daily record concordance list, check to make things convenient for the user; The log store strategy also compresses daily record, takes up room to reduce journal file; The journal file term of validity is limited, and expired daily record does not need to preserve, so to expired journal file poll deletion.
5. the log management optimization method of a kind of group system based on refinement strategy according to claim 1; It is characterized in that the daily record Review Policies is in the web system, log viewer to be set; Reader is classified by the daily record type and is checked, the daily record concordance list in the correspondence database, and quote according to daily record URL in the concordance list; Obtain real-time log information, thus realize to the classification of log information with gather; The user selects the corresponding time, and Log Types just can be checked the log information of all clustered nodes of cluster storage application module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2012101721732A CN102750326A (en) | 2012-05-30 | 2012-05-30 | Log management optimization method of cluster system based on downsizing strategy |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2012101721732A CN102750326A (en) | 2012-05-30 | 2012-05-30 | Log management optimization method of cluster system based on downsizing strategy |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN102750326A true CN102750326A (en) | 2012-10-24 |
Family
ID=47030511
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2012101721732A Pending CN102750326A (en) | 2012-05-30 | 2012-05-30 | Log management optimization method of cluster system based on downsizing strategy |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102750326A (en) |
Cited By (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103812691A (en) * | 2014-01-14 | 2014-05-21 | 浪潮电子信息产业股份有限公司 | Log statistical system and method for cluster load balancing system |
| CN104301360A (en) * | 2013-07-19 | 2015-01-21 | 阿里巴巴集团控股有限公司 | Method, log server and system for recording log data |
| CN104579777A (en) * | 2015-01-06 | 2015-04-29 | 浪潮软件股份有限公司 | Log management system and implementation method |
| CN105205167A (en) * | 2015-10-10 | 2015-12-30 | 国网信息通信产业集团有限公司 | Log data system |
| CN105260452A (en) * | 2015-10-12 | 2016-01-20 | 成都视达科信息技术有限公司 | System and method for collecting, searching, and analyzing offline log |
| CN105306261A (en) * | 2015-09-29 | 2016-02-03 | 北京奇艺世纪科技有限公司 | Method, device and system for collecting logs |
| CN105553716A (en) * | 2015-12-12 | 2016-05-04 | 天津南大通用数据技术股份有限公司 | Fetch and display system of each node log under large-scale cluster |
| CN105827522A (en) * | 2015-11-10 | 2016-08-03 | 广东亿迅科技有限公司 | Gateway equipment for processing log files |
| CN106130782A (en) * | 2016-07-19 | 2016-11-16 | 努比亚技术有限公司 | A kind of method and system obtaining server log |
| CN106407350A (en) * | 2016-09-05 | 2017-02-15 | 广州视睿电子科技有限公司 | Error log information filtering method and device |
| CN106656522A (en) * | 2015-10-28 | 2017-05-10 | 中国移动通信集团公司 | Data calculation method and system of cross-data center |
| CN106649044A (en) * | 2016-12-28 | 2017-05-10 | 深圳市深信服电子科技有限公司 | Log processing method, device and system based on container cloud system |
| CN107332720A (en) * | 2017-08-22 | 2017-11-07 | 河钢股份有限公司承德分公司 | A kind of device for collecting interchanger daily record |
| CN107423336A (en) * | 2017-04-27 | 2017-12-01 | 努比亚技术有限公司 | A kind of data processing method, device and computer-readable storage medium |
| CN107483238A (en) * | 2017-08-04 | 2017-12-15 | 郑州云海信息技术有限公司 | A kind of blog management method, cluster management node and system |
| CN107544894A (en) * | 2016-06-23 | 2018-01-05 | 中兴通讯股份有限公司 | The method, apparatus and server of a kind of log processing |
| CN110209518A (en) * | 2019-04-26 | 2019-09-06 | 福州慧校通教育信息技术有限公司 | A kind of multi-data source daily record data, which is concentrated, collects storage method and device |
| CN110297846A (en) * | 2019-05-28 | 2019-10-01 | 北京奇艺世纪科技有限公司 | A kind of log feature processing system, method, electronic equipment and storage medium |
| CN111008107A (en) * | 2019-11-30 | 2020-04-14 | 北京浪潮数据技术有限公司 | Big data cluster log storage method, device, equipment and storage medium |
| WO2021073499A1 (en) * | 2019-10-15 | 2021-04-22 | 上海御渡半导体科技有限公司 | Log-based message classification display structure and method |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030220998A1 (en) * | 1999-08-27 | 2003-11-27 | Raymond Byars Jennings | Server site restructuring |
| US20070088737A1 (en) * | 2005-10-18 | 2007-04-19 | Norihiko Kawakami | Storage system for managing a log of access |
| JP2008203939A (en) * | 2007-02-16 | 2008-09-04 | Nomura Research Institute Ltd | Log management device, log management method, program, and recording medium |
| CN102111285A (en) * | 2009-12-24 | 2011-06-29 | 北京云快线软件服务有限公司 | Method and system for managing log information |
| CN102411533A (en) * | 2011-08-08 | 2012-04-11 | 浪潮电子信息产业股份有限公司 | Log-management optimizing method for clustered storage system |
-
2012
- 2012-05-30 CN CN2012101721732A patent/CN102750326A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030220998A1 (en) * | 1999-08-27 | 2003-11-27 | Raymond Byars Jennings | Server site restructuring |
| US20070088737A1 (en) * | 2005-10-18 | 2007-04-19 | Norihiko Kawakami | Storage system for managing a log of access |
| JP2008203939A (en) * | 2007-02-16 | 2008-09-04 | Nomura Research Institute Ltd | Log management device, log management method, program, and recording medium |
| CN102111285A (en) * | 2009-12-24 | 2011-06-29 | 北京云快线软件服务有限公司 | Method and system for managing log information |
| CN102411533A (en) * | 2011-08-08 | 2012-04-11 | 浪潮电子信息产业股份有限公司 | Log-management optimizing method for clustered storage system |
Cited By (23)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104301360A (en) * | 2013-07-19 | 2015-01-21 | 阿里巴巴集团控股有限公司 | Method, log server and system for recording log data |
| CN104301360B (en) * | 2013-07-19 | 2019-03-12 | 阿里巴巴集团控股有限公司 | A kind of method of logdata record, log server and system |
| CN103812691A (en) * | 2014-01-14 | 2014-05-21 | 浪潮电子信息产业股份有限公司 | Log statistical system and method for cluster load balancing system |
| CN104579777A (en) * | 2015-01-06 | 2015-04-29 | 浪潮软件股份有限公司 | Log management system and implementation method |
| CN105306261A (en) * | 2015-09-29 | 2016-02-03 | 北京奇艺世纪科技有限公司 | Method, device and system for collecting logs |
| CN105205167A (en) * | 2015-10-10 | 2015-12-30 | 国网信息通信产业集团有限公司 | Log data system |
| CN105260452A (en) * | 2015-10-12 | 2016-01-20 | 成都视达科信息技术有限公司 | System and method for collecting, searching, and analyzing offline log |
| CN106656522A (en) * | 2015-10-28 | 2017-05-10 | 中国移动通信集团公司 | Data calculation method and system of cross-data center |
| CN105827522A (en) * | 2015-11-10 | 2016-08-03 | 广东亿迅科技有限公司 | Gateway equipment for processing log files |
| CN105553716A (en) * | 2015-12-12 | 2016-05-04 | 天津南大通用数据技术股份有限公司 | Fetch and display system of each node log under large-scale cluster |
| CN107544894A (en) * | 2016-06-23 | 2018-01-05 | 中兴通讯股份有限公司 | The method, apparatus and server of a kind of log processing |
| CN106130782A (en) * | 2016-07-19 | 2016-11-16 | 努比亚技术有限公司 | A kind of method and system obtaining server log |
| CN106407350A (en) * | 2016-09-05 | 2017-02-15 | 广州视睿电子科技有限公司 | Error log information filtering method and device |
| CN106649044A (en) * | 2016-12-28 | 2017-05-10 | 深圳市深信服电子科技有限公司 | Log processing method, device and system based on container cloud system |
| CN107423336B (en) * | 2017-04-27 | 2021-01-15 | 努比亚技术有限公司 | Data processing method and device and computer storage medium |
| CN107423336A (en) * | 2017-04-27 | 2017-12-01 | 努比亚技术有限公司 | A kind of data processing method, device and computer-readable storage medium |
| CN107483238A (en) * | 2017-08-04 | 2017-12-15 | 郑州云海信息技术有限公司 | A kind of blog management method, cluster management node and system |
| CN107332720A (en) * | 2017-08-22 | 2017-11-07 | 河钢股份有限公司承德分公司 | A kind of device for collecting interchanger daily record |
| CN110209518A (en) * | 2019-04-26 | 2019-09-06 | 福州慧校通教育信息技术有限公司 | A kind of multi-data source daily record data, which is concentrated, collects storage method and device |
| CN110297846A (en) * | 2019-05-28 | 2019-10-01 | 北京奇艺世纪科技有限公司 | A kind of log feature processing system, method, electronic equipment and storage medium |
| CN110297846B (en) * | 2019-05-28 | 2021-08-20 | 北京奇艺世纪科技有限公司 | Log feature processing system, method, electronic equipment and storage medium |
| WO2021073499A1 (en) * | 2019-10-15 | 2021-04-22 | 上海御渡半导体科技有限公司 | Log-based message classification display structure and method |
| CN111008107A (en) * | 2019-11-30 | 2020-04-14 | 北京浪潮数据技术有限公司 | Big data cluster log storage method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102750326A (en) | Log management optimization method of cluster system based on downsizing strategy | |
| CN102411533A (en) | Log-management optimizing method for clustered storage system | |
| CN109254982A (en) | A kind of stream data processing method, system, device and computer readable storage medium | |
| CN105824744A (en) | Real-time log collection and analysis method on basis of B2B (Business to Business) platform | |
| CN103064933B (en) | Data query method and system | |
| CN100596353C (en) | Method and system for providing log service | |
| CN101576915B (en) | Distributed B+ tree index system and building method | |
| CN111241078A (en) | Data analysis system, data analysis method and device | |
| CN108509326B (en) | Service state statistical method and system based on nginx log | |
| CN104125163B (en) | Data processing method and device and terminal | |
| CN107273267A (en) | Log analysis method based on elastic components | |
| CN109918349A (en) | Log processing method, device, storage medium and electronic device | |
| CN108268565B (en) | Method and system for processing user browsing behavior data based on data warehouse | |
| CN102780726A (en) | Log analysis method and log analysis system based on WEB platform | |
| CN108509437A (en) | A kind of ElasticSearch inquiries accelerated method | |
| CA2656024A1 (en) | Data processing over very large databases | |
| CN104933114A (en) | Mass log management cloud platform | |
| CN108228322B (en) | Distributed link tracking and analyzing method, server and global scheduler | |
| CN103678293A (en) | Data storage method and device | |
| CN104765823A (en) | Method and device for collecting website data | |
| CN104462096A (en) | Public opinion monitoring and analysis method and device | |
| Murugesan et al. | Audit log management in MongoDB | |
| CN110572364A (en) | Method for realizing threat alarm in virtual environment | |
| CN103714144A (en) | Device and method for information retrieval | |
| CN109446167A (en) | A kind of storage of daily record data, extracting method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20121024 |