CN104933114A - Mass log management cloud platform - Google Patents
Mass log management cloud platform Download PDFInfo
- Publication number
- CN104933114A CN104933114A CN201510305445.5A CN201510305445A CN104933114A CN 104933114 A CN104933114 A CN 104933114A CN 201510305445 A CN201510305445 A CN 201510305445A CN 104933114 A CN104933114 A CN 104933114A
- Authority
- CN
- China
- Prior art keywords
- log
- daily record
- cloud platform
- management cloud
- storage system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
Abstract
The invention discloses a mass log management cloud platform, which comprises a log collection system (01), a log processing system (02), a log indexing and storage system (03) and a log query and application system (04). The mass log management cloud platform is a cloud log management system which can carry out full-text indexing on a log to more quickly search and analyze the log; cloud storage is adopted to prevent the log from being limited to the volume of storage hardware; and the problem of log data loss caused by a single point of failure can be solved. A log message processing subsystem (021) is added in the log processing system (02) and is responsible for transmitting the received log to a real-time flow processing subsystem (022) so as to realize decoupling of the acquisition and processing of logs . If the log indexing and storage system fails, the log message processing subsystem (021) can automatically and temporarily stores a message in a hard disk in a persistence way, and therefore, a phenomenon that the log received by the system is lost due to the exception of the indexing and storage system can be avoided.
Description
Technical field
The present invention relates to computer application field, particularly relate to a kind of massive logs management cloud platform.
Background technology
Traditional Log Analysis System, normally with the daily record of the mode collecting device of this locality installation, before this by Log Sender on home server, be directly stored in database by analysis by analysis or not, then undertaken searching for and analyzing daily record data by administration interface.
Traditional Log Analysis System, when analyzing daily record, can run into following problem:
1., after daily record capacity increases, declining all appears in the daily record storage of system, inquiry, analytical performance, because do not have good horizontal extension ability, system performance is often limited to hardware performance.
2. daily record is not carried out to the ability of full-text index.
3. Single Point of Faliure.After the hardware storage device in system breaks down, data can be lost and are difficult to give for change.
Summary of the invention
For solving the problems of the technologies described above, the invention provides a kind of massive logs management cloud platform, helper applications developer or network O&M personnel check more easily and analyze daily record, thus improve the efficiency of development efficiency and problem analysis.
To achieve these goals, the present invention adopts following technical scheme.
A kind of massive logs management cloud platform, comprises result collection system, log processing system, daily record index and storage system, log query application system.
Described result collection system, for by log collection in platform, be the system of the massive logs polymerization of distributed, a reliable and High Availabitity.It supports the daily record of collecting the various protocols such as syslog, HTTP, Log4J, file, file change and form.
Described log processing system comprises log information processing subsystem and real-time streams processing subsystem.
Log information processing subsystem, be used for the collection of decoupling zero log information and log information analysis, store between logical relation, make system more flexibly, reliable.When storage system breaks down, the persistence mechanism of log information processing subsystem can ensure that daily record can not be lost.
Real-time streams processing subsystem, is used for real-time for log information to be distributed to each back-end processing system.
Described daily record index and storage system, comprise semi-structured storage system, full-text index system and destructuring storage system.
Semi-structured storage system, for storing, the structural data of inquiry log, being a memory mechanism flexibly, daily record can being split into self-defining field to preserve.And traditional relationship type storage mode must define field before user uses prerequisite.In addition, this system also has the ability that TB DBMS amount stores.
Full-text index system, can provide the full-text search of daily record, for system provides can the ability of real-time retrieval daily record in the daily record of TB level.It provides based on copying and the full-text index cluster of allocation methods.And synonym, near synonym, Chinese word segmentation ability are provided.
Destructuring storage system, backs up daily record and off-line analysis process.
Described log query application system is the application system of a distributed inquiry and analysis daily record.This system can with the various ways exhibiting collections such as form, histogram, Line Chart arrive by analysis after daily record, can close to real-time displaying log information, and alarm can be carried out according to the strategy formulated.
Beneficial effect of the present invention comprises:
Massive logs management cloud platform of the present invention is not local Log Administration System, but a high in the clouds Log Administration System, full-text index can be carried out to daily record, make search and analyze daily record quicker; High in the clouds is adopted to store the capacity being no longer confined to storage hardware; The daily record data loss problem because Single Point of Faliure causes can be solved.Increase log information processing subsystem in log processing system, it is responsible for the daily record of reception to be transmitted to real-time streams processing subsystem, realizes the collection of daily record and the decoupling zero of process.Time abnormal if there is daily record index or stocking system, log information processing subsystem can automatically interim by message duration in hard disk, can allow like this system acceptance to daily record can not lose because of the exception of index or stocking system.
Certainly, implement arbitrary product of the present invention might not need to reach above-described all advantages simultaneously.
Accompanying drawing explanation
Fig. 1 is the structural representation of massive logs of the present invention management cloud platform.
Wherein, 00, daily record place server; 01, result collection system; 02, log processing system; 021, log information processing subsystem; 022, real-time streams processing subsystem; 03, daily record index and storage system; 031, semi-structured storage system; 032, full-text index system; 033, destructuring storage system; 04, log query application system; 05, user.
Embodiment
Below in conjunction with accompanying drawing and embodiment, the invention will be further described.
As shown in Figure 1, a kind of massive logs management cloud platform, comprises result collection system 01, log processing system 02, daily record index and storage system 03, log query application system 04.
Daily record is issued result collection system 01 by the form of syslog by daily record place server 00.
Described result collection system 01, is in platform foremost, for by log collection in platform, being a massive logs paradigmatic system cluster, is distributed, a highly reliable result collection system.It supports the daily record of collecting the various protocols such as syslog, HTTP, Log4J, file, file change and form.
It has following characteristics:
A) high availability.Availability (availablity) refers to that in the fixed cycle, system failure runs T.T..Want the availability of raising system, just need the single-point of elimination system, improve the redundance of system.
B) high reliability.Reliability (reliability) refers in the transmitting procedure of data stream, ensures the reliable delivery of daily record.When one malfunctions, daily record can be sent on other nodes and can not lose.Log collection service provides the guaranteed reliability of three kinds of ranks, from respectively being to weak by force: end-to-end guarantee (end-to-end), receives data and first write on disk by daily record, when data transmit successfully, then deletes; If data send unsuccessfully, can resend; Local guarantee (Store on failure), as daily record take over party crash, writes this locality by daily record, after to be restored, continues to send; Ensureing (Best effort) without confirming, after Log Sender to take over party, can not confirm.
C) extensibility.Log collection service have employed three-tier architecture, and be respectively agent acquisition (agent), collect service (collector) and stores service (storage), every one deck all can horizontal extension.Wherein, all agent and collector are by master unified management, and this makes system easily monitor and safeguard, and master has allowed multiple, avoiding problems Single Point of Faliure problem.
D) holding load is balanced and fault-tolerant.
Described log processing system 02 comprises log information processing subsystem 021 and real-time streams processing subsystem 022.
Log information processing subsystem 021, be used for the collection of decoupling zero log information and log information analysis, store between logical relation, make system more flexibly, reliable.When storage system breaks down, the persistence mechanism of log information processing subsystem can ensure that daily record can not be lost.
Real-time streams processing subsystem 022, is used for real-time for log information to be distributed to each back-end processing system.
Described daily record index and storage system 03, comprise semi-structured storage system 031, full-text index system 032 and destructuring storage system 033.
Semi-structured storage system 031, for storing, the structural data of inquiry log, a memory mechanism flexibly, can split into self-defining field to preserve by daily record.And traditional relationship type storage mode must define field before user uses prerequisite.In addition, this system also has the ability that TB DBMS amount stores.
Full-text index system 032, can provide the full-text search of daily record, for system provides can the ability of real-time retrieval daily record in the daily record of TB level.It provides based on copying and the full-text index cluster of allocation methods.And synonym, near synonym, Chinese word segmentation ability are provided.
Destructuring storage system 033, backs up daily record and off-line analysis process.
Described log query application system 04 is the application system of a distributed inquiry and analysis daily record.This system can with the various ways exhibiting collections such as form, histogram, Line Chart arrive by analysis after daily record, can close to real-time displaying log information, and alarm can be carried out according to the strategy formulated.
User 05 is by log query application system 04 described in browser access.
By reference to the accompanying drawings the specific embodiment of the present invention is described although above-mentioned; but not limiting the scope of the invention; one of ordinary skill in the art should be understood that; on the basis of technical scheme of the present invention, those skilled in the art do not need to pay various amendment or distortion that creative work can make still within protection scope of the present invention.
Claims (9)
1. a massive logs management cloud platform, is characterized in that, comprise result collection system (01), log processing system (02), daily record index and storage system (03), log query application system (04);
Described result collection system (01), for by log collection in platform;
Described log processing system (02) comprises log information processing subsystem (021) and real-time streams processing subsystem (022); Log information processing subsystem (021), be used for the collection of decoupling zero log information and log information analysis, store between logical relation; Real-time streams processing subsystem (022), is used for real-time for log information to be distributed to each back-end processing system;
Described daily record index and storage system (03), comprise semi-structured storage system (031), full-text index system (032) and destructuring storage system (033); Semi-structured storage system (031), for storing, the structural data of inquiry log; Full-text index system (032), provides the full-text search of daily record; Destructuring storage system (033), backs up daily record and off-line analysis process;
Described log query application system (04), can exhibiting collection arrive by analysis after daily record, displaying log information that can be real-time, and according to formulate strategy carry out alarm.
2. massive logs management cloud platform as claimed in claim 1, is characterized in that, described result collection system (01) is the system that a massive logs that is distributed, reliable and High Availabitity is polymerized.
3. massive logs management cloud platform as claimed in claim 1 or 2, it is characterized in that, described result collection system (01) support collects syslog, HTTP, Log4J, file, the agreement of file change and the daily record of form.
4. massive logs management cloud platform as claimed in claim 1, it is characterized in that, when storage system breaks down, the persistence mechanism daily record of described log information processing subsystem (021) can not be lost.
5. massive logs management cloud platform as claimed in claim 1, it is characterized in that, daily record is split into self-defining field to preserve by described semi-structured storage system (031).
6. the massive logs management cloud platform as described in claim 1 or 5, it is characterized in that, described semi-structured storage system (031) has the ability that TB DBMS amount stores.
7. massive logs management cloud platform as claimed in claim 1, is characterized in that, described full-text index system (032), can real-time retrieval daily record in the daily record of TB level.
8. massive logs management cloud platform as claimed in claim 1, is characterized in that, described full-text index system (032), providing based on copying and the full-text index cluster of allocation methods, providing synonym, near synonym, Chinese word segmentation ability.
9. massive logs management cloud platform as claimed in claim 1, is characterized in that described log query application system (04) is the application system of a distributed inquiry and analysis daily record.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510305445.5A CN104933114A (en) | 2015-06-08 | 2015-06-08 | Mass log management cloud platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510305445.5A CN104933114A (en) | 2015-06-08 | 2015-06-08 | Mass log management cloud platform |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104933114A true CN104933114A (en) | 2015-09-23 |
Family
ID=54120281
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510305445.5A Pending CN104933114A (en) | 2015-06-08 | 2015-06-08 | Mass log management cloud platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104933114A (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105630869A (en) * | 2015-12-15 | 2016-06-01 | 北京奇虎科技有限公司 | Voice data storage method and device |
| CN106227644A (en) * | 2016-07-21 | 2016-12-14 | 柳州龙辉科技有限公司 | A kind of magnanimity information processing device |
| CN106227797A (en) * | 2016-07-21 | 2016-12-14 | 柳州龙辉科技有限公司 | A kind of processing method of massive logs information |
| CN106250287A (en) * | 2016-07-21 | 2016-12-21 | 柳州龙辉科技有限公司 | A kind of log information processing means |
| CN106250406A (en) * | 2016-07-21 | 2016-12-21 | 柳州龙辉科技有限公司 | A kind of log processing method |
| CN106844497A (en) * | 2016-12-26 | 2017-06-13 | 努比亚技术有限公司 | The check device and method of a kind of database code |
| CN108959445A (en) * | 2018-06-13 | 2018-12-07 | 云南电网有限责任公司信息中心 | Distributed information log processing method and processing device |
| CN109088782A (en) * | 2018-11-01 | 2018-12-25 | 郑州云海信息技术有限公司 | The log collecting method and device of distributed system |
| CN109992417A (en) * | 2019-03-20 | 2019-07-09 | 跬云(上海)信息科技有限公司 | Precomputation OLAP system and implementation method |
| US10445196B2 (en) | 2017-01-06 | 2019-10-15 | Microsoft Technology Licensing, Llc | Integrated application issue detection and correction control |
| CN111045898A (en) * | 2019-12-22 | 2020-04-21 | 北京浪潮数据技术有限公司 | Log collection method, device and equipment for multi-stage subsystem and readable storage medium |
| CN113515494A (en) * | 2020-04-09 | 2021-10-19 | 中国移动通信集团广东有限公司 | Database processing method based on distributed file system and electronic equipment |
| CN117494146B (en) * | 2023-12-29 | 2024-04-26 | 山东街景智能制造科技股份有限公司 | Model database management system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102279891A (en) * | 2011-09-02 | 2011-12-14 | 深圳中兴网信科技有限公司 | Retrieval method, device and system for concurrently searching information technology (IT) logs |
| CN102411533A (en) * | 2011-08-08 | 2012-04-11 | 浪潮电子信息产业股份有限公司 | Log-management optimizing method for clustered storage system |
| CN103177116A (en) * | 2013-04-08 | 2013-06-26 | 国电南瑞科技股份有限公司 | Distributed log handling and inquiring method based on two-stage index |
| CN103942210A (en) * | 2013-01-21 | 2014-07-23 | 中国移动通信集团上海有限公司 | Processing method, device and system of mass log information |
-
2015
- 2015-06-08 CN CN201510305445.5A patent/CN104933114A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102411533A (en) * | 2011-08-08 | 2012-04-11 | 浪潮电子信息产业股份有限公司 | Log-management optimizing method for clustered storage system |
| CN102279891A (en) * | 2011-09-02 | 2011-12-14 | 深圳中兴网信科技有限公司 | Retrieval method, device and system for concurrently searching information technology (IT) logs |
| CN103942210A (en) * | 2013-01-21 | 2014-07-23 | 中国移动通信集团上海有限公司 | Processing method, device and system of mass log information |
| CN103177116A (en) * | 2013-04-08 | 2013-06-26 | 国电南瑞科技股份有限公司 | Distributed log handling and inquiring method based on two-stage index |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105630869B (en) * | 2015-12-15 | 2019-02-05 | 北京奇虎科技有限公司 | A kind of storage method and device of voice data |
| CN105630869A (en) * | 2015-12-15 | 2016-06-01 | 北京奇虎科技有限公司 | Voice data storage method and device |
| CN106227644A (en) * | 2016-07-21 | 2016-12-14 | 柳州龙辉科技有限公司 | A kind of magnanimity information processing device |
| CN106227797A (en) * | 2016-07-21 | 2016-12-14 | 柳州龙辉科技有限公司 | A kind of processing method of massive logs information |
| CN106250287A (en) * | 2016-07-21 | 2016-12-21 | 柳州龙辉科技有限公司 | A kind of log information processing means |
| CN106250406A (en) * | 2016-07-21 | 2016-12-21 | 柳州龙辉科技有限公司 | A kind of log processing method |
| CN106844497A (en) * | 2016-12-26 | 2017-06-13 | 努比亚技术有限公司 | The check device and method of a kind of database code |
| US10445196B2 (en) | 2017-01-06 | 2019-10-15 | Microsoft Technology Licensing, Llc | Integrated application issue detection and correction control |
| CN108959445A (en) * | 2018-06-13 | 2018-12-07 | 云南电网有限责任公司信息中心 | Distributed information log processing method and processing device |
| CN109088782A (en) * | 2018-11-01 | 2018-12-25 | 郑州云海信息技术有限公司 | The log collecting method and device of distributed system |
| CN109992417A (en) * | 2019-03-20 | 2019-07-09 | 跬云(上海)信息科技有限公司 | Precomputation OLAP system and implementation method |
| CN111045898A (en) * | 2019-12-22 | 2020-04-21 | 北京浪潮数据技术有限公司 | Log collection method, device and equipment for multi-stage subsystem and readable storage medium |
| CN113515494A (en) * | 2020-04-09 | 2021-10-19 | 中国移动通信集团广东有限公司 | Database processing method based on distributed file system and electronic equipment |
| CN113515494B (en) * | 2020-04-09 | 2024-03-22 | 中国移动通信集团广东有限公司 | Database processing method based on distributed file system and electronic equipment |
| CN117494146B (en) * | 2023-12-29 | 2024-04-26 | 山东街景智能制造科技股份有限公司 | Model database management system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104933114A (en) | Mass log management cloud platform | |
| CN101753617B (en) | Cloud storage system and method | |
| CN101854388B (en) | Method and system concurrently accessing a large amount of small documents in cluster storage | |
| US10877810B2 (en) | Object storage system with metadata operation priority processing | |
| CN102750326A (en) | Log management optimization method of cluster system based on downsizing strategy | |
| KR101435789B1 (en) | System and Method for Big Data Processing of DLP System | |
| CN102708158B (en) | PostgreSQL (postgres structured query language) cloud storage filing and scheduling system | |
| CN107818120A (en) | Data processing method and device based on big data | |
| CN101408889A (en) | Method, apparatus and system for monitoring performance | |
| US11221785B2 (en) | Managing replication state for deleted objects | |
| CN104584524A (en) | Aggregating data in a mediation system | |
| CN103067525A (en) | Cloud storage data backup method based on characteristic codes | |
| CN105760236A (en) | Data collection method and system of distributed computer cluster | |
| CN102523251A (en) | Cloud storage architecture for processing mass data and cloud storage platform using the same | |
| US20180052858A1 (en) | Methods and procedures for timestamp-based indexing of items in real-time storage | |
| US20210165767A1 (en) | Barriers for Dependent Operations among Sharded Data Stores | |
| JP2018511861A (en) | Method and device for processing data blocks in a distributed database | |
| CN107330017A (en) | A kind of electric power mass data storage and query and statistical analysis method and its system based on subject example | |
| CN109783018A (en) | A kind of method and device of data storage | |
| CN103117878A (en) | Design method of Nagios-based distribution monitoring system | |
| CN111813332A (en) | High-performance, high-expansion and high-safety intelligent distributed storage system | |
| WO2021112911A1 (en) | Cross storage protocol access response for object data stores | |
| US11023354B2 (en) | Hyper-converged infrastructure (HCI) log system | |
| CN102820998A (en) | Dual-fault-tolerant service system applicable to office applications and data storage method of dual-fault-tolerant service system | |
| US8510473B1 (en) | Converting message character sets for a queue manager |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20150923 |
|
| RJ01 | Rejection of invention patent application after publication |