CN109960622A - A kind of method of data capture based on big data visual control platform - Google Patents
A kind of method of data capture based on big data visual control platform Download PDFInfo
- Publication number
- CN109960622A CN109960622A CN201711399515.3A CN201711399515A CN109960622A CN 109960622 A CN109960622 A CN 109960622A CN 201711399515 A CN201711399515 A CN 201711399515A CN 109960622 A CN109960622 A CN 109960622A
- Authority
- CN
- China
- Prior art keywords
- log
- lcd
- file
- specified
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 24
- 238000013481 data capture Methods 0.000 title claims abstract description 10
- 230000000007 visual effect Effects 0.000 title claims abstract description 10
- 239000000284 extract Substances 0.000 claims abstract description 3
- 230000005540 biological transmission Effects 0.000 claims description 18
- 238000012544 monitoring process Methods 0.000 claims description 8
- 238000000605 extraction Methods 0.000 claims description 4
- 238000000151 deposition Methods 0.000 claims 1
- 239000006185 dispersion Substances 0.000 abstract description 2
- 238000013480 data collection Methods 0.000 description 3
- 238000012423 maintenance Methods 0.000 description 2
- 239000003795 chemical substances by application Substances 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3055—Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/32—Monitoring with visual or acoustical indication of the functioning of the machine
- G06F11/321—Display for diagnostics, e.g. diagnostic result display, self-test user interface
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Quality & Reliability (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Present invention relates particularly to a kind of method of data capture based on big data visual control platform, including log summarizes and collects with log；The log summarizes the daily record reproduction for the different structure in path will disperse, different into a LCD；It is divided into multiple subpaths in the LCD and stores different types of log respectively；The management of the LCD is to divide log file contents according to the time to be stored respectively as journal file of the filename with timestamp, and overtime journal file is cleared up in timing；The LCD also includes one and extracts the data of dependency structure from specified DB database and be stored as the function of specified structure log.Method of data capture provided by the invention is for a server, the log of the different structure in dispersion, different paths is uniformly summarized and is copied under a specified path, it is managed according to unified mode, facilitate Data Transport Agent conveniently inerrably can obtain log information, and safeguards its performance.
Description
Technical field
The invention belongs to telecom operators' electronic channels and mobile Internet communication software field, and in particular to one kind is based on
The method of data capture of big data visual control platform.
Background technique
Visual control platform is the monitoring log by system, analyzes and show the platform ring of the various states of current system
Border, by the log information obtained from the operation node of system, and after being collected cleaning to it, summarizing and convert, by these
Information is stored in distributed search engine environment, then by analyzing these information, finally in a manner of chart etc.
Show or notify to O&M and system developer, to be provided effectively for indexs such as lifting system stability, high availability
Reference frame.First relate to the extraction to data, followed by data collection during monitoring, the data collection be by
After all kinds of journal files that data pick-up link generates are summarized, unified server node is sent to by certain mode
Carry out the process of tether management.
Summary of the invention
1, technical problem to be solved:
The present invention provides a kind of method of data capture based on big data visual control platform, and the data collection of stating is will to count
According to extract link generate all kinds of journal files summarized after, by certain mode be sent to unified server node into
The process of row tether management.
2, technical solution:
A kind of method of data capture based on big data visual control platform, it is characterised in that: the collection method includes day
Will summarizes to be collected with log；The log summarizes the daily record reproduction for the different structure in path will disperse, different to a LCD
That is in Logs Collection Directory；In the LCD according to different types of log, be divided into multiple subpaths point
Different types of log is not stored；The management of the LCD is to divide log file contents according to the time to be stored respectively as file
Journal file of the name with timestamp, and overtime journal file is cleared up in timing；The LCD also includes one from specified DB database
The middle data for extracting dependency structure and the function of being stored as specified structure log, specifically: DB is specified by Yaml configuration file
The link parameter of database and database table structure etc., and the structure of specified file destination, then LCD will self-timing from DB
Log information is extracted in database into particular log file.In above process it can be seen that log summarizes refers to: for one
The log of the different structure in dispersion, different paths is uniformly summarized and is copied under a specified path by platform server, according to
Unified mode is managed, and facilitates Data Transport Agent conveniently inerrably can obtain log information, and safeguard its performance, than
File handle number as reduced occupancy as far as possible, reduces occupancy of disk space etc..Therefore this method using pass through by
Daily record reproduction into a LCD (Logs Collection Directory), thus can also effectively realize data pick-up with
The decoupling of third party's monitoring tools.LCD be a path and manage automatically the path multiple processes general designation, include in LCD
In some subdirectory under the journal file to the catalogue of its internal control of automatic duplication destination address, while can also be to duplication
Content is again regular, and LCD also includes the log content being automatically deleted under its administrative directory according to certain rules, works as timed task
When failure, it can also safeguard that timed task executes its restarting.In an lcd, will according to different types of log, be divided into
Multiple subpaths store different types of log respectively.And the management of LCD will divide log file contents using according to the time
Storage becomes journal file of the filename with timestamp respectively, and the strategy of overtime journal file is cleared up in timing.
The log collects for for will be dispersed in all monitoring log transmissions on each server in LCD catalogue to system
The process of one storage;The transmission process is by log using file transmission engine FileBeat from the log of LCD trustship catalogue
It extracts and is transmitted in caching in file.
It further, further include obtaining the time of log, host using FileBeat addition in the transmission process of log
Name and custom information.
Further, the caching is that Redis is cached.
3, the utility model has the advantages that
(1) in this method by daily record reproduction into a LCD (Logs Collection Directory), so as to effective
The decoupling of ground realization data pick-up and third party's monitoring tools.
(2) while this method can guarantee the stability and accuracy of transmission using file transmission engine FileBeat,
Its own is simultaneously uncomplicated to be easy maintenance.And FileBeat is used, one can be added during log transmission
A little auxiliary informations.
Detailed description of the invention
Fig. 1 is that log extraction of the invention summarizes flow chart；
The flow chart that the log of the invention of the position Fig. 2 collects.
Specific embodiment
LCD be a path and manage automatically the path multiple processes general designation, in LCD comprising automatic duplication target
In some subdirectory under the journal file of location to the catalogue of its internal control, while can also be again regular to the content of duplication,
LCD also includes the log content being automatically deleted under its administrative directory according to certain rules, when timed task failure, also can
Maintenance timed task executes its restarting.In an lcd, will according to different types of log, be divided into multiple subpaths point
Different types of log is not stored, as shown in Figure 1.
Log collects link and is mainly used for will be dispersed in all monitoring logs on each server in LCD catalogue with most
Process of the excellent Capability transmission to unified storage.
For the obstruction for reducing transmission process, overstocked and resource the occupancy of node log is reduced, is needed using buffer queue,
The journal file gathered and content are transferred in buffer queue in time, and cooperate local LCD management application, in time release
The resource that node occupies.The product of buffer queue can consider to use Redis or Kafka according to actual hardware environment resource
It realizes.
Open source product can be used for log transmission there are multiple choices at present, more outstanding open source product such as Flume,
Logstash and Beats.Stability and accuracy of the Flume due to considering transmission emphatically, inevitably make it certainly
Body becomes complicated difficult to safeguard, and Logstash itself has been replaced by ELK official Beats, and this programme will use lightweight
File transmission engine FileBeat as by log from extracted in the journal file of LCD trustship catalogue and be transmitted to caching in
Preferred option, performance and stability, which are directed to for monitoring class log transmission, can reach requirement, transmission process such as Fig. 2 institute
Show.
Although the present invention has been described by way of example and in terms of the preferred embodiments, they be not it is for the purpose of limiting the invention, it is any ripe
This those skilled in the art is practised, without departing from the spirit and scope of the invention, can make various changes or retouch from working as, therefore guarantor of the invention
Shield range should be subject to what claims hereof protection scope was defined.
Claims (3)
1. a kind of method of data capture based on big data visual control platform, it is characterised in that: the collection method includes
Log summarizes to be collected with log；
The log summarizes the daily record reproduction for the different structure in path will disperse, different to an i.e. Logs of LCD
In Collection Directory；In the LCD according to different types of log, be divided into multiple subpaths and store respectively
Different types of log；The management of the LCD is when dividing log file contents according to the time to be stored respectively as filename band
Between the journal file that stabs, and overtime journal file is cleared up in timing；The LCD also includes one and extracts from specified DB database
The data of dependency structure and the function of being stored as specified structure log, specifically: DB database is specified by Yaml configuration file
Link parameter and database table structure etc., and the structure of specified file destination, then LCD will self-timing from DB database
Middle extraction log information is into particular log file；
The log collects for for will be dispersed in all monitoring log transmissions on each server in LCD catalogue to uniformly depositing
The process of storage;The transmission process is by log using file transmission engine FileBeat from the journal file of LCD trustship catalogue
Middle extraction is simultaneously transmitted in caching.
2. a kind of method of data capture based on big data visual control platform according to claim 1, feature exist
In: it further include obtaining time of log, host name and customized using FileBeat addition in the transmission process of log
Information.
3. a kind of method of data capture based on big data visual control platform according to claim 1, feature exist
In: the caching is that Redis is cached.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711399515.3A CN109960622A (en) | 2017-12-22 | 2017-12-22 | A kind of method of data capture based on big data visual control platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711399515.3A CN109960622A (en) | 2017-12-22 | 2017-12-22 | A kind of method of data capture based on big data visual control platform |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109960622A true CN109960622A (en) | 2019-07-02 |
Family
ID=67018733
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711399515.3A Pending CN109960622A (en) | 2017-12-22 | 2017-12-22 | A kind of method of data capture based on big data visual control platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109960622A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111104287A (en) * | 2019-12-20 | 2020-05-05 | 南京亚派科技股份有限公司 | Data acquisition method of dual-power automatic transfer switch online management system |
| CN117194549A (en) * | 2023-11-07 | 2023-12-08 | 上海柯林布瑞信息技术有限公司 | Data transmission method and device based on task data configuration |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130254878A1 (en) * | 2012-03-21 | 2013-09-26 | Owl Computing Technologies, Inc. | Method and apparatus for data transfer reconciliation |
| CN105824744A (en) * | 2016-03-21 | 2016-08-03 | 焦点科技股份有限公司 | Real-time log collection and analysis method on basis of B2B (Business to Business) platform |
| CN106844171A (en) * | 2016-12-27 | 2017-06-13 | 浪潮软件集团有限公司 | Mass operation and maintenance implementation method |
| CN107341258A (en) * | 2017-07-12 | 2017-11-10 | 中国工商银行股份有限公司 | A kind of log data acquisition method and system |
-
2017
- 2017-12-22 CN CN201711399515.3A patent/CN109960622A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130254878A1 (en) * | 2012-03-21 | 2013-09-26 | Owl Computing Technologies, Inc. | Method and apparatus for data transfer reconciliation |
| CN105824744A (en) * | 2016-03-21 | 2016-08-03 | 焦点科技股份有限公司 | Real-time log collection and analysis method on basis of B2B (Business to Business) platform |
| CN106844171A (en) * | 2016-12-27 | 2017-06-13 | 浪潮软件集团有限公司 | Mass operation and maintenance implementation method |
| CN107341258A (en) * | 2017-07-12 | 2017-11-10 | 中国工商银行股份有限公司 | A kind of log data acquisition method and system |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111104287A (en) * | 2019-12-20 | 2020-05-05 | 南京亚派科技股份有限公司 | Data acquisition method of dual-power automatic transfer switch online management system |
| CN117194549A (en) * | 2023-11-07 | 2023-12-08 | 上海柯林布瑞信息技术有限公司 | Data transmission method and device based on task data configuration |
| CN117194549B (en) * | 2023-11-07 | 2024-01-26 | 上海柯林布瑞信息技术有限公司 | Data transmission method and device based on task data configuration |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10929435B2 (en) | Content delivery network analytics management via edge stage collectors | |
| US20230164155A1 (en) | Systems and methods for automated retrieval, processing, and distribution of cyber-threat information | |
| CN102780726B (en) | A kind of log analysis method based on WEB platform and system | |
| US10956362B1 (en) | Searching archived data | |
| CN107818150B (en) | Log auditing method and device | |
| CN103152352B (en) | A kind of perfect information security forensics monitor method based on cloud computing environment and system | |
| CN103414759B (en) | Network disk file transmission method and device | |
| CN103731298A (en) | Large-scale distributed network safety data acquisition method and system | |
| US20150370723A1 (en) | System, Apparatus and Method for Prioritizing the Storage of Content Based on a Threat Index | |
| CN103546343B (en) | The network traffics methods of exhibiting of network traffic analysis system and system | |
| TW201015371A (en) | Storing log data efficiently while supporting querying | |
| CN109858243B (en) | Method and device for tracking virus source | |
| CN101197694A (en) | Central statistics and processing system and method for communication system log | |
| CN108197270B (en) | Distributed file system data recovery method | |
| CN110113188A (en) | Across subdomain communication O&M method, total O&M server and medium | |
| CN105577411A (en) | Cloud service monitoring method and device based on service origin | |
| US20140337471A1 (en) | Migration assist system and migration assist method | |
| CN109960622A (en) | A kind of method of data capture based on big data visual control platform | |
| CN109710767A (en) | Multilingual big data service platform | |
| US9055113B2 (en) | Method and system for monitoring flows in network traffic | |
| CN111740868A (en) | Alarm data processing method and device and storage medium | |
| CN115333966B (en) | Topology-based Nginx log analysis method, system and equipment | |
| CN105099735A (en) | Method and system for acquiring a mass of detailed logs | |
| CN103581119B (en) | System and method for displaying production process data at high speed | |
| CN106326280A (en) | Data processing method, apparatus and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190702 |