CN106657408A - Cross-platform log collecting and processing framework - Google Patents
Cross-platform log collecting and processing framework Download PDFInfo
- Publication number
- CN106657408A CN106657408A CN201710104007.1A CN201710104007A CN106657408A CN 106657408 A CN106657408 A CN 106657408A CN 201710104007 A CN201710104007 A CN 201710104007A CN 106657408 A CN106657408 A CN 106657408A
- Authority
- CN
- China
- Prior art keywords
- log
- platform
- cross
- daily record
- server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/602—Providing cryptographic facilities or services
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/04—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks
- H04L63/0428—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks wherein the data content is protected, e.g. by encrypting or encapsulating the payload
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Theoretical Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- Bioethics (AREA)
- Software Systems (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Computing Systems (AREA)
- Storage Device Security (AREA)
Abstract
The invention discloses a cross-platform log collecting and processing framework comprising a cross-platform collection agent module and a log processing server, wherein the cross-platform collection agent module obtains and encrypts logs of platforms and stores the encrypted logs in a local database of the collection agent, and the log processing server extracts the encrypted logs from the local database according to a preset rule. According to the cross-platform log collecting and processing framework disclosed by the invention, as collection modules of different platforms are arranged, information is collected and processed based on the same log template, the log information is encrypted to ensure the security, and the collection and security protection of logs of different system platforms can be realized.
Description
Technical field
The present invention relates to a kind of cross-platform log collection processes framework, belong to computer realm.
Background technology
With the development of each system platform, the user of a collection of loyalty is each developed, because each system platform is had
The characteristics of having, can adapt to specific technical need, therefore in a big system, it may be necessary to different platforms is realized each
The function of the feature of kind, meanwhile, by being collected what is obtained to daily record produced in network.It can be seen that, result collection system institute
The daily record situation collected is related to the reliability that acquired results are analyzed for basic data.That is, log collection
It is a very important link.
Existing technology is mainly collected in the face of the daily record of identical platform, difficult in the daily record in the face of different platform
Compile while to meet each platform corresponding journal file, it is therefore desirable to a log collection towards different platform
Framework.
The content of the invention
In order to solve the above problems, the present invention processes framework by providing a kind of cross-platform log collection.
On the one hand the technical solution used in the present invention processes framework for a kind of cross-platform log collection, including：Different platform is received
Collection proxy module and log processing server；The different platform collects proxy module and obtains and encrypt the daily record of each platform, will add
Close daily record is stored in the local data base for collecting agency, and the log processing server is according to default rule from local data base
Pull encrypted logs.
Preferably, it is described that daily record is obtained from log processing server based on backup policy also including Log backup server
And store.
Preferably, the log processing server includes key handling unit and MongoBD units, wherein, the key
Cell processing unit is used for the key and decryption encrypted logs that strangeer platform collects proxy module, and the MongoBD units are used
Daily record after storage decryption.
Preferably, the log processing server pulls encrypted logs from local data base, while send to upload successfully believing
Cease to different platform agent module, different platform agent module confirms that uploading successful information then deletes the encrypted logs for having uploaded.
Preferably, the local data base is SQLite databases.
Preferably, each platform collects proxy module and obtains daily record mould from log server by corresponding different platform
Plate, different platform collects proxy module and generates daily record preservation file based on the module.
Preferably, the log server connects different platform collection proxy module and Log backup server based on https.
Beneficial effects of the present invention are by arranging the collection module with different platform, being carried out based on same daily record template
The collection and process of information, while being encrypted to ensure security to log information, it is possible to achieve flat to different system
The collection and safeguard protection of the daily record of platform.
Description of the drawings
Fig. 1 show a kind of cross-platform log collection based on the embodiment of the present invention and processes block schematic illustration；
Fig. 2 show the schematic diagram based on the log collection flow process of the embodiment of the present invention.
Specific embodiment
With reference to embodiments the present invention will be described.
Based on inventive embodiment 1, as shown in Figure 1 a kind of cross-platform log collection processes framework, including：Different platform is collected
Proxy module and log processing server；The different platform collects proxy module and obtains and encrypt the daily record of each platform, will encrypt
Daily record is stored in the local data base for collecting agency, and the log processing server draws according to default rule from local data base
Take encrypted logs.
System also includes Log backup server, described to obtain daily record from log processing server based on backup policy and deposit
Storage.
The log processing server includes key handling unit and MongoBD units, wherein, the cipher key unit is processed
Unit is used for the key and decryption encrypted logs that strangeer platform collects proxy module, and the MongoBD units are used to store solution
Daily record after close.
The log processing server pulls encrypted logs from local data base, puts down to different while sending and uploading successful information
Platform proxy module, different platform agent module confirms that uploading successful information then deletes the encrypted logs for having uploaded.
The local data base is SQLite databases.
Each platform collects proxy module and obtains daily record template, different platform from log server by corresponding different platform
Collect proxy module and daily record preservation file is generated based on the module.
The format and content of the journal file of different platforms is not quite similar, and while preserving different file formats in storage
On be not favourable way, therefore, by processing the corresponding journal file of each platform, that is, obtain key information,
Such as information such as time, event type, is then integrated into a general form, is conducive to storage.
The log server connects different platform and collects proxy module and Log backup server based on https.
Based on inventive embodiment 2, log collection flow process as shown in Figure 2.
Client log is collected agency and is started, and according to configuration file log server is connected, and sends if successful connection
The machine key seed；Log server receives key seed and simultaneously produces a group key and using strategy, will using strategy send to
Client；The DB masterplate files that client is preserved key seed and locally preserved using strategy, Request Log；Log server connects
Receive request Concurrency and send local DB masterplates file to client；It is literary that client locally preserves DB according to template file duplication generation diary
Part, then carries out the interaction that diary uploads strategy, and port is intercepted in the local daily record of acquisition and upwards transmission port information is to log server,
Complete client and collect agency's startup.
The order and mode specified by key and using the customization control client of strategy processes the encryption work of daily record
Make；The preservation of the daily record of consolidation form is realized based on database template；The interaction of strategy is uploaded by daily record, the upper of daily record is made
Passing can be carried out according to the mode of regulation；Log server is set to obtain the end of client by the upload of upwards transmission port information
Message ceases to support access of the third party to client/server.
The log collection agency of this platform was received after daily record using the key received when starting and using strategy to day
Will content is encrypted, and is saved in local SQLite databases.When stored log collection agency can according to years months/
The mode of day sets up multiple SQLite databases, determines for log content to write corresponding SQLite according to the daily record of daily record write
In database.
When log collection agency according to upload policy mandates reach upload daily record opportunity when, log content can be passed through
The mode of HTTPS uploads to log server.Log server is received after log content, according to sending log collection to before
After the key strategy of agency and key are decrypted daily record, using the key of server end oneself log content is carried out to add
Close and be saved in MongoDB, while notify that log collection agency uploads successfully, log collection agency's deletion is local have been uploaded
Log content, is compressed to SQLite databases.
According to the backup policy of server end configuration, server end leads log content after the encryption of server end key
Go out for text, and again export is encrypted and compresses, with Year/Month/Day/when/point/second pass through as filename
The mode of HTTPS backups to Log backup server.
Log server end except receive log collection agency upload log content process and Log backup process with
Outward, an analysis process can be started according to configuration, the process can will be saved according to the demand of the 3rd side's log analysis software
The daily record of server is sent to the designated port of specified IP by the form specified.
The above, simply presently preferred embodiments of the present invention, the invention is not limited in above-mentioned embodiment, as long as
It reaches the technique effect of the present invention with identical means, should all belong to protection scope of the present invention.In the protection model of the present invention
Its technical scheme and/or embodiment can have a variety of modifications and variations in enclosing.
Claims (7)
1. a kind of cross-platform log collection processes framework, it is characterised in that include：
Different platform collects proxy module and log processing server；
The different platform collects proxy module and obtains and encrypt the daily record of each platform, and encrypted logs are stored in the sheet for collecting agency
Ground database, the log processing server pulls encrypted logs according to default rule from local data base.
2. a kind of cross-platform log collection according to claim 1 processes framework, it is characterised in that also including Log backup
Server, the Log backup server obtains daily record and stores based on backup policy from log processing server.
3. a kind of cross-platform log collection according to claim 1 processes framework, it is characterised in that the log processing clothes
Business device includes key handling unit and MongoBD units, wherein,
The cipher key unit processing unit is used for the key and decryption encrypted logs that strangeer platform collects proxy module, described
The daily record that MongoBD units are used for after storage decryption.
4. a kind of cross-platform log collection according to claim 1 processes framework, it is characterised in that the log processing clothes
Business device pulls encrypted logs from local data base, while send uploading successful information to different platform agent module, different platform agent
Module confirms that uploading successful information then deletes the encrypted logs for having uploaded.
5. a kind of cross-platform log collection according to claim 1 processes framework, it is characterised in that the local data base
For SQLite databases.
6. a kind of cross-platform log collection according to claim 1 processes framework, it is characterised in that each platform passes through
Corresponding different platform collects proxy module and obtains daily record template from log server, and different platform collects proxy module and is based on the module
Generate daily record and preserve file.
7. a kind of cross-platform log collection according to claim 2 processes framework, it is characterised in that the log server
Different platform is connected based on https and collects proxy module and Log backup server.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710104007.1A CN106657408A (en) | 2017-02-24 | 2017-02-24 | Cross-platform log collecting and processing framework |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710104007.1A CN106657408A (en) | 2017-02-24 | 2017-02-24 | Cross-platform log collecting and processing framework |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106657408A true CN106657408A (en) | 2017-05-10 |
Family
ID=58848297
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710104007.1A Pending CN106657408A (en) | 2017-02-24 | 2017-02-24 | Cross-platform log collecting and processing framework |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106657408A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109614379A (en) * | 2018-10-22 | 2019-04-12 | 中国平安人寿保险股份有限公司 | Log-output method, device, computer storage medium and computer equipment |
| CN109861958A (en) * | 2018-11-20 | 2019-06-07 | 新疆福禄网络科技有限公司 | Data gathering system and method based on Nginx |
| CN111585807A (en) * | 2020-04-30 | 2020-08-25 | 杭州涂鸦信息技术有限公司 | Log management method and related equipment |
| CN113378199A (en) * | 2021-06-28 | 2021-09-10 | 展讯通信(天津)有限公司 | Intelligent terminal system log saving method and device |
| CN116743738A (en) * | 2023-07-20 | 2023-09-12 | 北京道迩科技有限公司 | Log transmission method and device and electronic equipment |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102594598A (en) * | 2012-02-16 | 2012-07-18 | 浪潮(北京)电子信息产业有限公司 | Log management system and implementation method thereof |
| CN102937931A (en) * | 2012-09-25 | 2013-02-20 | 北京奇虎科技有限公司 | Method and system of log data transmission |
| CN103326896A (en) * | 2013-06-07 | 2013-09-25 | 合一信息技术（北京）有限公司 | System and method for collecting information data generated by a user on Internet |
| CN103412893A (en) * | 2013-07-24 | 2013-11-27 | 广东电子工业研究院有限公司 | Collecting system and collecting method of logs |
| CN103425750A (en) * | 2013-07-23 | 2013-12-04 | 国云科技股份有限公司 | Cross-platform and cross-application log collecting system and collecting managing method thereof |
| CN105099740A (en) * | 2014-05-15 | 2015-11-25 | 中国移动通信集团浙江有限公司 | Log management system and log collection method |
| CN105471635A (en) * | 2015-11-18 | 2016-04-06 | 中国建设银行股份有限公司 | Method, device and system for processing system logs |
| CN106027558A (en) * | 2016-07-04 | 2016-10-12 | 安徽天达网络科技有限公司 | Strong-safety system log implementation method |
-
2017
- 2017-02-24 CN CN201710104007.1A patent/CN106657408A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102594598A (en) * | 2012-02-16 | 2012-07-18 | 浪潮(北京)电子信息产业有限公司 | Log management system and implementation method thereof |
| CN102937931A (en) * | 2012-09-25 | 2013-02-20 | 北京奇虎科技有限公司 | Method and system of log data transmission |
| CN103326896A (en) * | 2013-06-07 | 2013-09-25 | 合一信息技术（北京）有限公司 | System and method for collecting information data generated by a user on Internet |
| CN103425750A (en) * | 2013-07-23 | 2013-12-04 | 国云科技股份有限公司 | Cross-platform and cross-application log collecting system and collecting managing method thereof |
| CN103412893A (en) * | 2013-07-24 | 2013-11-27 | 广东电子工业研究院有限公司 | Collecting system and collecting method of logs |
| CN105099740A (en) * | 2014-05-15 | 2015-11-25 | 中国移动通信集团浙江有限公司 | Log management system and log collection method |
| CN105471635A (en) * | 2015-11-18 | 2016-04-06 | 中国建设银行股份有限公司 | Method, device and system for processing system logs |
| CN106027558A (en) * | 2016-07-04 | 2016-10-12 | 安徽天达网络科技有限公司 | Strong-safety system log implementation method |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109614379A (en) * | 2018-10-22 | 2019-04-12 | 中国平安人寿保险股份有限公司 | Log-output method, device, computer storage medium and computer equipment |
| CN109861958A (en) * | 2018-11-20 | 2019-06-07 | 新疆福禄网络科技有限公司 | Data gathering system and method based on Nginx |
| CN111585807A (en) * | 2020-04-30 | 2020-08-25 | 杭州涂鸦信息技术有限公司 | Log management method and related equipment |
| CN113378199A (en) * | 2021-06-28 | 2021-09-10 | 展讯通信(天津)有限公司 | Intelligent terminal system log saving method and device |
| CN116743738A (en) * | 2023-07-20 | 2023-09-12 | 北京道迩科技有限公司 | Log transmission method and device and electronic equipment |
| CN116743738B (en) * | 2023-07-20 | 2024-04-05 | 北京道迩科技有限公司 | Log transmission method and device and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106657408A (en) | Cross-platform log collecting and processing framework | |
| CN106131048B (en) | Non-trust remote transaction file safe storage system for block chain | |
| US9542410B2 (en) | Source-to-processing file conversion in an electronic discovery enterprise system | |
| CN103595730B (en) | A kind of ciphertext cloud storage method and system | |
| US20100250931A1 (en) | Decryption of electronic communication in an electronic discovery enterprise system | |
| CN111598436A (en) | Voucher management system, method and medium | |
| CN104866394B (en) | A kind of distributed document backup method and system | |
| CN101854392A (en) | Personal data management method based on cloud computing environment | |
| KR20090117722A (en) | Confidential information distribution system and confidential information distribution method | |
| CN103973692A (en) | Automatic collecting system and method for electronic archives based on virtual printer | |
| CN107665399A (en) | A kind of personal file storage based on digital signature technology and credible management of electronic documents method | |
| CN103546544A (en) | Data management system on basis of cloud computing | |
| CN103326999A (en) | File safety management system based on cloud service | |
| CN101447064B (en) | Auditing management system and auditing management method | |
| CN103985073A (en) | Automatic electronic file collection system based on virtual printing and use method thereof | |
| CN111382985A (en) | To-do message integrated pushing system and working method | |
| Rahmatulloh et al. | Web services to overcome interoperability in fingerprint-based attendance system | |
| CN105701094A (en) | ETL data acquisition method and device | |
| JP5337334B2 (en) | Create and transmit secure reports of data | |
| CN106941476A (en) | A kind of method and system of SFTP data acquisitions and audit | |
| US11734446B2 (en) | Secret distribution system and secret distribution method of files | |
| CN111048164A (en) | Medical big data long-term storage system | |
| US20220413971A1 (en) | System and Method for Blockchain Based Backup and Recovery | |
| CN213122985U (en) | PIS authentication system | |
| CN112084522B (en) | Data storage method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20190823 Address after: 100089 Floor 1-4, No. 2 Building, No. 9 Courtyard, Dijin Road, Haidian District, Beijing Applicant after: Beijing Toyou Feiji Electronics Co., Ltd. Address before: 518000 Room 1402, Feiyada Science and Technology Building, Nanshan District, Shenzhen City, Guangdong Province Applicant before: Shenzhen City Rui Bo deposit information technology Co. Ltd. |
|
| TA01 | Transfer of patent application right | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170510 |
|
| RJ01 | Rejection of invention patent application after publication |