CN104935469A - Distributive storage method and system for log information - Google Patents
Distributive storage method and system for log information Download PDFInfo
- Publication number
- CN104935469A CN104935469A CN201510377582.XA CN201510377582A CN104935469A CN 104935469 A CN104935469 A CN 104935469A CN 201510377582 A CN201510377582 A CN 201510377582A CN 104935469 A CN104935469 A CN 104935469A
- Authority
- CN
- China
- Prior art keywords
- log
- terminal
- storage
- storage area
- daily record
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention provides a distributive storage method and a distributive storage system for log information. The method comprises the steps as follows: a log collecting terminal obtains a log recording file of a business behavior corresponding to a business type; the log collecting terminal determines a distributive storage area corresponding to the log recording file and transmits the log recording file to a log storage terminal located at the distributive storage area, and the log storage terminal stores the log recording file. The method and the system of the invention could effectively obtain related log information on different machines so as to be convenient for a user and an administrator to perform whole inquiry or analysis to the log, thereby greatly improving user experience.
Description
Technical field
The invention belongs to log information management field, particularly relate to a kind of log information distributed storage method and system.
Background technology
Prior art discloses method and the log server of a kind of daily record storage and process, first daily record data is stored as file, and is uploaded to distributed file system after compressing file, the method first compresses rear storage, decreases taking of memory space; Distributed storage makes the memory capacity of daily record can dynamic expansion; Copy backup is uploaded, and ensure that data can safe storage; During distributed data processing, synchronously can process multiple data block, improve the ability of large data processing, improve systematic function.
Such scheme only only discloses the distributed storage of daily record data, but in distributed structure/architecture, because log information is distributed on different machines, so the log information be associated on acquisition different machines, the ruuning situation of analytical system overall operation situation and each assembly just seems particularly important.
At present, when obtaining the log information be associated on different machines, need to carry out obtaining to different machines and on different machines the position of storing daily record also very random, this considerably increases user operation complexity, have impact on Consumer's Experience.
Therefore, the scheme of log information in the urgent need to a kind of effective acquisition different machines is associated.
Summary of the invention
The invention provides a kind of log information distributed storage method and system, to solve the problem.
The invention provides a kind of log information distributed storage method.Said method comprises the following steps: log collection terminal obtains the log record file of business conduct corresponding to type of service;
Described log collection terminal is determined the distributed storage region corresponding with described log record file and described log record file is sent to the daily record storage terminal that is positioned at described distributed storage region and is stored by described daily record storage terminal.
The present invention also provides a kind of log information distributed memory system, comprises log collection terminal, daily record storage terminal; Wherein, described log collection terminal is connected with described daily record storage terminal;
Described log collection terminal, for obtaining the log record file of business conduct corresponding to type of service; Also for determining the distributed storage region corresponding with described log record file and described log record file being sent to the daily record storage terminal be positioned at described distributed storage region;
Described daily record storage terminal, stores for the log information sent described log collection terminal.
By following scheme: log collection terminal obtains the log record file of business conduct corresponding to type of service; Described log collection terminal is determined the distributed storage region corresponding with described log record file and described log record file is sent to the daily record storage terminal that is positioned at described distributed storage region and is stored by described daily record storage terminal; Achieve the log information that effective acquisition different machines is associated, facilitate user and keeper to carry out overall inquiry or analysis to daily record, greatly improve Consumer's Experience.
By following scheme: described daily record storage terminal is one or more, the surge which prevent log information causes the problem of disk size deficiency.
By following scheme: described first storage area level of security is greater than described second storage area level of security; Described second storage area level of security is greater than described 3rd storage area level of security; Achieve according to log information importance like this, carry out the safe storage of differentiated, greatly improve system data fail safe.
Accompanying drawing explanation
Accompanying drawing described herein is used to provide a further understanding of the present invention, and form a application's part, schematic description and description of the present invention, for explaining the present invention, does not form inappropriate limitation of the present invention.In the accompanying drawings:
Figure 1 shows that the log information distributed storage method process chart of the embodiment of the present invention 1;
Figure 2 shows that the log information distributed memory system structure chart of the embodiment of the present invention 2;
Figure 3 shows that the another structure chart of log information distributed memory system of the embodiment of the present invention 3.
Embodiment
Hereinafter also describe the present invention in detail with reference to accompanying drawing in conjunction with the embodiments.It should be noted that, when not conflicting, the embodiment in the application and the feature in embodiment can combine mutually.
Figure 1 shows that the log information distributed storage method process chart of the embodiment of the present invention 1, comprise the following steps:
Step 101: log collection terminal obtains the log record file of business conduct corresponding to type of service;
Further, described type of service comprises database manipulation, accounting management, system management.
Further, the business conduct that described database manipulation is corresponding comprises: database data is checked, database data is deleted, database data adds, database data upgrades.
Further, the business conduct that described accounting management is corresponding comprises: online charging, offline charging, charging are reported to the police.
Further, the business conduct that described system management is corresponding comprises: system data statistical analysis, systems soft ware optimization renewal, the system failure are reported to the police.
Above-mentioned type of service, business conduct are enumerated, and only illustrate, are not used at this protection range limiting the present embodiment.
Step 102: described log collection terminal is determined the distributed storage region corresponding with described log record file and described log record file is sent to the daily record storage terminal that is positioned at described distributed storage region and stored by described daily record storage terminal.
Further, described daily record storage terminal is one or more.
Further, described log collection terminal storage type of service, business conduct, log record file and the distributed storage region table of comparisons, as shown in table 1:
Type of service, business conduct, log record file and the distributed storage region table of comparisons
Table 1
Further, described distributed storage region comprises the first storage area, the second storage area, the 3rd storage area.
Further, described first storage area is positioned at one or more daily record storage terminal; Described second storage area is positioned at one or more daily record storage terminal; Described 3rd storage area is positioned at one or more daily record storage terminal.
Further, described first storage area level of security is greater than described second storage area level of security; Described second storage area level of security is greater than described 3rd storage area level of security.
Wherein, level of security is higher, illustrates that the cryptographic algorithm adopted is more complicated; Level of security is lower, illustrates that the cryptographic algorithm adopted is simpler.
Step 103: after described daily record storage terminal stores, is sent to log management terminal by the storage area address of correspondence and carries out the management of daily record chained list by described log management terminal to described storage area address.
Further, described log management terminal to the process that described storage area address carries out the management of daily record chained list is:
By described first storage area address, the second storage area address, the 3rd storage area address according to level of security order from high to low, be successively set on the gauge outfit of daily record chained list to showing tail.
Such terminal use by checking the daily record chained list in described log management terminal, and then can obtain the relevant log record file of business, carries out log entry analysis.
Figure 2 shows that the log information distributed memory system structure chart of the embodiment of the present invention 2, comprise log collection terminal 201, daily record storage terminal 202; Wherein, described log collection terminal 201 is connected with described daily record storage terminal 202;
Described log collection terminal 201, for obtaining the log record file of business conduct corresponding to type of service; Also for determining the distributed storage region corresponding with described log record file and described log record file being sent to the daily record storage terminal 202 be positioned at described distributed storage region;
Described daily record storage terminal 202, stores for the log information sent described log collection terminal 201.
Figure 3 shows that the another structure chart of log information distributed memory system of the embodiment of the present invention 3, also comprise log management terminal 200; Wherein, described log management terminal 200 is connected with described daily record storage terminal 202;
Described daily record storage terminal 202, after storing daily record log file, is sent to described log management terminal 200 by the storage area address of correspondence;
Described log management terminal 200, for carrying out the management of daily record chained list to described storage area address.
By following scheme: log collection terminal obtains the log record file of business conduct corresponding to type of service; Described log collection terminal is determined the distributed storage region corresponding with described log record file and described log record file is sent to the daily record storage terminal that is positioned at described distributed storage region and is stored by described daily record storage terminal; Achieve the log information that effective acquisition different machines is associated, facilitate user and keeper to carry out overall inquiry or analysis to daily record, greatly improve Consumer's Experience.
By following scheme: described daily record storage terminal is one or more, the surge which prevent log information causes the problem of disk size deficiency.
By following scheme: described first storage area level of security is greater than described second storage area level of security; Described second storage area level of security is greater than described 3rd storage area level of security; Achieve according to log information importance like this, carry out the safe storage of differentiated, greatly improve system data fail safe.
The foregoing is only the preferred embodiments of the present invention, be not limited to the present invention, for a person skilled in the art, the present invention can have various modifications and variations.Within the spirit and principles in the present invention all, any amendment done, equivalent replacement, improvement etc., all should be included within protection scope of the present invention.
Claims (10)
1. a log information distributed storage method, is characterized in that, comprises the following steps:
Log collection terminal obtains the log record file of business conduct corresponding to type of service;
Described log collection terminal is determined the distributed storage region corresponding with described log record file and described log record file is sent to the daily record storage terminal that is positioned at described distributed storage region and is stored by described daily record storage terminal.
2. method according to claim 1, is characterized in that, described type of service comprises database manipulation, accounting management, system management.
3. method according to claim 2, is characterized in that, the business conduct that described database manipulation is corresponding comprises: database data is checked, database data is deleted, database data adds, database data upgrades; Business conduct corresponding to described accounting management comprises: online charging, offline charging, charging are reported to the police; Business conduct corresponding to described system management comprises: system data statistical analysis, systems soft ware optimization renewal, the system failure are reported to the police.
4. method according to claim 1, is characterized in that, described log collection terminal storage type of service, business conduct, log record file and the distributed storage region table of comparisons.
5. method according to claim 1, is characterized in that, described daily record storage terminal is one or more; Described distributed storage region comprises the first storage area, the second storage area, the 3rd storage area; Described first storage area, described second storage area, described 3rd storage area lay respectively at one or more daily record storage terminal.
6. method according to claim 5, is characterized in that, described first storage area level of security is greater than described second storage area level of security; Described second storage area level of security is greater than described 3rd storage area level of security.
7. method according to claim 1, is characterized in that, after described daily record storage terminal stores, the storage area address of correspondence is sent to log management terminal and carries out the management of daily record chained list by described log management terminal to described storage area address.
8. method according to claim 7, is characterized in that, described log management terminal to the process that described storage area address carries out the management of daily record chained list is:
By described first storage area address, the second storage area address, the 3rd storage area address according to level of security order from high to low, be successively set on the gauge outfit of daily record chained list to showing tail.
9. a log information distributed memory system, is characterized in that, comprises log collection terminal, daily record storage terminal; Wherein, described log collection terminal is connected with described daily record storage terminal;
Described log collection terminal, for obtaining the log record file of business conduct corresponding to type of service; Also for determining the distributed storage region corresponding with described log record file and described log record file being sent to the daily record storage terminal be positioned at described distributed storage region;
Described daily record storage terminal, stores for the log information sent described log collection terminal.
10. system according to claim 9, is characterized in that, also comprises log management terminal; Wherein, described log management terminal is connected with described daily record storage terminal;
Described daily record storage terminal, after storing daily record log file, is sent to described log management terminal by the storage area address of correspondence;
Described log management terminal, for carrying out the management of daily record chained list to described storage area address.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510377582.XA CN104935469A (en) | 2015-06-30 | 2015-06-30 | Distributive storage method and system for log information |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510377582.XA CN104935469A (en) | 2015-06-30 | 2015-06-30 | Distributive storage method and system for log information |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104935469A true CN104935469A (en) | 2015-09-23 |
Family
ID=54122445
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510377582.XA Pending CN104935469A (en) | 2015-06-30 | 2015-06-30 | Distributive storage method and system for log information |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104935469A (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105528454A (en) * | 2015-12-25 | 2016-04-27 | 北京奇虎科技有限公司 | Log treatment method and distributed cluster computing device |
| CN107305521A (en) * | 2016-04-20 | 2017-10-31 | 百度在线网络技术（北京）有限公司 | Log recording method and device |
| CN107508718A (en) * | 2016-06-14 | 2017-12-22 | 阿里巴巴集团控股有限公司 | Log Collect System, log collection unit and log collection method |
| CN108052675A (en) * | 2017-12-28 | 2018-05-18 | 惠州Tcl家电集团有限公司 | Blog management method, system and computer readable storage medium |
| CN108959497A (en) * | 2018-06-26 | 2018-12-07 | 郑州云海信息技术有限公司 | distributed file system log processing method, device, equipment and storage medium |
| CN109687994A (en) * | 2018-11-30 | 2019-04-26 | 北京奇艺世纪科技有限公司 | A kind of message treatment method, system, device and equipment |
| CN109902073A (en) * | 2019-04-03 | 2019-06-18 | 北京奇安信科技有限公司 | Log processing method, device, computer equipment and computer readable storage medium |
| CN110457258A (en) * | 2019-08-06 | 2019-11-15 | 北京字节跳动网络技术有限公司 | A kind of blog management method, device, equipment and medium |
| CN113347209A (en) * | 2021-07-30 | 2021-09-03 | 深圳市永达电子信息股份有限公司 | Service behavior analysis method, system, equipment and storage medium based on state machine |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1361489A (en) * | 2000-12-28 | 2002-07-31 | 深圳市中兴通讯股份有限公司 | Automatic circular daily record information storing method and system |
| US20040098427A1 (en) * | 2002-11-18 | 2004-05-20 | Luosheng Peng | Device memory management during electronic file updating |
| CN101256526A (en) * | 2008-03-10 | 2008-09-03 | 清华大学 | Method for implementing document condition compatibility maintenance in inspection point fault-tolerant technique |
| CN103532754A (en) * | 2013-10-12 | 2014-01-22 | 北京首信科技股份有限公司 | System and method for high-speed memory and distributed type processing of massive logs |
-
2015
- 2015-06-30 CN CN201510377582.XA patent/CN104935469A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1361489A (en) * | 2000-12-28 | 2002-07-31 | 深圳市中兴通讯股份有限公司 | Automatic circular daily record information storing method and system |
| US20040098427A1 (en) * | 2002-11-18 | 2004-05-20 | Luosheng Peng | Device memory management during electronic file updating |
| CN101256526A (en) * | 2008-03-10 | 2008-09-03 | 清华大学 | Method for implementing document condition compatibility maintenance in inspection point fault-tolerant technique |
| CN103532754A (en) * | 2013-10-12 | 2014-01-22 | 北京首信科技股份有限公司 | System and method for high-speed memory and distributed type processing of massive logs |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105528454A (en) * | 2015-12-25 | 2016-04-27 | 北京奇虎科技有限公司 | Log treatment method and distributed cluster computing device |
| CN107305521A (en) * | 2016-04-20 | 2017-10-31 | 百度在线网络技术（北京）有限公司 | Log recording method and device |
| CN107508718A (en) * | 2016-06-14 | 2017-12-22 | 阿里巴巴集团控股有限公司 | Log Collect System, log collection unit and log collection method |
| CN107508718B (en) * | 2016-06-14 | 2021-03-02 | 阿里巴巴集团控股有限公司 | Log acquisition system, log acquisition unit and log acquisition method |
| CN108052675A (en) * | 2017-12-28 | 2018-05-18 | 惠州Tcl家电集团有限公司 | Blog management method, system and computer readable storage medium |
| CN108959497A (en) * | 2018-06-26 | 2018-12-07 | 郑州云海信息技术有限公司 | distributed file system log processing method, device, equipment and storage medium |
| CN109687994A (en) * | 2018-11-30 | 2019-04-26 | 北京奇艺世纪科技有限公司 | A kind of message treatment method, system, device and equipment |
| CN109902073A (en) * | 2019-04-03 | 2019-06-18 | 北京奇安信科技有限公司 | Log processing method, device, computer equipment and computer readable storage medium |
| CN109902073B (en) * | 2019-04-03 | 2020-12-29 | 奇安信科技集团股份有限公司 | Log processing method and device, computer equipment and computer readable storage medium |
| CN110457258A (en) * | 2019-08-06 | 2019-11-15 | 北京字节跳动网络技术有限公司 | A kind of blog management method, device, equipment and medium |
| CN113347209A (en) * | 2021-07-30 | 2021-09-03 | 深圳市永达电子信息股份有限公司 | Service behavior analysis method, system, equipment and storage medium based on state machine |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104935469A (en) | Distributive storage method and system for log information | |
| US20200150890A1 (en) | Data Deduplication Method and Apparatus | |
| US10222987B2 (en) | Data deduplication with augmented cuckoo filters | |
| CN107229420B (en) | Data storage method, reading method, deleting method and data operating system | |
| CN111125089A (en) | Time sequence data storage method, device, server and storage medium | |
| CN104281533B (en) | A kind of method and device of data storage | |
| CN101673192B (en) | Method for time-sequence data processing, device and system therefor | |
| US11176110B2 (en) | Data updating method and device for a distributed database system | |
| CN104978324B (en) | Data processing method and device | |
| CN112714359B (en) | Video recommendation method and device, computer equipment and storage medium | |
| CN105512283A (en) | Data quality management and control method and device | |
| CN105007193A (en) | Multi-layer information processing method, system thereof and cluster management node | |
| CN109074227A (en) | A kind of method and storage system of data check | |
| CN106776795B (en) | Data writing method and device based on Hbase database | |
| CN106326082B (en) | Method and device for recording log in network system | |
| CN103902407A (en) | Virtual machine recovery method and server | |
| CN102523290A (en) | Data processing method, device and system | |
| CN106161637A (en) | A kind of cloud backup method and device | |
| CN106899558B (en) | Access request processing method and device and storage medium | |
| CN109271097B (en) | Data processing method, data processing device and server | |
| CN103778064A (en) | Data management method and device | |
| CN110737389A (en) | Method and device for storing data | |
| CN110851758A (en) | Webpage visitor number statistical method and device | |
| CN106998436B (en) | Video backup method and device | |
| CN105068896A (en) | Data processing method and device based on RAID backup |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20150923 |
|
| RJ01 | Rejection of invention patent application after publication |