CN105429784A - Central collection server, and method and system for log file processing - Google Patents
Central collection server, and method and system for log file processing Download PDFInfo
- Publication number
- CN105429784A CN105429784A CN201510728963.8A CN201510728963A CN105429784A CN 105429784 A CN105429784 A CN 105429784A CN 201510728963 A CN201510728963 A CN 201510728963A CN 105429784 A CN105429784 A CN 105429784A
- Authority
- CN
- China
- Prior art keywords
- journal file
- domain
- carrying
- name information
- name
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention relates to a central collection server, and a method and a system for log file processing. The method comprises steps that a log directory sent by an edge node server is split according to domain names, and multiple log files named according to domain name information are generated; when preset sending conditions are satisfied, the log files which are named according to the domain name information are sent to a data center. Through the method, log processing pressure of the data center can be alleviated, and log processing efficiency is improved.
Description
Technical field
The present invention relates to content distributing network technical field, particularly relate to a kind of central authorities and collect server, journal file processing method and system.
Background technology
Along with the fast development of Internet technology, many internet terminals such as handheld device, desktop terminal are widely used, and also create the information of magnanimity simultaneously.Magnanimity information " large data " are behind people's independent behaviour consciousness under cover, can produce the potential value guiding of a lot of commercial activity, thus the fast processing of these mass datas and the demand of analytical technology just day by day urgent.
Content distributing network (ContentDeliveryNetwork, CDN) service provider is by a large amount of edge node server for client provides the service of acceleration, and the journal file quantity that each edge node server produces is huge.But central authorities of the prior art collect server and are generally only responsible for receiving the journal file that each edge node server produces, and process the journal file of reception is sent to data center.Like this, data center needs the journal file processing enormous amount, there is googol according to processing pressure.
Summary of the invention
The problem that the embodiment of the present invention solves is the pressure of the log processing how alleviating data center, improves the efficiency of log processing.
For solving the problem, embodiments provide a kind of journal file processing method, described method comprises:
The Log Directory that edge node server sends is split according to domain name, generates multiple journal file carrying out naming according to domain-name information;
When meeting the transmission condition preset, the journal file carrying out naming according to domain-name information generated is sent to data center.
Alternatively, described title of carrying out the journal file named according to domain-name information comprises the information of form of domain name, identification marking, rise time and journal file.
Alternatively, described default transmission condition is that the cache-time carrying out the journal file named according to domain-name information reaches default time threshold.
Alternatively, described default transmission condition reaches default amount threshold for stored size of carrying out the journal file named according to domain-name information.
The embodiment of the present invention additionally provides a kind of central authorities and collects server, and described central authorities collect server and comprise:
Daily record split cells, the Log Directory be suitable for edge node server sends splits according to domain name, generates multiple journal file carrying out naming according to domain-name information;
Transmitting element, is suitable for when meeting the transmission condition preset, and the journal file carrying out naming according to domain-name information generated is sent to data center.
Alternatively, described title of carrying out the journal file named according to domain-name information comprises the information of form of domain name, identification marking, rise time and journal file.
Alternatively, described default transmission condition is that the cache-time carrying out the journal file named according to domain-name information reaches default time threshold.
Alternatively, described default transmission condition reaches default amount threshold for stored size of carrying out the journal file named according to domain-name information.
The embodiment of the present invention additionally provides a kind of journal file treatment system, and described system comprises above-mentioned central authorities and collects server.
Compared with prior art, technical scheme of the present invention has following advantage:
Above-mentioned scheme, compresses journal file together split by mixing, obtain multiple basis to domain-name information carry out naming the journal file obtained, the processing pressure of data center can be alleviated, promote the treatment effeciency of journal file.
Accompanying drawing explanation
Fig. 1 is the flow chart of a kind of journal file processing method in the embodiment of the present invention;
Fig. 2 is the flow chart of the another kind of journal file processing method in the embodiment of the present invention;
Fig. 3 is the structural representation of the central authorities' collection server in the embodiment of the present invention.
Embodiment
For solving the above-mentioned problems in the prior art, the technical scheme that the embodiment of the present invention adopts splits by mixing compression journal file together, obtain multiple basis to domain-name information carry out naming the journal file obtained, the processing pressure of data center can be alleviated, promote the treatment effeciency of journal file.
For enabling above-mentioned purpose of the present invention, feature and advantage more become apparent, and are described in detail specific embodiments of the invention below in conjunction with accompanying drawing.
Fig. 1 embodiments provides a kind of flow chart of journal file processing method.As shown in Figure 1, in concrete enforcement, described journal file processing method can comprise following step:
Step S101: the Log Directory that edge node server sends is split according to domain name, generates multiple journal file carrying out naming according to domain-name information.
Step S102: when meeting the transmission condition preset, carries out name according to domain-name information be sent to data center by what generates.
Below in conjunction with Fig. 2, the journal file processing method in the embodiment of the present invention is further described in detail.
Shown in Figure 2, in concrete enforcement, a kind of journal file processing method in the embodiment of the present invention can comprise following step:
Step S201: receive the Log Directory that edge node server sends.
In concrete enforcement, CDN service business accelerates to serve for client provides by a large amount of edge node server.In this case, the edge node server in CDN will produce the daily record of enormous amount.Wherein, the journal file produced by edge node server carries out analyzing and processing, can get the business information of comparatively comprehensively user, think that user provides better service.
In concrete enforcement, the journal file that edge node server produces in running is first through being stored in the log buffer district of its internal memory, be arranged on the log collection service of edge node server, can edge node server produce journal file carry out regular collection, and collected journal file is treated as the journal file of standard, namely there is the journal file of preset format.
Wherein, reach default time threshold at the cache-time of the journal file of preset format, or when the size of the journal file of preset format reaches default amount threshold, the journal file of described preset format is moved to and uploads Log Directory.Meet preset condition time, described edge node server can by daily record upload route will upload Log Directory be uploaded to central authorities collection server.Central authorities collect server and then can receive the Log Directory that legal edge node server sends.
Step S202: split according to domain name by received Log Directory, generates multiple journal file carrying out naming according to domain-name information.
In concrete enforcement, an edge of table node server can be multiple offering customers service simultaneously, and therefore, the journal file of Log Directory normally many preset format mixes together.In order to alleviate the log processing pressure at follow-up data center, server is collected by central authorities can when receiving the Log Directory that edge node server sends, received Log Directory is split according to domain name, obtain multiple journal file, to alleviate the log processing pressure at follow-up data center, improve the efficiency of log processing.
Wherein, the journal file obtained can be named based on the domain-name information of described journal file, the multiple journal file carrying out naming according to domain-name information of final generation.Such as, in an embodiment of the present invention, the final multiple titles of carrying out the journal file named according to domain-name information generated can comprise the information of form of domain name, identification marking, rise time and journal file.
Such as, the title of 3 journal files in the Log Directory a.gz of edge node server is as follows:
.....www.dnion.com……；
.....www.qq.com……；
.....www.sina.com…；
Now, to above-mentioned journal file according to the domain name in the 9th field, Log Directory a.gz can be split 3 journal files according to domain-name information name:
Www.dnion.com_md5_ timestamp .gz;
Www.qq.com_md5_ timestamp .gz;
Www.sina.com_md5_ timestamp .gz.
Wherein, md5 is Message Digest Algorithm 5 (MessageDigestAlgorithmMD5) character string of stochastic generation, timestamp represents and splits and generate the information of the time of the corresponding journal file according to domain-name information name, and .gz represents the form of Log Directory.
Here " md5_ timestamp " field that it is pointed out that in the title of journal file obtained after splitting in above-mentioned example is used for the identification marking as described journal file, mutually distinguishes between the journal file being about to have identical domain name.It will be appreciated by those skilled in the art that, described identification marking also can adopt other form, as long as the journal file with same domain name obtained after splitting can be carried out distinguishing each other, the present invention does not limit at this.
Step S203: judge the whether satisfied condition preset of journal file carrying out naming according to domain-name information stored; When judged result is for being, step S203 can be performed; Otherwise, then any operation can not be performed.
In concrete enforcement, described default transmission condition can be arranged according to the actual needs.Such as, in an embodiment of the present invention, described default transmission condition is that the cache-time carrying out the journal file named according to domain-name information reaches default time threshold; In still another embodiment of the process, described default transmission condition can be collected in server for central authorities the size of carrying out the journal file named according to domain-name information stored and reached default amount threshold.
Step S204: the generated journal file carrying out naming according to domain-name information is sent to data center.
In concrete enforcement, when determining to meet the condition preset, server is collected by central authorities can be sent to data center, to carry out further analyzing and processing to journal file by the journal file carrying out naming according to domain-name information stored.
The device corresponding to the journal file processing method in the embodiment of the present invention is done introduction detailed further below.
The structural representation of server is collected by Fig. 3 central authorities shown in the embodiment of the present invention.Server 300 is collected by central authorities as shown in Figure 3, can comprise daily record split cells 301 and transmitting element 302, wherein:
Described daily record split cells 301, the Log Directory be suitable for edge node server sends splits according to domain name, generates multiple journal file carrying out naming according to domain-name information.
In concrete enforcement, the title of domain name journal file set comprises the information of form of domain name, identification marking, rise time and journal file.
Described transmitting element 302, is suitable for when meeting the transmission condition preset, and the journal file carrying out naming according to domain-name information generated is sent to data center.
In concrete enforcement, described default transmission condition is that the cache-time of domain name journal file set reaches default time threshold, or the size of the domain name journal file set stored reaches default amount threshold.
The embodiment of the present invention additionally provides a kind of journal file treatment system, and described system comprises above-mentioned central authorities and collects server 300.
One of ordinary skill in the art will appreciate that all or part of step in the various methods of above-described embodiment is that the hardware that can carry out instruction relevant by program has come, this program can be stored in computer-readable recording medium, and storage medium can comprise: ROM, RAM, disk or CD etc.
Done detailed introduction to the method and system of the embodiment of the present invention above, the present invention is not limited to this.Any those skilled in the art, without departing from the spirit and scope of the present invention, all can make various changes or modifications, and therefore protection scope of the present invention should be as the criterion with claim limited range.
Claims (9)
1. a journal file processing method, is characterized in that, comprising:
The Log Directory that edge node server sends is split according to domain name, generates multiple journal file carrying out naming according to domain-name information;
When meeting the transmission condition preset, the journal file carrying out naming according to domain-name information generated is sent to data center.
2. journal file processing method according to claim 1, is characterized in that, described title of carrying out the journal file named according to domain-name information comprises the information of form of domain name, identification marking, rise time and journal file.
3. journal file processing method according to claim 1, is characterized in that, described default transmission condition is that the cache-time carrying out the journal file named according to domain-name information reaches default time threshold.
4. journal file processing method according to claim 1, is characterized in that, described default transmission condition reaches default amount threshold for stored size of carrying out the journal file named according to domain-name information.
5. a server is collected by central authorities, it is characterized in that, comprising:
Daily record split cells, the Log Directory be suitable for edge node server sends splits according to domain name, generates multiple journal file carrying out naming according to domain-name information;
Transmitting element, is suitable for when meeting the transmission condition preset, and the journal file carrying out naming according to domain-name information generated is sent to data center.
6. server is collected by central authorities according to claim 5, it is characterized in that, described title of carrying out the journal file named according to domain-name information comprises the information of form of domain name, identification marking, rise time and journal file.
7. server is collected by central authorities according to claim 5, it is characterized in that, described default transmission condition is that the cache-time carrying out the journal file named according to domain-name information reaches default time threshold.
8. server is collected by central authorities according to claim 5, it is characterized in that, described default transmission condition reaches default amount threshold for stored size of carrying out the journal file named according to domain-name information.
9. a journal file treatment system, is characterized in that, server is collected by the central authorities comprised described in any one of claim 5-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510728963.8A CN105429784A (en) | 2015-10-30 | 2015-10-30 | Central collection server, and method and system for log file processing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510728963.8A CN105429784A (en) | 2015-10-30 | 2015-10-30 | Central collection server, and method and system for log file processing |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105429784A true CN105429784A (en) | 2016-03-23 |
Family
ID=55507735
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510728963.8A Pending CN105429784A (en) | 2015-10-30 | 2015-10-30 | Central collection server, and method and system for log file processing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105429784A (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106027595A (en) * | 2016-04-25 | 2016-10-12 | 乐视控股（北京）有限公司 | Access log processing method and system for CDN node |
| CN106850761A (en) * | 2016-12-30 | 2017-06-13 | 江苏天联信息科技发展有限公司 | Journal file storage method and device |
| CN106878397A (en) * | 2017-01-21 | 2017-06-20 | 浙江沛宏网络科技有限公司 | A kind of WEB user behaviors feedback method and system |
| CN107332719A (en) * | 2017-08-16 | 2017-11-07 | 北京云端智度科技有限公司 | A kind of method that daily record is analyzed in real time in CDN system |
| CN107704478A (en) * | 2017-01-16 | 2018-02-16 | 贵州白山云科技有限公司 | A kind of method and system for writing daily record |
| CN108234233A (en) * | 2016-12-22 | 2018-06-29 | 阿里巴巴集团控股有限公司 | Log processing method and device |
| CN111339183A (en) * | 2020-02-11 | 2020-06-26 | 腾讯云计算（北京）有限责任公司 | Data processing method, edge node, data center and storage medium |
| CN112685375A (en) * | 2019-10-18 | 2021-04-20 | 上海哔哩哔哩科技有限公司 | CDN-based log analysis method and device |
| CN113190522A (en) * | 2021-06-29 | 2021-07-30 | 阿里云计算有限公司 | Log processing method and device, computing equipment and storage medium |
| CN115150767A (en) * | 2022-09-06 | 2022-10-04 | 湖南智警公共安全技术研究院有限公司 | Wireless sensor network data transmission method based on edge calculation |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101958837A (en) * | 2010-09-30 | 2011-01-26 | 北京世纪互联工程技术服务有限公司 | Log processing system, log processing method, node server and center server |
| CN102447633A (en) * | 2011-12-29 | 2012-05-09 | 北京亿赞普网络技术有限公司 | Log transmission method and system |
| CN105005528A (en) * | 2015-06-26 | 2015-10-28 | 浪潮(北京)电子信息产业有限公司 | Log information extraction method and apparatus |
-
2015
- 2015-10-30 CN CN201510728963.8A patent/CN105429784A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101958837A (en) * | 2010-09-30 | 2011-01-26 | 北京世纪互联工程技术服务有限公司 | Log processing system, log processing method, node server and center server |
| CN102447633A (en) * | 2011-12-29 | 2012-05-09 | 北京亿赞普网络技术有限公司 | Log transmission method and system |
| CN105005528A (en) * | 2015-06-26 | 2015-10-28 | 浪潮(北京)电子信息产业有限公司 | Log information extraction method and apparatus |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106027595A (en) * | 2016-04-25 | 2016-10-12 | 乐视控股（北京）有限公司 | Access log processing method and system for CDN node |
| CN108234233A (en) * | 2016-12-22 | 2018-06-29 | 阿里巴巴集团控股有限公司 | Log processing method and device |
| CN108234233B (en) * | 2016-12-22 | 2021-09-24 | 阿里巴巴集团控股有限公司 | Log processing method and device |
| CN106850761A (en) * | 2016-12-30 | 2017-06-13 | 江苏天联信息科技发展有限公司 | Journal file storage method and device |
| CN107704478A (en) * | 2017-01-16 | 2018-02-16 | 贵州白山云科技有限公司 | A kind of method and system for writing daily record |
| CN106878397A (en) * | 2017-01-21 | 2017-06-20 | 浙江沛宏网络科技有限公司 | A kind of WEB user behaviors feedback method and system |
| CN107332719A (en) * | 2017-08-16 | 2017-11-07 | 北京云端智度科技有限公司 | A kind of method that daily record is analyzed in real time in CDN system |
| CN112685375A (en) * | 2019-10-18 | 2021-04-20 | 上海哔哩哔哩科技有限公司 | CDN-based log analysis method and device |
| CN112685375B (en) * | 2019-10-18 | 2023-04-28 | 上海哔哩哔哩科技有限公司 | Log analysis method and device based on CDN |
| CN111339183A (en) * | 2020-02-11 | 2020-06-26 | 腾讯云计算（北京）有限责任公司 | Data processing method, edge node, data center and storage medium |
| CN113190522A (en) * | 2021-06-29 | 2021-07-30 | 阿里云计算有限公司 | Log processing method and device, computing equipment and storage medium |
| CN115150767A (en) * | 2022-09-06 | 2022-10-04 | 湖南智警公共安全技术研究院有限公司 | Wireless sensor network data transmission method based on edge calculation |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105429784A (en) | Central collection server, and method and system for log file processing | |
| Bruns et al. | Tools and methods for capturing Twitter data during natural disasters | |
| CN103970793B (en) | Information query method, client and server | |
| CN105847020B (en) | Message pushing method and device | |
| US20150227496A1 (en) | Method and system for microblog resource sharing | |
| CN102546836A (en) | Method, terminal, server and system for uploading files | |
| CN102546668B (en) | Method, device and system for counting unique visitors | |
| CN103778148A (en) | Life cycle management method and equipment for data file of Hadoop distributed file system | |
| CN104506493A (en) | HLS content source returning and caching realization method | |
| CN105357041A (en) | Edge node server, and log file uploading method and system | |
| CN112765103B (en) | File analysis method, system, device and equipment | |
| Bruns et al. | Notes towards the scientific study of public communication on Twitter | |
| CN103281192B (en) | Method, device and system for retrieving data | |
| CN102752388A (en) | Browser-based interactive system, browser-based interactive method, browser and cloud server | |
| CN113742518B (en) | Methods, apparatus and computer program products for storing and providing video | |
| CN110097397B (en) | Feedback-based information pushing method and device and electronic equipment | |
| CN105430534A (en) | Intelligent device reporting data method and system | |
| CN110839077A (en) | File request processing method, request feedback information processing method and related components | |
| CN104978325A (en) | Webpage processing method and device, and user terminal | |
| CN102075355B (en) | Log system and using method thereof | |
| CN113141403B (en) | Log transmission method and device | |
| CN104113509B (en) | Pushing method for media files and server | |
| CN105429783A (en) | Edge node server, log file processing method and log file processing system | |
| CN114679324A (en) | Data exchange method, tool, system, equipment and medium | |
| CN102333121B (en) | Method for sending and receiving HTTP (Hyper Text Transfer Protocol) attachment as well as server and terminal equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB03 | Change of inventor or designer information |
Inventor after: Tang Qiankun Inventor after: Yang Ning Inventor after: Gu Qingrong Inventor after: Kang Kai Inventor before: Tang Qiankun Inventor before: Yang Ning Inventor before: Gu Qingrong |
|
| COR | Change of bibliographic data | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20160323 |