CN103414608B - Rapid web flow collection statistical system and method - Google Patents
Rapid web flow collection statistical system and method Download PDFInfo
- Publication number
- CN103414608B CN103414608B CN201310357520.3A CN201310357520A CN103414608B CN 103414608 B CN103414608 B CN 103414608B CN 201310357520 A CN201310357520 A CN 201310357520A CN 103414608 B CN103414608 B CN 103414608B
- Authority
- CN
- China
- Prior art keywords
- flow
- data
- node
- service
- service traffics
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a rapid web flow collection statistical system and method. Based on processing on one-machine web server real-time statistical service flow, the web flow real-time statistical aim is achieved with less resource cost. According to the technical scheme, the rapid web flow collection statistical system comprises a flow collection analysis device, a flow transferring merging device and a flow final-merging device. The flow collection analysis device is deployed on a content distribution network node and collects and analyzes a flow log in real time, service flow data for generating a single node are counted, and the service flow data of the single node are sent to the flow transferring merging device. The flow transferring merging device is deployed on a flow transferring merging node, merging counting is carried out on the service flow data of the content distribution network node, middle flow data of a service are generated in real time, and then the middle flow data are transmitted to the flow final-merging device. The flow final-merging device is deployed on a flow final-merging node, merging counting is carried out on the middle flow data of the service in real time, and final flow data of the service are generated.
Description
Technical field
The present invention relates to web flow amount statistical technique, more particularly to content distributing network（CDN）The web flow amount of service provider
Collecting statistical system and method.
Background technology
CDN service business typically can dispose large-scale CDN to accelerate client to provide the service that effectively accelerate to CDN
Node, these coverage wide ranges, scope even covering the whole world, and for monitor in real time client and node application service
Various datas on flows change, need rapidly to be acquired statistics to the web flow amount on these nodes.
Existing web flow amount statistical technique, the traffic statistics due to being related to service layer will for example count client's domain name
Traffic related data, be to count the flow number of client and node application service by collecting and analyzing web access logs
According to.
This traditional web flow amount statistical technique, the web access logs being distributed across on each node to be processed, day
The scale of will amount is directly proportional with the scale of web visit capacities, and the scale of the daily web access logs amount of CDN service business typically all reaches
TB ranks even PB ranks, i.e. massive logs, it may be said that this technology is suffered from the drawback that：
1) in order to process magnanimity web access log, and these daily records or scattered, need to consume substantial amounts of resource,
Including the consumption of bandwidth for transmission and machinery equipment.
2) statistical data delay is larger, because the time of collection, process and analysis web access logs is long, and due to frame
Level on structure is more, even if increasing resource, it is also difficult to realize real-time statistics.
The content of the invention
It is an object of the invention to solve the above problems, there is provided a kind of quick web flow collection statistical system and side
Method, is the process carried out based on unit web server real-time statistics service traffics, and with less Resources Consumption web flow amount is realized
The problem of real-time statistics.
The technical scheme is that：Present invention is disclosed a kind of quick web flow collection statistical system, including flow
Acquisition and analysis device, flow transfer merge device and flow finally merges device, wherein：
Flow collection analytical equipment, is deployed on content delivery network node, Real-time Collection analysis traffic log, statistics life
Into the service traffics data of single-point, the service traffics data is activation of single-point to flow transfer is merged into device；
Flow transfer merges device, is deployed in flow transfer merge node, by the business on content delivery network node
Data on flows merges statistics, the middle data on flows of business is generated in real time, then is transferred to flow and finally merge device；
Flow finally merges device, is deployed in the final merge node of flow, and the middle flow of statistical service is merged in real time
Data, generate the final data on flows of business.
One embodiment of quick web flow collection statistical system of the invention, flow transfer merge node is root
Choose according to the Node distribution scope of content distributing network.
One embodiment of quick web flow collection statistical system of the invention, the final merge node of flow is root
The coverage of flow transfer merge node is chosen according to selected node.
One embodiment of quick web flow collection statistical system of the invention, flow transfer merge node according to
The node scale of content distributing network carries out extending transversely.
One embodiment of quick web flow collection statistical system of the invention, data on flows include the Connection Time,
Number of request and request method.
Present invention further teaches a kind of quick web flow amount gathers statistical method, on single content delivery network node
Implement, including：
Loading business statistics rule；
Obtain untreated traffic log queue；
Judge that whether traffic log queue is empty, previous step is then returned if it is empty, one is then taken out from queue if not empty
Individual traffic log；
Obtain the corresponding time point of the traffic log；
Parse the traffic log；
According to the service traffics data of the business statistics rule-statistical time point；
The traffic log is deleted after the service traffics data for exporting the time point.
Quick web flow amount of the invention gathers an embodiment of statistical method, parses in traffic log name
To the corresponding time point of the traffic log.
Present invention contrast prior art has following beneficial effect：The present invention system include flow collection analytical equipment,
Flow transfer merges device and flow finally merges device, by arranging an extendible middle merging layer, by network
Merge machine in the middle of detection random selection preferably, the service traffics data transfer of web server is gone out, due to service traffics
Data are simultaneously little, and network routing has been done again, can substantially accomplish real-time Transmission.
Description of the drawings
Fig. 1 shows the structure chart of the preferred embodiment of the quick web flow collection statistical system of the present invention.
Fig. 2 shows the data flow diagram of the quick web flow collection statistical system of the present invention.
Fig. 3 shows the flow chart of the preferred embodiment of the quick web flow amount collection statistical method of the present invention.
Specific embodiment
With reference to the accompanying drawings and examples the invention will be further described.
The embodiment of quick web flow collection statistical system
Fig. 1 shows the structure of the preferred embodiment of the quick web flow collection statistical system of the present invention.Refer to figure
1, the system of the present embodiment includes that flow collection analytical equipment 1, flow transfer merge device 2 and flow finally merges device 3.
Flow collection analytical equipment 1 is deployed on content delivery network node, on a web server Real-time Collection analysis stream
Amount daily record, the essential information that the daily record only needs comprising service traffics data statisticss, and format（Web server needs
According to this journal format record traffic log）As long as doing simple dissection process, it is possible to quick obtaining service traffics data,
The traffic log that flow collection analytical equipment 1 is collected is the lattice of lightweight for general web access logs
The data of formula, this part works for the analyzing and processing of general web access logs, and workload is considerably less, substantially
The performance of web server is not affected.The statistics of flow analysis device 1 generates the service traffics data of single-point, by the Business Stream of single-point
Amount data is activation to flow transfer merges device 2.Here data on flows is not sense stricto data on flows, contains diversification
Data, such as including Connection Time, number of request and request method etc..
Certain Centroid is transferred to after the service traffics data of statistics generation unit and merges statistics, due to CDN sections
Point machine is very more, usually thousand ranks or into ten thousand ranks, disperses in different geographical locations, to be here provided with one
Individual extendible middle merging layer, by merging machine in the middle of network detection random selection preferably, the business of web server
Data on flows is transferred out, and due to service traffics data and less, network routing has been done again, is essentially ensures that the real-time biography of data
The load balancing of defeated and flow transfer merge node.
Flow transfer merges device 2 and is deployed in flow transfer merge node, will be interior according to the business statistics rule of configuration
Service traffics data on content distributing network node merge statistics, and the middle data on flows of business is generated in real time, then are transferred to stream
Amount is final to merge device 3.Chosen according to the Node distribution scope of content distributing network in the present embodiment, that is, inquire after network choosing
Preferably flow transfer merge node is taken, the network coverage of general selected node is preferable.Flow transfer merge node assume responsibility for industry
The part of business data on flows merges statistical work, while also serving as content delivery network node between the final merge node of flow
One terminal of network, and can be carried out according to the node scale of content distributing network extending transversely.
Flow finally merges device 3 and is deployed in the final merge node of flow, real-time according to the business statistics rule of configuration
Merge the middle data on flows of statistical service, generate the final data on flows of business.The final merge node of flow is according to selected
Node is chosen to the coverage of flow transfer merge node.
Fig. 2 shows the data flow of web flow collection statistical system.Fig. 2 is referred to, flow collection analytical equipment 1 is real-time
Collection and analysis traffic log, and the single-point service traffics data is activation for generating is merged into device 2 to flow transfer.Flow transfer
Merge device 2 and analyze each node traffic data on flows in real time, flow data is activation in the middle of the business of generation is finally closed to flow
And device 3.
Quickly web flow amount gathers the embodiment of statistical method
Fig. 3 shows the flow process of the preferred embodiment of the quick web flow amount collection statistical method of the present invention.Refer to figure
3, details are as follows for the implementation steps of the method for the present embodiment.
Step S10：Loading business statistics rule.
Which field is business statistics rule mainly configuration object of statistics will count, such as using domain name as object of statistics,
Each domain name flow per minute and number of request are counted, number of request of each domain name each conditional code per minute etc. is counted.
Step S11：Obtain untreated traffic log queue.
Once traffic log catalogue has newly-increased traffic log, flow collection analytical equipment then scans the untreated stream of generation
Amount journal queue.
Step S12：Judge that whether traffic log queue is empty, if it is empty then return to step S11, then enters if not empty step
Rapid S13.
Step S13：A traffic log is taken out from queue.
Step S14：Obtain the corresponding time point of the traffic log.
The parsing from traffic log name obtains the corresponding time point of the traffic log, and service traffics data statisticss are typically minimum
In units of one minute, so traffic log was recorded by minute.
Step S15：Parse the traffic log.
Flow collection analytical equipment is analyzed process to traffic log by predefined form, obtains the number of each field
According to.
Step S16：According to the service traffics data of the business statistics rule-statistical time point.
Flow collection analytical equipment counts the time point service traffics data according to business statistics rule, such as statistics should
The flow and number of request of each domain name of time point.
Step S17：Export the service traffics data of the time point.
Flow collection analytical equipment is by the service traffics data persistence of the time point to hard disk.
Step S18：Delete the traffic log.
Flow collection analytical equipment is analyzing and processing the traffic log, then the traffic log is deleted, subsequently into step
S12。
Above-described embodiment is available to those of ordinary skill in the art to realize and use the present invention, the common skill in this area
Art personnel can make various modifications or change to above-described embodiment without departing from the present invention in the case of the inventive idea, thus
Protection scope of the present invention is not limited by above-described embodiment, and should meet the inventive features that claims are previously mentioned
Maximum magnitude.
Claims (3)
1. a kind of quick web flow collection statistical system, including flow collection analytical equipment, flow transfer merge device and
Flow finally merges device, wherein：
Flow collection analytical equipment, is deployed on content delivery network node, and Real-time Collection analysis only includes service traffics data
The essential information and formatted traffic log of statistics needs, statistics generates the service traffics data of single-point, by single-point
Service traffics data is activation to flow transfer merges device, and the service traffics data of wherein single-point contain Connection Time, request
The data of the diversification of book and request method；
Flow transfer merges device, the flow transfer merge node being deployed according to selected by the range of nodes of content distributing network
On, the service traffics data on content delivery network node are merged into statistics, the middle data on flows of business is generated in real time, then pass
It is defeated finally to merge device to flow, and flow transfer merge node laterally expanded according to the node scale of content distributing network
Exhibition；
Flow finally merges device, is deployed in the final merge node of flow, and the middle data on flows of statistical service is merged in real time,
The final data on flows of generation business.
2. quick web flow collection statistical system according to claim 1, it is characterised in that flow finally merges section
Point is chosen according to the coverage of flow transfer merge node.
3. a kind of quick web flow amount gathers statistical method, including：
The flow collection analytical equipment Real-time Collection analysis on content delivery network node is deployed in only comprising service traffics data
The essential information and formatted traffic log of statistics needs, statistics generates the service traffics data of single-point, by single-point
Service traffics data is activation to flow transfer merges device, and the service traffics data of wherein single-point contain Connection Time, request
The data of the diversification of book and request method；
Flow transfer merging device is deployed in the flow transfer merge node according to selected by the range of nodes of content distributing network
On, the service traffics data on content delivery network node are merged into statistics, the middle data on flows of business is generated in real time, then pass
It is defeated finally to merge device to flow, and flow transfer merge node laterally expanded according to the node scale of content distributing network
Exhibition；
The flow being deployed in the final merge node of flow finally merges the middle data on flows that device merges in real time statistical service,
The final data on flows of generation business.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310357520.3A CN103414608B (en) | 2013-08-15 | 2013-08-15 | Rapid web flow collection statistical system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310357520.3A CN103414608B (en) | 2013-08-15 | 2013-08-15 | Rapid web flow collection statistical system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103414608A CN103414608A (en) | 2013-11-27 |
| CN103414608B true CN103414608B (en) | 2017-05-17 |
Family
ID=49607594
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310357520.3A Active CN103414608B (en) | 2013-08-15 | 2013-08-15 | Rapid web flow collection statistical system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103414608B (en) |
Families Citing this family (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103905428B (en) * | 2014-01-28 | 2017-06-23 | 北京奇虎科技有限公司 | The method and system of visiting information in a kind of network |
| CN104202759A (en) * | 2014-09-05 | 2014-12-10 | 上海斐讯数据通信技术有限公司 | WIFI (Wireless Fidelity) traffic statistical method |
| CN105791247B (en) * | 2014-12-25 | 2019-02-05 | 中国移动通信集团公司 | A kind of flow system pair service implementation method and relevant device and system |
| CN106027272A (en) * | 2016-04-26 | 2016-10-12 | 乐视控股（北京）有限公司 | CDN (Content Delivery Network) node server traffic time deduction method and system |
| CN105791150B (en) * | 2016-05-09 | 2018-11-30 | 中国联合网络通信集团有限公司 | CDN network dispositions method and system |
| CN106302020B (en) * | 2016-08-18 | 2019-08-16 | 上海帝联信息科技股份有限公司 | Network bandwidth statistical method and device |
| CN106656616A (en) * | 2016-12-29 | 2017-05-10 | 北京天元创新科技有限公司 | Whole network flow analysis method of computer network |
| CN109561051A (en) * | 2017-09-26 | 2019-04-02 | 中兴通讯股份有限公司 | Content distributing network safety detection method and system |
| CN107707414A (en) * | 2017-11-22 | 2018-02-16 | 北京搜狐新媒体信息技术有限公司 | The monitoring system and method for CDN |
| CN110401657B (en) * | 2019-07-24 | 2020-09-25 | 网宿科技股份有限公司 | Processing method and device for access log |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20060012134A (en) * | 2004-08-02 | 2006-02-07 | 주식회사 케이티 | Realtime service management system for enterprise and a method thereof |
| CN1925423A (en) * | 2005-08-30 | 2007-03-07 | 飞塔信息科技（北京）有限公司 | Log device, system and method with function of analyzing network traffic |
| CN201018524Y (en) * | 2007-03-26 | 2008-02-06 | 南京邮电大学 | Mobile communications network based flux remote monitoring system |
| CN101159636A (en) * | 2007-11-23 | 2008-04-09 | 中国电信股份有限公司 | System and method for detecting illegal access |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102571856B (en) * | 2010-12-17 | 2015-04-22 | 中国移动通信集团公司 | Method, device and system for selecting transition node |
-
2013
- 2013-08-15 CN CN201310357520.3A patent/CN103414608B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20060012134A (en) * | 2004-08-02 | 2006-02-07 | 주식회사 케이티 | Realtime service management system for enterprise and a method thereof |
| CN1925423A (en) * | 2005-08-30 | 2007-03-07 | 飞塔信息科技（北京）有限公司 | Log device, system and method with function of analyzing network traffic |
| CN201018524Y (en) * | 2007-03-26 | 2008-02-06 | 南京邮电大学 | Mobile communications network based flux remote monitoring system |
| CN101159636A (en) * | 2007-11-23 | 2008-04-09 | 中国电信股份有限公司 | System and method for detecting illegal access |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103414608A (en) | 2013-11-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103414608B (en) | Rapid web flow collection statistical system and method | |
| CN105490854B (en) | Real-time logs collection method, system and application server cluster | |
| CN105357054B (en) | Website traffic analysis method, device and electronic equipment | |
| CN104488231B (en) | Method, apparatus and system for selectively monitoring flow | |
| CN104778188B (en) | A kind of distributed apparatus log collection method | |
| US9531620B2 (en) | Control plane packet traffic statistics | |
| CN104424229B (en) | A kind of calculation method and system that various dimensions are split | |
| WO2017198227A1 (en) | Interactive internet protocol television system and real-time acquisition method for user data | |
| CN103870297B (en) | The performance data collection system and method for virtual machine in cloud computing environment | |
| EP2544408A1 (en) | Parallel processing for multiple instance real-time monitoring | |
| CN109962790A (en) | A kind of network quality monitoring method, device, electronic equipment and storage medium | |
| KR20170106648A (en) | High-capacity network data processing techniques | |
| US9729563B2 (en) | Data transfer for network interaction fraudulence detection | |
| CN103617287A (en) | Log management method and device in distributed environment | |
| CN104753732A (en) | Distribution based network traffic analysis system and method | |
| CN104504006B (en) | The method and system of data acquisition and parsing to news client | |
| CN103166980B (en) | Internet data pulls method and system | |
| CN108900374A (en) | A kind of data processing method and device applied to DPI equipment | |
| CN107147535A (en) | A kind of distributed network measurement data statistical analysis technique | |
| CN106656616A (en) | Whole network flow analysis method of computer network | |
| CN107579874A (en) | The method and device that a kind of detection flows collecting device data acquisition is failed to report | |
| CN107332685A (en) | A kind of method based on big data O＆M daily record applied in state's net cloud | |
| CN104486116A (en) | Multidimensional query method and multidimensional query system of flow data | |
| CN111459986A (en) | Data computing system and method | |
| CN102664789A (en) | Method and system for processing large-scale data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |