CN107070748A - A kind of processing system and method for the big data that communicates - Google Patents
A kind of processing system and method for the big data that communicates Download PDFInfo
- Publication number
- CN107070748A CN107070748A CN201710239991.2A CN201710239991A CN107070748A CN 107070748 A CN107070748 A CN 107070748A CN 201710239991 A CN201710239991 A CN 201710239991A CN 107070748 A CN107070748 A CN 107070748A
- Authority
- CN
- China
- Prior art keywords
- big data
- communication
- module
- way
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/02—Capturing of monitoring data
- H04L43/028—Capturing of monitoring data by filtering
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/04—Processing captured monitoring data, e.g. for logfile generation
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Mining & Analysis (AREA)
- Communication Control (AREA)
Abstract
The present invention relates to a kind of processing system and method for the big data that communicates, its system includes acquisition module, filtering module, sort module, memory module, parsing module and distribution module, and acquisition module is used for the collection communication big data in the way of queue persistence；Filtering module is used to filter the communication big data of collection, removes repetition, invalid communication big data；Classification mould is used to classify to the communication big data after filtering according to data structure；Memory module is used for storage communication big data of classifying；Parsing module is used to by the way of asynchronous crawl obtain communication big data from the memory module, and using corresponding analysis mode parsing communication big data；Distribution module is used to be distributed transmission according to corresponding protocol link to the communication big data after parsing.Present system using classification by the way of to communication big data handle, while before treatment, remove repetition, invalid data, can improve to communication big data processing efficiency.
Description
Technical field
The present invention relates to communication big data field, and in particular to a kind of processing system and method for communication big data.
Background technology
In recent years, big data turns into the topic that the communications field is probed into extensively, because current GSM is in
Under big data environmental background, the great resource value of big data, infomercial value have become communication enterprise one valuable
Assets, the embodiment as mobile communications industry strength and competitiveness, with certain economic value and effect.But existing communication
Data processing will be often handled mass data, there is the problem of efficiency is low, operand is low.
The content of the invention
The technical problems to be solved by the invention are to provide a kind of processing system and method for the big data that communicates, can be to sea
The communication big data of amount carries out efficiently quickly processing.
The technical scheme that the present invention solves above-mentioned technical problem is as follows：A kind of processing system for the big data that communicates, including adopt
Collect module, filtering module, sort module, memory module, parsing module and distribution module,
The acquisition module, it is used for the collection communication big data in the way of queue persistence；
The filtering module, its communication big data for being used for collection is filtered, and is removed repetition, invalid communication and is counted greatly
According to；
The sort module, it is used to classify to the communication big data after filtering according to data structure；
The memory module, it is used for storage communication big data of classifying；
The parsing module, it is used to by the way of asynchronous crawl obtain communication big data from the memory module,
And using corresponding analysis mode parsing communication big data；
The distribution module, it is used to be distributed biography according to corresponding protocol link to the communication big data after parsing
It is defeated.
The beneficial effects of the invention are as follows：A kind of processing system for the big data that communicates of the present invention is by the way of classification to communication
Big data is handled, at the same before treatment, remove repetition, invalid data, can improve to communication big data processing effect
Rate.
On the basis of above-mentioned technical proposal, the present invention can also do following improvement.
Further, the data structure includes network, linear structure and tree structure.
Further, the memory module includes coarse gridding unit, linear memory unit and tree-like memory cell, the net
Lattice memory cell, linear memory unit and tree-like memory cell correspond to save mesh structure, linear structure and tree structure respectively
Communication big data.
Further, the communication big data of network is parsed by the way of distribution.
Further, the communication big data of linear structure is parsed by the way of linear.
Further, the communication big data of tree structure is parsed by the way of tree-like.
Based on a kind of above-mentioned processing system for the big data that communicates, the present invention also provides a kind of processing side for the big data that communicates
Method.
A kind of processing method for the big data that communicates, comprises the following steps,
S1, the collection communication big data in the way of queue persistence；
S2, the communication big data to collection is filtered, and removes repetition, invalid communication big data；
S3, classifies according to data structure to the communication big data after filtering；
S4, classification storage communication big data；
S5, obtains communication big data, and use corresponding parsing by the way of asynchronous crawl from the memory module
Mode parses communication big data；
S6, transmission is distributed to the communication big data after parsing according to corresponding protocol link.
The beneficial effects of the invention are as follows：A kind of processing method for the big data that communicates of the present invention is by the way of classification to communication
Big data is handled, at the same before treatment, remove repetition, invalid data, can improve to communication big data processing effect
Rate.
On the basis of above-mentioned technical proposal, the present invention can also do following improvement.
Further, the data structure includes network, linear structure and tree structure.
Further, the communication of the communication big data of network, the communication big data of linear structure and tree structure is counted greatly
According to being stored respectively with grid, linear and tree-like form.
Further, the communication big data of network is parsed by the way of distribution, and the communication of linear structure is counted greatly
Parsed according to by the way of linear, the communication big data of tree structure is parsed by the way of tree-like.
Brief description of the drawings
Fig. 1 is a kind of structured flowchart of the processing system for the big data that communicates of the present invention；
Fig. 2 is a kind of flow chart of the processing method for the big data that communicates of the present invention.
Embodiment
The principle and feature of the present invention are described below in conjunction with accompanying drawing, the given examples are served only to explain the present invention, and
It is non-to be used to limit the scope of the present invention.
As shown in figure 1, a kind of processing system for the big data that communicates, including acquisition module, filtering module, sort module, deposit
Module, parsing module and distribution module, the acquisition module are stored up, it is used for the big number of collection communication in the way of queue persistence
According to；The filtering module, its communication big data for being used for collection is filtered, and removes repetition, invalid communication big data；Institute
Sort module is stated, it is used to classify to the communication big data after filtering according to data structure；The memory module, it is used for
Classification storage communication big data；The parsing module, it is used to by the way of asynchronous crawl obtain from the memory module
Communicate big data, and using corresponding analysis mode parsing communication big data；The distribution module, it is used for logical after parsing
Letter big data is distributed transmission according to corresponding protocol link.
In this specific embodiment, the data structure includes network, linear structure and tree structure.The storage
Module includes coarse gridding unit, linear memory unit and tree-like memory cell, the coarse gridding unit, linear memory unit
Correspond to the communication big data of save mesh structure, linear structure and tree structure respectively with tree-like memory cell.Network
Communication big data is parsed by the way of distribution.The communication big data of linear structure is parsed by the way of linear.
The communication big data of tree structure is parsed by the way of tree-like.
A kind of processing system for the big data that communicates of the present invention is handled communication big data by the way of classification, simultaneously
Before treatment, remove repetition, invalid data, the efficiency to communication big data processing can be improved.
Based on a kind of above-mentioned processing system for the big data that communicates, the present invention also provides a kind of processing side for the big data that communicates
Method.
A kind of processing method for the big data that communicates, comprises the following steps,
S1, the collection communication big data in the way of queue persistence；
S2, the communication big data to collection is filtered, and removes repetition, invalid communication big data；
S3, classifies according to data structure to the communication big data after filtering；
S4, classification storage communication big data；
S5, obtains communication big data, and use corresponding parsing by the way of asynchronous crawl from the memory module
Mode parses communication big data；
S6, transmission is distributed to the communication big data after parsing according to corresponding protocol link.
Specifically, the data structure includes network, linear structure and tree structure.The communication of network is counted greatly
Carried out respectively with grid, linear and tree-like form according to the communication big data of, the communication big data of linear structure and tree structure
Storage.The communication big data of network is parsed by the way of distribution, and the communication big data of linear structure is using linear
Mode parsed, the communication big data of tree structure is parsed by the way of tree-like.
A kind of processing method for the big data that communicates of the present invention is handled communication big data by the way of classification, simultaneously
Before treatment, remove repetition, invalid data, the efficiency to communication big data processing can be improved.
The foregoing is only presently preferred embodiments of the present invention, be not intended to limit the invention, it is all the present invention spirit and
Within principle, any modification, equivalent substitution and improvements made etc. should be included in the scope of the protection.
Claims (10)
1. a kind of processing system for the big data that communicates, it is characterised in that：Including acquisition module, filtering module, sort module, storage
Module, parsing module and distribution module,
The acquisition module, it is used for the collection communication big data in the way of queue persistence；
The filtering module, its communication big data for being used for collection is filtered, and removes repetition, invalid communication big data；
The sort module, it is used to classify to the communication big data after filtering according to data structure；
The memory module, it is used for storage communication big data of classifying；
The parsing module, it is used to by the way of asynchronous crawl obtain communication big data from the memory module, and adopts
With corresponding analysis mode parsing communication big data；
The distribution module, it is used to be distributed transmission according to corresponding protocol link to the communication big data after parsing.
2. a kind of processing system of big data that communicates according to claim 1, it is characterised in that：The data structure includes
Network, linear structure and tree structure.
3. a kind of processing system of big data that communicates according to claim 2, it is characterised in that：The memory module includes
Coarse gridding unit, linear memory unit and tree-like memory cell, the coarse gridding unit, linear memory unit and tree-like are deposited
Storage unit corresponds to the communication big data of save mesh structure, linear structure and tree structure respectively.
4. a kind of processing system of big data that communicates according to claim 3, it is characterised in that：The communication of network is big
Data are parsed by the way of distribution.
5. a kind of processing system of big data that communicates according to claim 3, it is characterised in that：The communication of linear structure is big
Data are parsed by the way of linear.
6. a kind of processing system of big data that communicates according to claim 3, it is characterised in that：The communication of tree structure is big
Data are parsed by the way of tree-like.
7. a kind of processing method for the big data that communicates, it is characterised in that：Comprise the following steps,
S1, the collection communication big data in the way of queue persistence；
S2, the communication big data to collection is filtered, and removes repetition, invalid communication big data；
S3, classifies according to data structure to the communication big data after filtering；
S4, classification storage communication big data；
S5, obtains communication big data, and use corresponding analysis mode by the way of asynchronous crawl from the memory module
Parsing communication big data；
S6, transmission is distributed to the communication big data after parsing according to corresponding protocol link.
8. a kind of processing method of big data that communicates according to claim 7, it is characterised in that：The data structure includes
Network, linear structure and tree structure.
9. a kind of processing method of big data that communicates according to claim 8, it is characterised in that：The communication of network is big
The communication big data of data, the communication big data of linear structure and tree structure is entered with grid, linear and tree-like form respectively
Row storage.
10. a kind of processing method of big data that communicates according to claim 9, it is characterised in that：The communication of network
Big data is parsed by the way of distribution, and the communication big data of linear structure is parsed by the way of linear, tree-like
The communication big data of structure is parsed by the way of tree-like.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710239991.2A CN107070748A (en) | 2017-04-13 | 2017-04-13 | A kind of processing system and method for the big data that communicates |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710239991.2A CN107070748A (en) | 2017-04-13 | 2017-04-13 | A kind of processing system and method for the big data that communicates |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107070748A true CN107070748A (en) | 2017-08-18 |
Family
ID=59599676
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710239991.2A Pending CN107070748A (en) | 2017-04-13 | 2017-04-13 | A kind of processing system and method for the big data that communicates |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107070748A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108961074A (en) * | 2018-06-22 | 2018-12-07 | 武汉欣网创业科技开发有限公司 | A kind of insurance gridding system |
| CN113475848A (en) * | 2021-06-22 | 2021-10-08 | 北京德风新征程科技有限公司 | Intelligent workbench based on industrial internet big data |
| CN113486112A (en) * | 2021-06-21 | 2021-10-08 | 北京德风新征程科技有限公司 | Big data distribution platform based on industrial internet |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020116538A1 (en) * | 2001-02-22 | 2002-08-22 | International Business Machines Corporation | High-performance memory queue |
| CN1756190A (en) * | 2004-09-30 | 2006-04-05 | 北京航空航天大学 | Distributed performance data acquisition method |
| WO2007049282A2 (en) * | 2005-10-26 | 2007-05-03 | Cortica Ltd. | A computing device, a system and a method for parallel processing of data streams |
| CN103414748A (en) * | 2013-07-12 | 2013-11-27 | 广东电子工业研究院有限公司 | Cloud platform monitoring architecture and monitoring realizing method thereof |
| CN104065741A (en) * | 2014-07-04 | 2014-09-24 | 用友软件股份有限公司 | Data collection system and method |
| CN104980254A (en) * | 2015-06-10 | 2015-10-14 | 柳州市智融科技有限公司 | Internet information processing system |
| CN105099740A (en) * | 2014-05-15 | 2015-11-25 | 中国移动通信集团浙江有限公司 | Log management system and log collection method |
| CN105321121A (en) * | 2014-08-04 | 2016-02-10 | 国网山西省电力公司信息通信分公司 | Power cloud platform based power utilization information acquisition system |
| CN105391742A (en) * | 2015-12-18 | 2016-03-09 | 桂林电子科技大学 | Hadoop-based distributed intrusion detection system |
| CN105516189A (en) * | 2016-01-08 | 2016-04-20 | 乐山师范学院 | Network security enforcement system and method based on big data platform |
| WO2017008604A1 (en) * | 2015-07-15 | 2017-01-19 | 中兴通讯股份有限公司 | Data processing method, device and system |
| CN106470216A (en) * | 2015-08-14 | 2017-03-01 | 江贻芳 | A kind of Content Management System based on information sharing, interaction |
| CN106484857A (en) * | 2016-10-09 | 2017-03-08 | 珠海经济特区远宏科技有限公司大连分公司 | Data collecting system and its method |
| CN106534107A (en) * | 2016-11-04 | 2017-03-22 | 北方工业大学 | Internet of Things message service system |
-
2017
- 2017-04-13 CN CN201710239991.2A patent/CN107070748A/en active Pending
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020116538A1 (en) * | 2001-02-22 | 2002-08-22 | International Business Machines Corporation | High-performance memory queue |
| CN1756190A (en) * | 2004-09-30 | 2006-04-05 | 北京航空航天大学 | Distributed performance data acquisition method |
| WO2007049282A2 (en) * | 2005-10-26 | 2007-05-03 | Cortica Ltd. | A computing device, a system and a method for parallel processing of data streams |
| CN103414748A (en) * | 2013-07-12 | 2013-11-27 | 广东电子工业研究院有限公司 | Cloud platform monitoring architecture and monitoring realizing method thereof |
| CN105099740A (en) * | 2014-05-15 | 2015-11-25 | 中国移动通信集团浙江有限公司 | Log management system and log collection method |
| CN104065741A (en) * | 2014-07-04 | 2014-09-24 | 用友软件股份有限公司 | Data collection system and method |
| CN105321121A (en) * | 2014-08-04 | 2016-02-10 | 国网山西省电力公司信息通信分公司 | Power cloud platform based power utilization information acquisition system |
| CN104980254A (en) * | 2015-06-10 | 2015-10-14 | 柳州市智融科技有限公司 | Internet information processing system |
| WO2017008604A1 (en) * | 2015-07-15 | 2017-01-19 | 中兴通讯股份有限公司 | Data processing method, device and system |
| CN106470216A (en) * | 2015-08-14 | 2017-03-01 | 江贻芳 | A kind of Content Management System based on information sharing, interaction |
| CN105391742A (en) * | 2015-12-18 | 2016-03-09 | 桂林电子科技大学 | Hadoop-based distributed intrusion detection system |
| CN105516189A (en) * | 2016-01-08 | 2016-04-20 | 乐山师范学院 | Network security enforcement system and method based on big data platform |
| CN106484857A (en) * | 2016-10-09 | 2017-03-08 | 珠海经济特区远宏科技有限公司大连分公司 | Data collecting system and its method |
| CN106534107A (en) * | 2016-11-04 | 2017-03-22 | 北方工业大学 | Internet of Things message service system |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108961074A (en) * | 2018-06-22 | 2018-12-07 | 武汉欣网创业科技开发有限公司 | A kind of insurance gridding system |
| CN113486112A (en) * | 2021-06-21 | 2021-10-08 | 北京德风新征程科技有限公司 | Big data distribution platform based on industrial internet |
| CN113475848A (en) * | 2021-06-22 | 2021-10-08 | 北京德风新征程科技有限公司 | Intelligent workbench based on industrial internet big data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107070748A (en) | A kind of processing system and method for the big data that communicates | |
| CN102289522B (en) | Method of intelligently classifying texts | |
| CN102929961B (en) | Based on the data processing method and the device thereof that build rapid data classification passage | |
| CN106453423B (en) | A kind of filtration system and method for the spam based on user individual setting | |
| CN102724219A (en) | A network data computer processing method and a system thereof | |
| CN102970244A (en) | Network message processing method of multi-CPU (Central Processing Unit) inter-core load balance | |
| CN105471670A (en) | Flow data classification method and device | |
| CN104408169A (en) | Multi-dimensional expression language based dimension query method and device | |
| CN105516941A (en) | Interception method and device of spam messages | |
| CN107248252A (en) | A kind of efficient forest fire detecting system | |
| CN107368606A (en) | Data classification method, device and data sorting system | |
| CN209373598U (en) | Real time data and external data association process system | |
| CN109033431A (en) | A kind of efficient electric power networks data acquisition and intelligent analysis system and processing method | |
| CN208940010U (en) | A kind of intranet and extranet synchronization system | |
| CN101883185B (en) | Automatic short message deleting method and system | |
| CN101296155B (en) | Contents classification method and system | |
| CN103957154A (en) | Network data package URL filtering method | |
| CN105681287A (en) | Screening rule based user service allocation screening method | |
| CN1614607A (en) | Filtering method and system for e-mail refuse | |
| CN106919627A (en) | The treating method and apparatus of hot word | |
| CN103580955B (en) | A kind of big flow IP analysis systems and method based on multi-node flow data duplication removal | |
| CN105847063A (en) | Core network data management method and system | |
| CN106022374B (en) | The method and device that a kind of pair of history flow data is classified | |
| CN102955791A (en) | Searching and classifying service system for network information | |
| CN105223403A (en) | The wavelet packet extracting method of a kind of combining inverter net side resonance current information |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170818 |