CN106060805A - Big data processing system based on secure wireless transmission - Google Patents
Big data processing system based on secure wireless transmission Download PDFInfo
- Publication number
- CN106060805A CN106060805A CN201610345910.2A CN201610345910A CN106060805A CN 106060805 A CN106060805 A CN 106060805A CN 201610345910 A CN201610345910 A CN 201610345910A CN 106060805 A CN106060805 A CN 106060805A
- Authority
- CN
- China
- Prior art keywords
- data
- module
- big data
- cluster server
- big
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W12/00—Security arrangements; Authentication; Protecting privacy or anonymity
- H04W12/02—Protecting privacy or anonymity, e.g. protecting personally identifiable information [PII]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/04—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks
- H04L63/0428—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks wherein the data content is protected, e.g. by encrypting or encapsulating the payload
Landscapes
- Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Hardware Design (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mobile Radio Communication Systems (AREA)
Abstract
The invention discloses a big data processing system based on secure wireless transmission, relating to a data processing system. The system comprises a front end collection device (10), a 3G/4G wireless transmission device (20) and a cluster server device (30) which are successively in connection. The front end collection device (10) comprises a collection data module (11) and a data encryption module (12) which are embedded therein and have the front and back interacted; the cluster server device (30) comprises a big data decryption module (31), a big data storage module (32) and a big data analysis retrieval module (33) which are embedded therein and are successively interactive. The big data processing system has the characteristics of large data collection size and real-time uploading; in a transmission process, data is encrypted, and transmission security is high; the system employs wireless transmission, and is convenient to arrange nets and maintain; the cluster server processes big data.
Description
Technical field
The present invention relates to data handling system, particularly relate to a kind of based on the big data handling system of secure wireless transmission.
Background technology
Along with the fast development of the Internet, various information data collection capacities also get more and more；Sensitive data is concentrated, and it is right to service
As scope is expanded further, application scenario is increasingly complex, and data application management security risk needs to control effectively；It is transmitted across
In journey, the transmission medium of networking also relies primarily on copper cable or optical cable, constitutes cable network, but cable network to be subject in some occasion
To wiring restriction: wiring, diversion project amount big；Circuit is easily damaged；Each node in net is irremovable, particularly when will be
When node farther out links up, lay the cable wiring manufacture difficulty of dedicated communication line greatly, costly, the longest, align
Serious bottlenecks is defined at the networking requirement expanded rapidly.
Visible, traditional data acquisition transfer management has some limitations, and needs to improve.
Summary of the invention
The purpose of the present invention is that the shortcoming and defect overcoming available data to gather transmission process, it is provided that a kind of based on peace
Full Wireless transmits big data handling system.
The data of magnanimity front-end collection, through encryption, are wirelessly transferred deciphering restoring data after uploading cluster server and enter
Row is analyzed storage and is processed, and has safe transmission and the advantage of big data height concurrent processing, and data storage capacity is big, storage data
Reliability, integrity and concordance are high.
The object of the present invention is achieved like this:
Magnanimity front-end collection data after encryption are decrypted by being wirelessly transmitted to cluster server, enter storage system it
Before, need to carry out data normalization process, i.e. data extracted, clean, associate, comparison, the operation such as mark, improve data
Quality and relatedness, solve different circuit and approach obtain, the Data duplication of separate sources data, form are inconsistent, close
Join the problems such as insufficient, make data integrity, correctness, concordance, terseness, access high efficiency be protected, data standard
Change process part meets the demand of real-time, high efficiency, ease for maintenance, high scalability.
Specifically:
One, based on the big data handling system of secure wireless transmission (abbreviation system)
Native system includes front-end collection equipment, 3G/4G radio transmission apparatus and the cluster server equipment being sequentially communicated.
Described front-end collection equipment include embedded before and after mutual collection data module and data encryption module；
Described cluster server equipment includes the most mutual embedded big data decryption module, big data memory module and big
Data analysis retrieval module.
Two, based on the big data processing method of secure wireless transmission (abbreviation method)
This method comprises the following steps:
1. the collection data module in front-end collection equipment gathers data, and the data collected are encrypted by encrypting module；
2. encryption data is carried out reporting cluster server apparatus by 3G/4G radio transmission apparatus 20；
3. cluster server equipment accepts the encryption data that the front-end collection that 3G/4G radio transmission apparatus reports arrives, by big number
Being decrypted according to deciphering module, the data after deciphering carry out data by big data memory module and are stored into data base；
4. the data of storage are analyzed and user search by cluster server equipment by big data analysis retrieval module.
The present invention has following advantages and a good effect:
1. gather data volume big, upload in real time；
2. encryption in data transmission procedure, transmission security is high；
3. use wireless being transmitted, arrange net convenient and be easy to safeguard；
4. cluster server processes big data.
Accompanying drawing explanation
Fig. 1 is the block diagram of native system；
Wherein:
10 front-end collection equipment,
11 gather data module, 12 data encryption module；
20 3G/4G radio transmission apparatus；
30 cluster server equipment；
31 big data decryption module, 32 big data memory modules, 33 big data analysis retrieval modules.
Detailed description of the invention
One, system
1, overall
Such as Fig. 1, native system includes front-end collection equipment 10,3G/4G radio transmission apparatus 20 and the cluster server being sequentially communicated
Equipment 30.
2, function device
1) front-end collection equipment 10
Front-end collection equipment 10 include embedded before and after mutual collection data module 11 and data encryption module 12.
(1) data module 11 is gathered
Gathering data module 11 and use embedded collection data equipment, in the case of ensureing performance, cost is relative
Relatively low, its function customizable.
(2) data encryption module 12
Data encryption module uses CJSON, RSA-2048, BASE64, hmac algorithm to be encrypted.
2) 3G/4G radio transmission apparatus 20
A kind of equipment supports that 3G/4G dialing is got online without being tethered to a cable.
3) cluster server equipment 30
Cluster server equipment 30 includes embedded the most mutual big data decryption module 31, big data memory module 32 and big
Data analysis retrieval module 33.
(1) big data decryption module 31
Big data decryption module 31 is a kind of number supported after being encrypted CJSON, RSA-2048, BASE64, hmac algorithm
According to the module being decrypted.
(2) big data memory module 32
Big data memory module 32 is a kind of to use the storage of hadoop/hbase distributed mass data, have property extending transversely and
Data reliability, can process the module of PB DBMS.
(3) big data analysis retrieval module 33
Big data analysis retrieval module 33 is a kind of employing multitype database mixed intermediate storage policy design data, can when user uses
To analyze the module of result rapidly according to different rules.
Working mechanism
The collection data module 11 that in the present invention, front-end collection equipment 10 comprises is wireless in opening monitoring pattern capture certain limit
The signal of telecommunication carries out various protocal analysis, parses target data and is stored in memory pool, when the data gathered in memory pool arrive one
Time quantitatively, data encryption module 12 uses enciphering and deciphering algorithm agreement (such as CJSON, RSA-2048, BASE64, HMAC) to enter data
Row encryption, 3G/4G radio transmission apparatus 20 will encryption after data by the Internet transmission to cluster server equipment 30,
Big data decryption module 31 in cluster server equipment 30 receives the encryption data transmitted and is decrypted, and big data are deposited
Storage module 32 uses hadoop/hbase distributed mass data to store, and has property extending transversely and data reliability, can process
PB DBMS, uses spark technology to possess big data analysis real-time computing and the computing of many iterative datas simultaneously, uses
The map/reduce technology of hadoop can calculate task with off-line batch processing, based on storm/ in the processing procedure of data stream
The streaming mode of spark processes data, allows more loose coupling between processing module, it is possible to achieve to the dynamically configuration of workflow and
Optimize；Big data analysis retrieval module 33 uses multitype database mixed intermediate storage policy to design data, for the data encrypted
Mongodb based on memory database is used to store TB rank data, can according to different rules quickly when user uses
Analyze result, data are retrieved, systems management data and ephemeral data are used based on oracle relationship type number
According to storehouse, can preferably design system operation flow.
Claims (2)
1. one kind based on the big data handling system of secure wireless transmission, it is characterised in that:
Including the front-end collection equipment (10) being sequentially communicated, 3G/4G radio transmission apparatus (20) and cluster server equipment (30)；
Front-end collection equipment (10) include embedded before and after mutual collection data module (11) and data encryption module (12)；
Cluster server equipment (30) includes the most mutual embedded big data decryption module (31), big data memory module
(32) and big data analysis retrieval module (33).
2. big data processing method based on system described in claim 1, it is characterised in that comprise the following steps:
1. the collection data module (11) in front-end collection equipment (10) gathers data, the data that encrypting module (12) will collect
It is encrypted；
2. encryption data is carried out reporting cluster server apparatus (30) by 3G/4G radio transmission apparatus (20)；
3. cluster server equipment (30) accepts the encryption data that the front-end collection that 3G/4G radio transmission apparatus (20) reports arrives,
Being decrypted by big data decryption module (31), the data after deciphering carry out data storage by big data memory module (32)
Enter data base；
4. the data of storage are analyzed and user by cluster server equipment (30) by big data analysis retrieval module (33)
Retrieval.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610345910.2A CN106060805A (en) | 2016-05-23 | 2016-05-23 | Big data processing system based on secure wireless transmission |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610345910.2A CN106060805A (en) | 2016-05-23 | 2016-05-23 | Big data processing system based on secure wireless transmission |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106060805A true CN106060805A (en) | 2016-10-26 |
Family
ID=57177692
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610345910.2A Withdrawn CN106060805A (en) | 2016-05-23 | 2016-05-23 | Big data processing system based on secure wireless transmission |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106060805A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106453442A (en) * | 2016-12-28 | 2017-02-22 | 安徽云软信息科技有限公司 | Cloud computing-based massive big data secure cloud storage system |
| CN108234508A (en) * | 2018-01-16 | 2018-06-29 | 武汉虹旭信息技术有限责任公司 | Movable portable safety pattern information terminal obtains system and method |
| CN110572441A (en) * | 2019-08-16 | 2019-12-13 | 苏州迈科网络安全技术股份有限公司 | Ultra-large-scale DPI data processing system and method based on edge calculation |
| CN112364365A (en) * | 2020-11-23 | 2021-02-12 | 中国联合网络通信集团有限公司 | Industrial data encryption method, edge server and computer readable storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101075932A (en) * | 2007-06-29 | 2007-11-21 | 当代天启技术（北京）有限公司 | Method and system for long-distance metering data |
| CN102143188A (en) * | 2011-04-10 | 2011-08-03 | 上海擎龙通信技术有限公司 | Control system based on cloud computing encryption storage service |
| CN103987057A (en) * | 2014-06-05 | 2014-08-13 | 深圳市环境监测中心站 | Data transmission system |
-
2016
- 2016-05-23 CN CN201610345910.2A patent/CN106060805A/en not_active Withdrawn
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101075932A (en) * | 2007-06-29 | 2007-11-21 | 当代天启技术（北京）有限公司 | Method and system for long-distance metering data |
| CN102143188A (en) * | 2011-04-10 | 2011-08-03 | 上海擎龙通信技术有限公司 | Control system based on cloud computing encryption storage service |
| CN103987057A (en) * | 2014-06-05 | 2014-08-13 | 深圳市环境监测中心站 | Data transmission system |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106453442A (en) * | 2016-12-28 | 2017-02-22 | 安徽云软信息科技有限公司 | Cloud computing-based massive big data secure cloud storage system |
| CN108234508A (en) * | 2018-01-16 | 2018-06-29 | 武汉虹旭信息技术有限责任公司 | Movable portable safety pattern information terminal obtains system and method |
| CN110572441A (en) * | 2019-08-16 | 2019-12-13 | 苏州迈科网络安全技术股份有限公司 | Ultra-large-scale DPI data processing system and method based on edge calculation |
| CN110572441B (en) * | 2019-08-16 | 2022-03-18 | 苏州迈科网络安全技术股份有限公司 | Ultra-large-scale DPI data processing system and method based on edge calculation |
| CN112364365A (en) * | 2020-11-23 | 2021-02-12 | 中国联合网络通信集团有限公司 | Industrial data encryption method, edge server and computer readable storage medium |
| CN112364365B (en) * | 2020-11-23 | 2023-05-23 | 中国联合网络通信集团有限公司 | Industrial data encryption method, edge server and computer readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8938534B2 (en) | Automatic provisioning of new users of interest for capture on a communication network | |
| CN107667510A (en) | The detection of Malware and malicious application | |
| CN109670321A (en) | Date storage method, data query method and device | |
| CN106060805A (en) | Big data processing system based on secure wireless transmission | |
| CN104488231A (en) | Real-time network monitoring and subscriber identification with an on-demand appliance | |
| CN102315974A (en) | Stratification characteristic analysis-based method and apparatus thereof for on-line identification for TCP, UDP flows | |
| Jirsik et al. | Toward stream-based IP flow analysis | |
| CN106506266B (en) | Network flow analysis method based on GPU, Hadoop/Spark mixing Computational frame | |
| CN105103496A (en) | System and method for extracting and preserving metadata for analyzing network communications | |
| CN104079885A (en) | Nobody-monitored and linkage-tracked network camera shooting method and device | |
| CN111224940A (en) | Anonymous service traffic correlation identification method and system nested in encrypted tunnel | |
| CN107196930A (en) | Method, system and the mobile terminal of computer network abnormality detection | |
| US20200302216A1 (en) | Data stream identification method and apparatus | |
| CN104486116A (en) | Multidimensional query method and multidimensional query system of flow data | |
| CN103618733A (en) | Data filtering system and method applied to mobile internet | |
| CN106326461A (en) | Real time processing guarantee method and system based on network signaling record | |
| CN107273554A (en) | Elevator intelligent monitoring system and method | |
| Janabi et al. | Convolutional neural network based algorithm for early warning proactive system security in software defined networks | |
| CN107820052A (en) | Fusion method and system based on assessment of bids business datum and video data | |
| CN104535893B (en) | A kind of electric network fault processing unit and its processing method based on finely granular access control | |
| CN110149245A (en) | The compressed sensing based high-speed network flow method of sampling and device | |
| CN109753009A (en) | A kind of level monitoring online data management system | |
| CN106067892A (en) | Management data acquisition | |
| CN109344333A (en) | A kind of internet big data analysis extracting method and system | |
| CN103220188A (en) | Hyper text transport protocol (HTTP) data acquisition equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20161026 |