CN109977700A - A kind of big data processing system based on network security - Google Patents
A kind of big data processing system based on network security Download PDFInfo
- Publication number
- CN109977700A CN109977700A CN201910265847.5A CN201910265847A CN109977700A CN 109977700 A CN109977700 A CN 109977700A CN 201910265847 A CN201910265847 A CN 201910265847A CN 109977700 A CN109977700 A CN 109977700A
- Authority
- CN
- China
- Prior art keywords
- data
- module
- big
- data processing
- analysis
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
- G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
- G06F21/6227—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database where protection concerns the structure of data, e.g. records, types, queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/70—Protecting specific internal or peripheral components, in which the protection of a component leads to protection of the entire computer
- G06F21/71—Protecting specific internal or peripheral components, in which the protection of a component leads to protection of the entire computer to assure secure computing or processing of information
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Software Systems (AREA)
- General Health & Medical Sciences (AREA)
- Bioethics (AREA)
- Health & Medical Sciences (AREA)
- Databases & Information Systems (AREA)
- Mathematical Physics (AREA)
- Computer And Data Communications (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
- Information Transfer Between Computers (AREA)
Abstract
A kind of big data processing system based on network security, including data acquisition module, Data induction module, internet, data memory module, data processing module, data statistic analysis, data result are shown and data monitoring module；The data acquisition module is connected with the Data induction module, and the Data induction module is connected with the internet；Security firewall is provided on the internet, the internet is connected with the data memory module；The data memory module is connected with the data monitoring module；The data processing module is connected with the internet；The data statistic analysis is connected with the data processing module；The data result displaying is connected with the data statistic analysis.The present invention carries out security protection to big data processing by network security, and network tie improves big data treatment effeciency while improving data processing accuracy.
Description
Technical field
The present invention relates to big data processing technology field more particularly to a kind of big data processing systems based on network security
System.
Background technique
Big data processing system it is many kinds of, there is presently no generally acknowledged classification methods.It masters a skill for clarity
Current situation is here classified to representative system from two angles of loadtype and data type, is on the one hand specified
The scope of application of various systems；On the other hand it can be seen that still unsolved blank field.
From the angle of loadtype, current system can be divided into batch processing, streaming computing and interactive inquiry etc. three
Class.Batch processing system emphasizes that the high-throughput of system, such as Hadoop system are just positioned at batch processing.Streaming computing is in recent years
The research field risen, many application scenarios require the timeliness of calculating high, it is desirable to be able in delay as small as possible
Interior completion, such as online advertisement recommendation, intrusion detection and cheating identification etc..Current representative streaming computing system includes Yahoo
S4, Twitter the MillWheel of Storm, Google, the TimeStream of Microsoft and California, USA university Berkeley point
The DStream etc. in school.Interactive inquiry analysis is mainly used in large-scale data warehouse, from Hive to the Dremel of early stage,
Impala and Shark etc., various SQL-On-Ha-doop and MPP (massiveparallelprocessing,
MPP) system continuously emerges.
From the angle of data type, current system provides a variety of data abstractions such as set, table, figure and matrix.Usually
One programming framework is only suitable for solving the problems, such as certain class, can not be all suitable for various problem domains.For example, MapReduce is suitble to solve
Certainly mutually independent collection class data between record；Piccolo stores list data using distributed memory, can accelerate iteration
The operational efficiency of calculating；MadLINQ provides large-scale matrix operation, can simplify a major class machine learning and data mining algorithm
Exploitation.
With the rise that big data is handled, many problems are also faced in big data treatment process, such as how to be improved big
The data information security of data processing system treatment effeciency, big data processing system is incomplete, and data information leakage causes to send out
Raw heavy losses all influence the stability of big data processing system.
To solve the above problems, proposing a kind of big data processing system based on network security in the application.
Summary of the invention
(1) goal of the invention
To solve technical problem present in background technique, the present invention proposes a kind of big data processing based on network security
System carries out security protection, network tie while improving data processing accuracy to big data processing by network security
Improve big data treatment effeciency.
(2) technical solution
To solve the above problems, the present invention provides a kind of big data processing system based on network security, including interconnection
Net, data memory module, data processing module, data statistic analysis, data result is shown and data monitoring module；
Security firewall is provided on the internet, the internet is connected with the data memory module；The number
It is connected according to memory module with the data monitoring module；The data processing module is connected with the internet；The data system
Meter analysis is connected with the data processing module；The data result displaying is connected with the data statistic analysis；
The data memory module stores data, prevents event of data loss, accesses convenient for data search；
The data processing module handles data；The Internet module is used to connect the hinge of each data module, described mutual
Firewall is provided on networking module to ensure data safety；The data statistic analysis handles data processing module
Data result carries out statistics and analysis；The data result shows that the content of statistical analysis of data is shown, and is easy to understand
The processing result of big data；The data monitoring module is monitored data storage, prevents data memory module storing data
Send loss or abnormal.
Preferably, Data induction module is connected on the internet, connection Ei there are data in the Data induction module
Acquisition module.
Preferably, the data acquisition module includes click data, business datum, external data and artificial input data.
Preferably, the data memory module includes off-line data storage and preliminary data storage.
Preferably, the data monitoring module is connected with alarm module.
Preferably, the data processing module includes micro- batch processing module and stream process module, micro- batch processing module
It is provided with near-line data storage；The stream process module is provided with real-time data memory.
Preferably, the data statistic analysis is connected with the data memory module, the data statistic analysis with it is described
Firewall is provided between data memory module.
Preferably, the data statistic analysis includes extemporaneous analysis, data analysis and data mining.
Above-mentioned technical proposal of the invention has following beneficial technical effect:
1, the connection hinge of big data processing system is connected to by the network security of internet and firewall, it is effectively right
Big data handles the protection of information, prevents big data information from occurring to leak in transmission process or lose, while passing through internet
Connection effectively improve the treatment effeciency of big data processing system；
2, the double data storage of data memory module setting effectively protects storing data, wherein a data
When loss of data occurs for library loss data can be given for change by preliminary data in time, improve the safety of big data processing system data
Property；
3, comprehensive and data processing the diversity of data acquisition effectively improves data-handling efficiency and data processing knot
The accuracy of fruit.
Detailed description of the invention
Fig. 1 is a kind of structural schematic diagram of the big data processing system based on network security proposed by the present invention.
Specific embodiment
In order to make the objectives, technical solutions and advantages of the present invention clearer, With reference to embodiment and join
According to attached drawing, the present invention is described in more detail.It should be understood that these descriptions are merely illustrative, and it is not intended to limit this hair
Bright range.In addition, in the following description, descriptions of well-known structures and technologies are omitted, to avoid this is unnecessarily obscured
The concept of invention.
As shown in Figure 1, the present invention provides a kind of big data processing system based on network security, including internet, number
According to memory module, data processing module, data statistic analysis, data result is shown and data monitoring module；
Security firewall is provided on the internet, the internet is connected with the data memory module；The number
It is connected according to memory module with the data monitoring module；The data processing module is connected with the internet；The data system
Meter analysis is connected with the data processing module；The data result displaying is connected with the data statistic analysis；
The data memory module stores data, prevents event of data loss, accesses convenient for data search；
The data processing module handles data；The Internet module is used to connect the hinge of each data module, described mutual
Firewall is provided on networking module to ensure data safety；The data statistic analysis handles data processing module
Data result carries out statistics and analysis；The data result shows that the content of statistical analysis of data is shown, and is easy to understand
The processing result of big data；The data monitoring module is monitored data storage, prevents data memory module storing data
Send loss or abnormal.
In an alternative embodiment, Data induction module, the Data induction module are connected on the internet
Upper connection Ei has data acquisition module.
In an alternative embodiment, the data acquisition module include click data, business datum, external data and
Artificial input data, click data obtains user access activity in such a way that the page is embedded in JS code, and is sent to web services
Backstage record log then the click steam log generated on each server is converged to by way of real-time or batch
In HDFS file system, business datum includes the related data of business information, such as user information, merchandise news, order information；
External data includes the data systems such as external system data, Web system data and mobile APP；Artificial input data passes through reality
When interface data are written manually；The data acquisition of multiple spot improves big data acquisition range, improves database stores information,
It is more accurate convenient for data processing.
In an alternative embodiment, the data memory module includes off-line data storage and preliminary data storage；
Data memory module stores big data information, stores two ways to data by off-line data storage and preliminary data
Stored, it is ensured that the safety of data, wherein a data manually improper modification or lose when, it is ensured that another data
Storage can provide correct data information, keep data processing accurate.
In an alternative embodiment, the data monitoring module is connected with alarm module.Data monitoring module logarithm
It is monitored according to storage, it is ensured that the information security of data storage, when some data storage information sends improper modification or number
When according to losing, alarm module starting is alarmed and is locked by internet and firewall to another data storage, it is ensured that number
According to that will not lose completely, data safety is protected.
In an alternative embodiment, the data processing module includes micro- batch processing module and stream process module, institute
It states micro- batch processing module and is provided with near-line data storage；The stream process module is provided with real-time data memory；It needs to handle big
The measuring data of the task is usually most suitable for being handled with batch operation.No matter data set directly is handled from persistent storage device,
Or data set is loaded into memory first, batch processing system has just fully considered the amount of data in the design process, it is possible to provide sufficient
Process resource.Since performance of the batch processing in terms of coping with a large amount of persistant datas is extremely outstanding, it is often used in going through
History data are analyzed.Stream processing system can handle the data almost without limitation, but the same time can only handle one or very
Low volume data, difference only maintain minimal amount of state between recording.Although most of system is provided for maintaining certain states
Method, but stream process is less mainly for side effect, and more functional processing optimizes.
In an alternative embodiment, the data statistic analysis is connected with the data memory module, the data
Firewall is provided between statistical analysis and the data memory module.The resulting data of data statistic analysis pass through firewall in
It is stored in data memory module, facilitates the access of statistical analysis of data result, the effective statistical analysis of data of firewall
As a result transmission carries out safeguard protection, prevents data statistic analysis result from sending leakage.
In an alternative embodiment, the data statistic analysis includes extemporaneous analysis, data analysis and data mining；
Extemporaneous analysis is the BI analysis that business personnel can manage in " I creates " catalogue and share oneself creation.For carry or
The BI analysis that person shares away, FineBI support the extemporaneous analysis checked under state operation, enjoy the analysis as founder
Function, such as dimension switching, index switching, filter analysis and data drilling analysis etc., this greatly improves Knowledge Conversion sum number
According to the efficiency of analysis；Data analysis analyzes data, the data analysis including R language；Data mining is based on front
Inquiry data carry out data mining, to meet high level data analysis requirements.
In the present invention, data acquisition module is acquired big data, carries out comprehensive data by multiple acquisition mode and adopts
Collection, it is ensured that the data content of big data processing is more extensive, improves the accuracy of data processed result；Data acquisition module is by data
It is sent to after acquisition in Data induction module, Data induction module carries out adopting arrangement in data, by useful data preparation point
Class and being sent in data memory module by internet stores, and is purged to mistake and invalid data, it is ensured that big
The database information of data simplifies the accuracy for being accurately conducive to improve big data processing result；It is anti-that internet is provided with safety
Wall with flues improves data-handling efficiency while ensuring big data information security as connection big data system hinge；Data are deposited
The double data storage of storage module effectively ensures safely data, prevents timely when a database is lost by information
It gives loss data for change, protects data safety；Data monitoring module monitors data storage in real time, it is ensured that database by
Data are protected at the first time when attack, while starting alarm module and alarming, protect data safety；Data processing mould
Block and data statistics analysis module carry out processing and statistical data analysis as a result, converting required letter for data to data
It ceases and as a result, data result is shown to be shown to data result, convenient for big data processing result ocular and clear
Solution.The present invention carries out security protection, network while improving data processing accuracy to big data processing by network security
Tie improves big data treatment effeciency.
It should be understood that above-mentioned specific embodiment of the invention is used only for exemplary illustration or explains of the invention
Principle, but not to limit the present invention.Therefore, that is done without departing from the spirit and scope of the present invention is any
Modification, equivalent replacement, improvement etc., should all be included in the protection scope of the present invention.In addition, appended claims purport of the present invention
Covering the whole variations fallen into attached claim scope and boundary or this range and the equivalent form on boundary and is repairing
Change example.
Claims (8)
1. a kind of big data processing system based on network security, which is characterized in that including internet, data memory module, number
According to processing module, data statistic analysis, data result is shown and data monitoring module；
Security firewall is provided on the internet, the internet is connected with the data memory module；The data are deposited
Storage module is connected with the data monitoring module；The data processing module is connected with the internet；The data statistics point
Analysis is connected with the data processing module；The data result displaying is connected with the data statistic analysis；
The data memory module stores data, prevents event of data loss, accesses convenient for data search；It is described
Data processing module handles data；The Internet module is used to connect the hinge of each data module, the internet
Firewall is provided in module to ensure data safety；The data that the data statistic analysis handles data processing module
As a result statistics and analysis is carried out；The data result shows that the content of statistical analysis of data is shown, and is easy to understand big number
According to processing result；The data monitoring module is monitored data storage, prevents data memory module storing data from sending
It loses or abnormal.
2. a kind of big data processing system based on network security according to claim 1, which is characterized in that the interconnection
It is connected with Data induction module on the net, connection Ei has data acquisition module in the Data induction module.
3. a kind of big data processing system based on network security according to claim 1, which is characterized in that the data
Acquisition module includes click data, business datum, external data and artificial input data.
4. a kind of big data processing system based on network security according to claim 1, which is characterized in that the data
Memory module includes off-line data storage and preliminary data storage.
5. a kind of big data processing system based on network security according to claim 1, which is characterized in that the data
Monitoring module is connected with alarm module.
6. a kind of big data processing system based on network security according to claim 1, which is characterized in that the data
Processing module includes micro- batch processing module and stream process module, and micro- batch processing module is provided with near-line data storage；It is described
Stream process module is provided with real-time data memory.
7. a kind of big data processing system based on network security according to claim 1, which is characterized in that the data
Statistical analysis is connected with the data memory module, is provided between the data statistic analysis and the data memory module anti-
Wall with flues.
8. a kind of big data processing system based on network security according to claim 1, which is characterized in that the data
Statistical analysis includes extemporaneous analysis, data analysis and data mining.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910265847.5A CN109977700A (en) | 2019-04-03 | 2019-04-03 | A kind of big data processing system based on network security |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910265847.5A CN109977700A (en) | 2019-04-03 | 2019-04-03 | A kind of big data processing system based on network security |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109977700A true CN109977700A (en) | 2019-07-05 |
Family
ID=67082692
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910265847.5A Pending CN109977700A (en) | 2019-04-03 | 2019-04-03 | A kind of big data processing system based on network security |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109977700A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111107090A (en) * | 2019-12-20 | 2020-05-05 | 深圳职业技术学院 | Data layer system of information security attack and defense platform |
| CN111858783A (en) * | 2020-07-10 | 2020-10-30 | 脑谷人工智能研究院(南京)有限公司 | Data induction and arrangement platform based on big data intelligent analysis |
| CN114268503A (en) * | 2021-12-24 | 2022-04-01 | 湘潭大学 | Network space data security situation sensing and early warning method and system |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5875481A (en) * | 1997-01-30 | 1999-02-23 | International Business Machines Corporation | Dynamic reconfiguration of data storage devices to balance recycle throughput |
| CN105468741A (en) * | 2015-11-25 | 2016-04-06 | 曙光信息产业(北京)有限公司 | Police affair big data processing system |
| CN106685703A (en) * | 2016-12-13 | 2017-05-17 | 浙江工业大学 | Intelligent data collection and visual monitoring system |
| CN107360153A (en) * | 2017-07-07 | 2017-11-17 | 国家电网公司 | A kind of network security protection system on big data |
| CN107979610A (en) * | 2017-12-14 | 2018-05-01 | 广东天网安全信息科技有限公司 | The safety protecting method that a kind of fire wall communicates in big data |
| CN108764823A (en) * | 2018-05-11 | 2018-11-06 | 甘肃祥龙科技服务有限责任公司 | A kind of data analysis statistical system based on science service |
-
2019
- 2019-04-03 CN CN201910265847.5A patent/CN109977700A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5875481A (en) * | 1997-01-30 | 1999-02-23 | International Business Machines Corporation | Dynamic reconfiguration of data storage devices to balance recycle throughput |
| CN105468741A (en) * | 2015-11-25 | 2016-04-06 | 曙光信息产业(北京)有限公司 | Police affair big data processing system |
| CN106685703A (en) * | 2016-12-13 | 2017-05-17 | 浙江工业大学 | Intelligent data collection and visual monitoring system |
| CN107360153A (en) * | 2017-07-07 | 2017-11-17 | 国家电网公司 | A kind of network security protection system on big data |
| CN107979610A (en) * | 2017-12-14 | 2018-05-01 | 广东天网安全信息科技有限公司 | The safety protecting method that a kind of fire wall communicates in big data |
| CN108764823A (en) * | 2018-05-11 | 2018-11-06 | 甘肃祥龙科技服务有限责任公司 | A kind of data analysis statistical system based on science service |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111107090A (en) * | 2019-12-20 | 2020-05-05 | 深圳职业技术学院 | Data layer system of information security attack and defense platform |
| CN111858783A (en) * | 2020-07-10 | 2020-10-30 | 脑谷人工智能研究院(南京)有限公司 | Data induction and arrangement platform based on big data intelligent analysis |
| CN114268503A (en) * | 2021-12-24 | 2022-04-01 | 湘潭大学 | Network space data security situation sensing and early warning method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20210019674A1 (en) | Risk profiling and rating of extended relationships using ontological databases | |
| Ullah et al. | Architectural tactics for big data cybersecurity analytics systems: a review | |
| US11228595B2 (en) | Evaluating security of data access statements | |
| Ikegwu et al. | Big data analytics for data-driven industry: a review of data sources, tools, challenges, solutions, and research directions | |
| US9479518B1 (en) | Low false positive behavioral fraud detection | |
| US20210112101A1 (en) | Data set and algorithm validation, bias characterization, and valuation | |
| CN109977700A (en) | A kind of big data processing system based on network security | |
| US11570214B2 (en) | Crowdsourced innovation laboratory and process implementation system | |
| US20220198322A1 (en) | Techniques for auto-remediating security issues with artificial intelligence | |
| US20210136121A1 (en) | System and method for creation and implementation of data processing workflows using a distributed computational graph | |
| US11048738B2 (en) | Records search and management in compliance platforms | |
| US11475013B2 (en) | System, method and computer program for ingesting, processing, storing, and searching technology asset data | |
| Savas et al. | Tactical big data analytics: challenges, use cases, and solutions | |
| CN105630797A (en) | Data processing method and system | |
| Karatisoglou et al. | BRIDGE: BRIDGing the gap bEtween CTI production and consumption | |
| Song et al. | A multi-stage uncertain risk decision-making method with reference point based on extended LINMAP method | |
| CN113965497B (en) | Server abnormity identification method and device, computer equipment and readable storage medium | |
| Priyanka et al. | Fundamentals of wireless sensor networks using machine learning approaches: Advancement in big data analysis using Hadoop for oil pipeline system with scheduling algorithm | |
| Choi et al. | An easy-to-use framework to build and operate ai-based intrusion detection for in-situ monitoring | |
| Puzis et al. | ATHAFI: Agile threat hunting and forensic investigation | |
| US10970341B2 (en) | Predictive modeling in event processing systems for big data processing in cloud | |
| CN110209903A (en) | A kind of industry interconnection cloud platform system based on big data | |
| Pintas et al. | SciLightning: a cloud provenance-based event notification for parallel workflows | |
| US20210377313A1 (en) | Threat Mitigation System and Method | |
| US20210092159A1 (en) | System for the prioritization and dynamic presentation of digital content |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190705 |
|
| RJ01 | Rejection of invention patent application after publication |