CN114157463A - Big data analysis-based network information security early warning platform and early warning method - Google Patents
Big data analysis-based network information security early warning platform and early warning method Download PDFInfo
- Publication number
- CN114157463A CN114157463A CN202111390236.7A CN202111390236A CN114157463A CN 114157463 A CN114157463 A CN 114157463A CN 202111390236 A CN202111390236 A CN 202111390236A CN 114157463 A CN114157463 A CN 114157463A
- Authority
- CN
- China
- Prior art keywords
- unit
- data
- network
- early warning
- analysis
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000007405 data analysis Methods 0.000 title claims abstract description 20
- 238000000034 method Methods 0.000 title claims abstract description 18
- 238000012544 monitoring process Methods 0.000 claims abstract description 73
- 238000004458 analytical method Methods 0.000 claims abstract description 54
- 238000012545 processing Methods 0.000 claims abstract description 51
- 230000005540 biological transmission Effects 0.000 claims abstract description 49
- 238000013480 data collection Methods 0.000 claims abstract description 41
- 238000012423 maintenance Methods 0.000 claims abstract description 21
- 238000004364 calculation method Methods 0.000 claims description 27
- 238000007726 management method Methods 0.000 claims description 7
- 239000012634 fragment Substances 0.000 claims description 6
- 238000004140 cleaning Methods 0.000 claims description 3
- 238000013467 fragmentation Methods 0.000 claims description 3
- 238000006062 fragmentation reaction Methods 0.000 claims description 3
- 230000008569 process Effects 0.000 claims description 3
- 230000000007 visual effect Effects 0.000 claims description 2
- 230000007246 mechanism Effects 0.000 abstract description 4
- 238000011160 research Methods 0.000 abstract description 4
- 238000005516 engineering process Methods 0.000 abstract description 2
- 238000010224 classification analysis Methods 0.000 description 5
- 238000004148 unit process Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 230000001960 triggered effect Effects 0.000 description 2
- 241000700605 Viruses Species 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0631—Management of faults, events, alarms or notifications using root cause analysis; using analysis of correlation between notifications, alarms or events based on decision criteria, e.g. hierarchy, tree or time analysis
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Security & Cryptography (AREA)
- Environmental & Geological Engineering (AREA)
- Computer Hardware Design (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a network information safety early warning platform and an early warning method based on big data analysis, which comprise a data collection unit, an operation and maintenance unit, a positioning unit, a transmission unit, a data processing unit, a distributed database, an analysis monitoring unit and an alarm unit, wherein the output end of the data collection unit is respectively connected with the input ends of the transmission unit and the analysis monitoring unit, the output end of the analysis monitoring unit is connected with the input end of the transmission unit, the output end of the operation and maintenance is connected with the input end of the transmission unit, and the output end of the transmission unit is connected with the input end of the data processing unit. The network information safety early warning platform can effectively monitor and analyze safety risks in a network, meanwhile, information safety early warning technology research based on network data, network information and network content analysis can be developed based on the network information safety early warning platform, a network and information safety event early warning working mechanism is established, and judgment basis is provided for network safety early warning and disposal.
Description
Technical Field
The invention relates to the technical field of network security, in particular to a network information security early warning platform and an early warning method based on big data analysis.
Background
The network security includes network device security, network information security and network software security, which means that the hardware, software and data in the system of the network system are protected and are not damaged, changed and leaked due to accidental or malicious reasons, the system continuously, reliably and normally operates, and the network service is not interrupted. The system has the characteristics of confidentiality, integrity, availability, controllability and auditability.
From the perspective of network operation and managers, it is desirable to protect and control the operations of accessing, reading and writing information of local network, avoid the threats of 'trapdoor', virus, illegal access, denial of service, illegal occupation and illegal control of network resources, and prevent and defend the attack of network hackers. For security and confidentiality departments, the departments hope to filter and prevent blocking of illegal, harmful or national confidential information, avoid leakage of confidential information, avoid harm to the society and huge loss to the country, and particularly urgently build a network information security early warning platform and an early warning method based on big data analysis based on the network structure and the current situation of security protection.
Disclosure of Invention
Aiming at the problems, the invention provides a network information safety early warning platform and an early warning method based on big data analysis, and the early warning platform and the early warning method have the advantage of maintaining network safety.
The technical scheme of the invention is as follows: a network information safety early warning platform based on big data analysis comprises a data collection unit, an operation and maintenance unit, a positioning unit, a transmission unit, a data processing unit, a distributed database, an analysis monitoring unit and an alarm unit, the output end of the data collection unit is respectively connected with the input ends of the transmission unit and the analysis monitoring unit, the output end of the analysis monitoring unit is connected with the input end of the transmission unit, the output end of the operation and maintenance unit is connected with the input end of the transmission unit, the output end of the transmission unit is connected with the input end of the data processing unit, the output end of the data processing unit is connected with the input end of the distributed database, the output ends of the data processing unit and the distributed database are connected with an alarm unit, and the output end of the data processing unit is connected with a display unit and a backup unit.
The working principle of the technical scheme is as follows:
the network information, the network data and the network content uploaded or edited by different IPs are collected by a data collection unit and then are respectively transmitted to an analysis monitoring unit and a transmission unit, the analysis monitoring unit monitors the network data, the network information and the network content transmitted by the data collection unit by network information monitoring, network data monitoring and network content monitoring and then transmits the monitoring data to the transmission unit, the transmission unit transmits the data of the data collection unit and the analysis monitoring unit to a data processing unit, the data processing unit processes the data transmitted by the transmission unit and judges whether the data transmitted by the data collection unit has network danger or not according to the data transmitted by the analysis monitoring unit, if so, the early warning unit starts to early warn, if not, the data of the data collection unit is transmitted to a distributed database by the data processing unit, an analysis unit in the distributed database performs classification analysis on data, a comparison unit compares the data with the data in the distributed database, a preview unit previews data contents according to the comparison data, a presentation unit presents preview results, whether the data transmitted by a data acquisition unit has network danger or not is judged again, if the data transmitted by the data acquisition unit has the network danger, a prewarning unit starts to prewarn, and if the data transmitted by the data acquisition unit does not have the network danger, network information, network data and network contents uploaded or edited by different IPs are uploaded and backed up by a data acquisition unit, the network information safety prewarning platform can effectively monitor and analyze the safety risk in the network, meanwhile, an information safety prewarning technical research based on analysis of the network data, the network information and the network contents can be developed based on the network information safety prewarning platform, and a network and information safety event prewarning working mechanism is established, the data after being processed by the data processing unit can be transmitted to the backup unit for backup storage, and the data can be conveniently read and checked in the later period.
In a further technical scheme, the distributed database comprises an analysis unit, a comparison unit, a rehearsal unit and a presentation unit, data analyzed by the analysis unit is transmitted to the comparison unit, data compared by the comparison unit is transmitted to the rehearsal unit, and data previewed by the rehearsal unit is transmitted to the presentation unit for display.
The analysis unit in the distributed database carries out classification analysis on data, the comparison unit compares the data with the data in the distributed database, the preview unit previews data contents according to the comparison data, the presentation unit presents preview results, whether the data transmitted by the data acquisition unit is dangerous or not is judged again, if the data transmitted by the data acquisition unit is dangerous, the early warning unit starts up to carry out early warning, and if the data transmitted by the data acquisition unit is dangerous, the data acquisition unit uploads and backups network information, network data and network contents uploaded or edited by different IPs.
In a further technical scheme, the alarm unit comprises a sound-light alarm device and a remote popup window alarm.
When the alarm unit is triggered, the sound-light alarm device is started to emit buzzes and lights, a worker can be informed to check and the uploaded IP is locked, remote popup window alarm is divided into a worker popup window and an uploader popup window, the former can inform the worker to check and the uploaded IP is locked, and the latter can warn the uploaded IP and stop the action.
In a further technical solution, the data collection unit includes network information collection, network data collection, and network content collection.
The data acquisition unit can acquire network information, network data and network content uploaded or edited by different IPs through three modules of network information acquisition, network data acquisition and network content acquisition, and then respectively transmit the network information, the network data and the network content to the analysis monitoring unit and the transmission unit.
In a further technical solution, the analyzing and monitoring unit includes network information monitoring, network data monitoring, and network content monitoring.
The analysis monitoring unit monitors the network data, the network information and the network content transmitted by the data collection unit through three units of network information monitoring, network data monitoring and network content monitoring, and then transmits the monitoring data to the transmission unit.
In a further technical solution, the operation and maintenance unit includes an operation module, a management module, and a maintenance module.
The staff transmits the command data to the transmission unit through the operation module, the management module and the maintenance module in the operation and maintenance unit, the transmission unit transmits the command data to the data processing unit, and the data processing unit processes the command data and then performs self management and maintenance.
In a further technical scheme, the positioning unit adopts a high-precision global network IP positioning system.
The IP of uploading dangerous data is quickly positioned by the positioning unit, so that the network security maintainers in the later period can find the uploader in time conveniently, and the network security is improved.
In a further technical scheme, the input end of the distributed database is connected with a data acquisition unit, and the data acquisition unit comprises network data acquisition, manual input acquisition and self-editing acquisition.
Corresponding comparison data can be acquired from a network through network data acquisition to fill the distributed database, corresponding comparison data can be manually input into the distributed database through manual input acquisition to fill the distributed database, and corresponding comparison data in the distributed database can be automatically edited and acquired to fill the distributed database.
The invention also discloses a network information safety early warning method based on big data analysis, which comprises the following steps:
s1: the data collection unit collects information when different IPs upload or edit the information, and then transmits the information to the analysis monitoring unit and the transmission unit respectively;
s2: the analysis monitoring unit monitors the information transmitted by the data collection unit and transmits the monitoring data to the transmission unit;
s3: the transmission unit transmits the received data to the data processing unit, the data processing unit performs calculation processing on the data according to memory calculation and real-time flow calculation, whether network danger exists in the data transmitted by the data acquisition unit is judged according to the data transmitted by the analysis monitoring unit, if the network danger exists, the early warning unit is started to perform early warning, and if the network danger does not exist, the data processing unit transmits the data to the distributed database;
the data processing unit is specifically:
a) the memory calculation uses a Spark framework to realize data calculation based on the memory;
b) the real-time flow calculation is used for receiving and calculating the data of the data collection unit in real time, realizing the processing of cleaning and analyzing the data through calculation service, and outputting the result to a distributed database;
s4: the distributed database analyzes, compares, previews and presents the data, judges whether the data transmitted by the data acquisition unit is dangerous or not, if so, the early warning unit is started to carry out early warning, and if not, the data acquisition unit uploads and backups network information, network data and network content uploaded or edited by different IPs.
In a further technical solution, in the step S3, the memory calculation uses a Spark frame, and a Spark elastic distributed data set should be constructed, as follows:
s301: defining InputSpilt of a Spark distributed computing framework, realizing a subclass of Java class InputSpilt according to API provided by Spark, customizing a subclass of Java class InputFormat class, and realizing a Java class getSpilts method of the subclass;
s302: the sparkContext creates RDD according to the file/directory and the optional fragment, and firstly, an application program creates an instance of the sparkContext;
s303: creating Hadoop RDD according to the InputFormat in Hadoop configuration;
s304: defining a mode for reading InputSpilt data, and acquiring a Recordreader from an InputFormat according to Hadoop configuration and fragmentation to read data;
s305: and the subclass of the custom java class Recordreader is used for self-defining a mode of reading the fragment data and constructing a Spark distributed data set by combining the self-defined InputFormat.
In a further technical scheme, the real-time flow calculation adopts either a Storm framework or a Spark framework or a combination of the Storm framework and the Spark framework.
The invention has the beneficial effects that:
1. the network information, the network data and the network content uploaded or edited by different IPs are collected through a data collection unit and then are respectively transmitted to an analysis monitoring unit and a transmission unit, the analysis monitoring unit monitors the network data, the network information and the network content transmitted by the data collection unit through network information monitoring, network data monitoring and network content monitoring and then transmits the monitoring data to the transmission unit, the transmission unit transmits the data of the data collection unit and the analysis monitoring unit to a data processing unit, the data processing unit processes the data transmitted by the transmission unit, whether the data transmitted by the data collection unit has network danger or not is judged according to the data transmitted by the analysis monitoring unit, and if the data transmitted by the analysis monitoring unit has the network danger, an early warning unit is started to perform early warning.
2. The data processing unit can transmit the data of the data collecting unit to the distributed database, an analyzing unit in the distributed database carries out classification analysis on the data, the data in the comparison unit is compared with the data in the distributed database, the preview unit carries out preview on the data content according to the comparison data, the preview result is displayed by the display unit, whether the data transmitted by the data collecting unit is dangerous or not is judged again, if the data is dangerous, the early warning unit can be started to carry out early warning, and if the data is not dangerous, the data collecting unit can upload and backup network information, network data and network content uploaded or edited by different IPs.
3. The network information safety early warning platform can effectively monitor and analyze safety risks in a network, meanwhile, information safety early warning technology research based on network data, network information and network content analysis can be developed based on the network information safety early warning platform, a network and information safety event early warning working mechanism is established, and judgment basis is provided for network safety early warning and disposal.
Drawings
FIG. 1 is a block diagram of the system platform of the present invention;
FIG. 2 is a block diagram of a data acquisition unit of the present invention.
FIG. 3 is a schematic diagram of an analytical monitoring process according to the present invention.
FIG. 4 is a block diagram of the operation and maintenance process of the present invention.
Detailed Description
The embodiments of the present invention will be further described with reference to the accompanying drawings.
Example (b):
as shown in fig. 1-4, the network information security early warning platform based on big data analysis comprises a data collection unit, an operation and maintenance unit, a positioning unit, a transmission unit, a data processing unit, a distributed database, an analysis monitoring unit and an alarm unit, the output end of the data collection unit is respectively connected with the input ends of the transmission unit and the analysis monitoring unit, the output end of the analysis monitoring unit is connected with the input end of the transmission unit, the output end of the operation and maintenance unit is connected with the input end of the transmission unit, the output end of the transmission unit is connected with the input end of the data processing unit, the output end of the data processing unit is connected with the input end of the distributed database, the output ends of the data processing unit and the distributed database are connected with an alarm unit, and the output end of the data processing unit is connected with a display unit and a backup unit.
The network information, the network data and the network content uploaded or edited by different IPs are collected by a data collection unit and then are respectively transmitted to an analysis monitoring unit and a transmission unit, the analysis monitoring unit monitors the network data, the network information and the network content transmitted by the data collection unit by network information monitoring, network data monitoring and network content monitoring and then transmits the monitoring data to the transmission unit, the transmission unit transmits the data of the data collection unit and the analysis monitoring unit to a data processing unit, the data processing unit processes the data transmitted by the transmission unit and judges whether the data transmitted by the data collection unit has network danger or not according to the data transmitted by the analysis monitoring unit, if so, the early warning unit starts to early warn, if not, the data of the data collection unit is transmitted to a distributed database by the data processing unit, an analysis unit in the distributed database performs classification analysis on data, a comparison unit compares the data with the data in the distributed database, a preview unit previews data contents according to the comparison data, a presentation unit presents preview results, whether the data transmitted by a data acquisition unit has network danger or not is judged again, if the data transmitted by the data acquisition unit has the network danger, a prewarning unit starts to prewarn, and if the data transmitted by the data acquisition unit does not have the network danger, network information, network data and network contents uploaded or edited by different IPs are uploaded and backed up by a data acquisition unit, the network information safety prewarning platform can effectively monitor and analyze the safety risk in the network, meanwhile, an information safety prewarning technical research based on analysis of the network data, the network information and the network contents can be developed based on the network information safety prewarning platform, and a network and information safety event prewarning working mechanism is established, and a judgment basis is provided for network security early warning and disposal.
In another embodiment, as shown in fig. 1 to fig. 3, the distributed database includes an analysis unit, a comparison unit, a preview unit and a presentation unit, the data analyzed by the analysis unit is transmitted to the comparison unit, the data compared by the comparison unit is transmitted to the preview unit, the data previewed by the preview unit is transmitted to the presentation unit for display, the alarm unit includes an audible and visual alarm device and a remote pop-up window alarm, the data collection unit includes network information collection, network data collection and network content collection, the analysis and monitoring unit includes network information monitoring, network data monitoring and network content monitoring, the positioning unit employs a high-precision global network IP positioning system, the input end of the distributed database is connected with the data collection unit, the data collection unit includes network data collection, and a display unit, Manual entry collection and self-editing collection.
An analysis unit in the distributed database performs classification analysis on data, a comparison unit compares the data with the data in the distributed database, a preview unit previews data contents according to the comparison data, a presentation unit presents preview results, whether the data transmitted by a data acquisition unit has network danger or not is judged again, if the data transmitted by the data acquisition unit has the network danger, an early warning unit is started to perform early warning, if the data transmitted by the data acquisition unit does not have the danger, the data acquisition unit uploads or backups network information, network data and network contents uploaded or edited by different IPs, when an alarm unit is triggered, an acousto-optic alarm device is started to emit buzzing and light to inform a worker to check and lock the uploaded IPs, and remote popup window alarm is divided into a worker popup window and an uploader popup window which can inform the worker to check, the uploading IP is locked and can be warned by the later to stop the behavior, the data acquisition unit can collect the network information, the network data and the network content uploaded or edited by different IPs through three modules of network information acquisition, network data acquisition and network content acquisition and then respectively transmit the collected network information, the network data and the network content to the analysis monitoring unit and the transmission unit, the analysis monitoring unit monitors the network data, the network information and the network content transmitted by the data acquisition unit through three units of network information monitoring, network data monitoring and network content monitoring and then transmits the monitored data to the transmission unit, the IP for uploading dangerous data can be quickly positioned through the positioning unit, the later network safety maintainers can conveniently find the uploader in time, the network safety is improved, the corresponding comparison data can be acquired from the network through the network data acquisition to fill the distributed database, the corresponding contrast data is manually input into the distributed database to fill the distributed database, the corresponding contrast data is greatly input into the distributed database through self-editing and collecting, the data processed by the data processing unit can be transmitted to the display unit to be displayed, workers and other network users can conveniently check the data, the data processed by the data processing unit can be transmitted to the backup unit to be backed up and stored, and later-stage retrieval and checking are conveniently carried out.
In another embodiment, as shown in fig. 4, the operation and maintenance unit includes an operation module, a management module, and a maintenance module.
The staff transmits the command data to the transmission unit through the operation module, the management module and the maintenance module in the operation and maintenance unit, the transmission unit transmits the command data to the data processing unit, and the data processing unit processes the command data and then performs self management and maintenance.
Example 2:
on the basis of embodiment 1, as shown in fig. 1 to 4, the early warning method of the network information security early warning platform based on big data analysis includes the following steps:
s1: the data collection unit collects information when different IPs upload or edit the information, and then transmits the information to the analysis monitoring unit and the transmission unit respectively.
S2: the analysis monitoring unit monitors the information transmitted by the data collection unit and transmits the monitoring data to the transmission unit.
S3: the transmission unit transmits the received data to the data processing unit, the data processing unit performs calculation processing on the data according to memory calculation and real-time flow calculation, whether network danger exists in the data transmitted by the data acquisition unit is judged according to the data transmitted by the analysis monitoring unit, if the network danger exists, the early warning unit is started to perform early warning, and if the network danger does not exist, the data processing unit transmits the data to the distributed database;
the data processing unit is specifically:
a) the memory calculation uses a Spark framework to realize data calculation based on the memory;
b) the real-time flow calculation is used for receiving and calculating the data of the data collection unit in real time, realizing the processing of cleaning and analyzing the data through calculation service, and outputting the result to a distributed database;
wherein, the real-time flow calculation adopts any one of a Storm framework, a Spark framework or a combination of the Storm framework and the Spark framework; the flow computing mode can well analyze large-scale flow data in real time in the changing motion process, capture possibly useful information and send the result to the next computing node.
S4: the distributed database analyzes, compares, previews and presents the data, judges whether the data transmitted by the data acquisition unit is dangerous or not, if so, the early warning unit is started to carry out early warning, and if not, the data acquisition unit uploads and backups network information, network data and network content uploaded or edited by different IPs.
In another embodiment, the memory calculation in step S3 uses a Spark framework, and a Spark elastic distributed data set should be constructed, as follows:
s301: defining InputSpilt of a Spark distributed computing framework, realizing a subclass of Java class InputSpilt according to API provided by Spark, customizing a subclass of Java class InputFormat class, and realizing a Java class getSpilts method of the subclass;
s302: the sparkContext creates RDD according to the file/directory and the optional fragment, and firstly, an application program creates an instance of the sparkContext;
s303: creating Hadoop RDD according to the InputFormat in Hadoop configuration;
s304: defining a mode for reading InputSpilt data, and acquiring a Recordreader from an InputFormat according to Hadoop configuration and fragmentation to read data;
s305: and the subclass of the custom java class Recordreader is used for self-defining a mode of reading the fragment data and constructing a Spark distributed data set by combining the self-defined InputFormat.
The above-mentioned embodiments only express the specific embodiments of the present invention, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the inventive concept, which falls within the scope of the present invention.
Claims (10)
1. The network information safety early warning platform based on big data analysis is characterized by comprising a data collection unit, an operation and maintenance unit, a positioning unit, a transmission unit, a data processing unit, a distributed database, an analysis monitoring unit and an alarm unit, the output end of the data collection unit is respectively connected with the input ends of the transmission unit and the analysis monitoring unit, the output end of the analysis monitoring unit is connected with the input end of the transmission unit, the output end of the operation and maintenance unit is connected with the input end of the transmission unit, the output end of the transmission unit is connected with the input end of the data processing unit, the output end of the data processing unit is connected with the input end of the distributed database, the output ends of the data processing unit and the distributed database are connected with an alarm unit, and the output end of the data processing unit is connected with a display unit and a backup unit.
2. The big data analysis-based network information safety early warning platform according to claim 1, wherein the distributed database comprises an analysis unit, a comparison unit, a rehearsal unit and a presentation unit, the data analyzed by the analysis unit is transmitted to the comparison unit, the data compared by the comparison unit is transmitted to the rehearsal unit, and the data previewed by the rehearsal unit is transmitted to the presentation unit for display;
the input end of the distributed database is connected with a data acquisition unit, and the data acquisition unit comprises network data acquisition, manual input acquisition and self-editing acquisition.
3. The big data analysis-based network information security early warning platform according to claim 2, wherein the alarm unit comprises an audible and visual alarm device and a remote pop-up window alarm.
4. The big data analysis-based network information security early warning platform according to claim 3, wherein the data collection unit comprises network information collection, network data collection, and network content collection.
5. The big data analysis-based network information security early warning platform according to claim 4, wherein the analysis monitoring unit comprises network information monitoring, network data monitoring and network content monitoring.
6. The big data analysis-based network information security early warning platform according to claim 5, wherein the operation and maintenance unit comprises an operation module, a management module and a maintenance module.
7. The big data analysis-based network information security early warning platform according to claim 6, wherein the positioning unit employs a high-precision global network IP positioning system.
8. The network information safety early warning method based on big data analysis is characterized by comprising the following steps:
s1: the data collection unit collects information when different IPs upload or edit the information, and then transmits the information to the analysis monitoring unit and the transmission unit respectively;
s2: the analysis monitoring unit monitors the information transmitted by the data collection unit and transmits the monitoring data to the transmission unit;
s3: the transmission unit transmits the received data to the data processing unit, the data processing unit performs calculation processing on the data according to memory calculation and real-time flow calculation, whether network danger exists in the data transmitted by the data acquisition unit is judged according to the data transmitted by the analysis monitoring unit, if the network danger exists, the early warning unit is started to perform early warning, and if the network danger does not exist, the data processing unit transmits the data to the distributed database;
the data processing unit is specifically:
a) the memory calculation uses a Spark framework to realize data calculation based on the memory;
b) the real-time flow calculation is used for receiving and calculating the data of the data collection unit in real time, realizing the processing of cleaning and analyzing the data through calculation service, and outputting the result to a distributed database;
the flow computing mode can well analyze large-scale flow data in real time in the changing motion process, capture possibly useful information and send the result to the next computing node.
S4: the distributed database analyzes, compares, previews and presents the data, judges whether the data transmitted by the data acquisition unit is dangerous or not, if so, the early warning unit is started to carry out early warning, and if not, the data acquisition unit uploads and backups network information, network data and network content uploaded or edited by different IPs.
9. The big data analysis-based network information security early warning and early warning method according to claim 8, wherein in the step S3, the memory calculation uses a Spark framework, and a Spark elastic distributed data set is constructed, as follows:
s301: defining InputSpilt of a Spark distributed computing framework, realizing a subclass of Java class InputSpilt according to API provided by Spark, customizing a subclass of Java class InputFormat class, and realizing a Java class getSpilts method of the subclass;
s302: the sparkContext creates RDD according to the file/directory and the optional fragment, and firstly, an application program creates an instance of the sparkContext;
s303: creating Hadoop RDD according to the InputFormat in Hadoop configuration;
s304: defining a mode for reading InputSpilt data, and acquiring a Recordreader from an InputFormat according to Hadoop configuration and fragmentation to read data;
s305: and the subclass of the custom java class Recordreader is used for self-defining a mode of reading the fragment data and constructing a Spark distributed data set by combining the self-defined InputFormat.
10. The big data analysis-based network information security early warning and early warning method according to claim 8, wherein the real-time flow calculation adopts any one of a Storm framework and a Spark framework alone or a combination of the Storm framework and the Spark framework.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111390236.7A CN114157463A (en) | 2021-11-23 | 2021-11-23 | Big data analysis-based network information security early warning platform and early warning method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111390236.7A CN114157463A (en) | 2021-11-23 | 2021-11-23 | Big data analysis-based network information security early warning platform and early warning method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114157463A true CN114157463A (en) | 2022-03-08 |
Family
ID=80457299
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111390236.7A Pending CN114157463A (en) | 2021-11-23 | 2021-11-23 | Big data analysis-based network information security early warning platform and early warning method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114157463A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115373361A (en) * | 2022-10-24 | 2022-11-22 | 江苏智云天工科技有限公司 | Factory production safety early warning method and system based on industrial Internet |
| CN115987695A (en) * | 2023-03-21 | 2023-04-18 | 融科联创(天津)信息技术有限公司 | Network security monitoring system based on big data analysis |
| CN116318985A (en) * | 2023-03-02 | 2023-06-23 | 中承信达(天津)技术股份公司 | Computer network security early warning system and method based on big data |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2013075297A1 (en) * | 2011-11-23 | 2013-05-30 | 湖南深拓智能设备股份有限公司 | Remote real-time monitoring system based on cloud computing |
| CN103401699A (en) * | 2013-07-18 | 2013-11-20 | 深圳先进技术研究院 | Cloud data center security monitoring early warning system and method |
| CN104394415A (en) * | 2014-12-09 | 2015-03-04 | 中国电子科技集团公司第二十八研究所 | Method for distributed decoding of video big data |
| CN105718804A (en) * | 2015-12-25 | 2016-06-29 | 驻马店职业技术学院 | Hardware instruction call control system |
| CN110138812A (en) * | 2019-07-11 | 2019-08-16 | 南昌诺汇医药科技有限公司 | Network Safety Analysis system |
| CN111721355A (en) * | 2020-05-14 | 2020-09-29 | 中铁第一勘察设计院集团有限公司 | Railway contact net monitoring data acquisition system |
| CN113569254A (en) * | 2021-07-27 | 2021-10-29 | 黑龙江祥辉通信工程有限公司 | High-safety network information protection system and protection method thereof |
-
2021
- 2021-11-23 CN CN202111390236.7A patent/CN114157463A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2013075297A1 (en) * | 2011-11-23 | 2013-05-30 | 湖南深拓智能设备股份有限公司 | Remote real-time monitoring system based on cloud computing |
| CN103401699A (en) * | 2013-07-18 | 2013-11-20 | 深圳先进技术研究院 | Cloud data center security monitoring early warning system and method |
| CN104394415A (en) * | 2014-12-09 | 2015-03-04 | 中国电子科技集团公司第二十八研究所 | Method for distributed decoding of video big data |
| CN105718804A (en) * | 2015-12-25 | 2016-06-29 | 驻马店职业技术学院 | Hardware instruction call control system |
| CN110138812A (en) * | 2019-07-11 | 2019-08-16 | 南昌诺汇医药科技有限公司 | Network Safety Analysis system |
| CN111721355A (en) * | 2020-05-14 | 2020-09-29 | 中铁第一勘察设计院集团有限公司 | Railway contact net monitoring data acquisition system |
| CN113569254A (en) * | 2021-07-27 | 2021-10-29 | 黑龙江祥辉通信工程有限公司 | High-safety network information protection system and protection method thereof |
Non-Patent Citations (1)
| Title |
|---|
| 翁跃鑫;聂小伟;林海祥;吕震东;刘跃鸿;: "基于大数据的网络信息安全分析平台应用研究", 电脑知识与技术, no. 07, 5 March 2019 (2019-03-05) * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115373361A (en) * | 2022-10-24 | 2022-11-22 | 江苏智云天工科技有限公司 | Factory production safety early warning method and system based on industrial Internet |
| CN116318985A (en) * | 2023-03-02 | 2023-06-23 | 中承信达(天津)技术股份公司 | Computer network security early warning system and method based on big data |
| CN115987695A (en) * | 2023-03-21 | 2023-04-18 | 融科联创(天津)信息技术有限公司 | Network security monitoring system based on big data analysis |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114157463A (en) | Big data analysis-based network information security early warning platform and early warning method | |
| CN108763957B (en) | Database security audit system, method and server | |
| CN103491354A (en) | System operation monitoring and controlling visual platform | |
| CN108848067A (en) | The OPC protocol security means of defence of intelligence learning and preset read-only white list rule | |
| CN110300100A (en) | The association analysis method and system of log audit | |
| CN108270716A (en) | A kind of audit of information security method based on cloud computing | |
| KR20160008267A (en) | User's behavior analysis system on the network based video surveillance system | |
| CN110351277A (en) | Electric power monitoring system security protection alarm method | |
| CN111770108A (en) | Network safety system based on artificial intelligence | |
| CN114143064B (en) | Multi-source network security alarm event tracing and automatic disposal method and device | |
| CN111178828A (en) | Method and system for building fire safety early warning | |
| CN107105017A (en) | A kind of well site of oil field information management monitoring system | |
| KR20030056652A (en) | Blacklist management apparatus in a policy-based network security management system and its proceeding method | |
| CN105869367B (en) | Fault calibration method based on networking video monitoring | |
| CN111147521A (en) | Enterprise private network security event management system | |
| CN116861419B (en) | Active defending log alarming method on SSR | |
| CN103618635A (en) | Classified association management and control system of informationized equipment | |
| CN117375985A (en) | Method and device for determining security risk index, storage medium and electronic device | |
| CN117220917A (en) | Network real-time monitoring method based on cloud computing | |
| CN113162897A (en) | Industrial control network security filtering system and method | |
| CN116614258A (en) | Network danger prediction model of security situation awareness system | |
| CN115622873A (en) | Comprehensive log analysis system | |
| CN104376254A (en) | Method and system for auditing log | |
| CN111049853A (en) | Security authentication system based on computer network | |
| KR101973728B1 (en) | Integration security anomaly symptom monitoring system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |