CN115766135A - Network monitoring system and method for federal learning - Google Patents
Network monitoring system and method for federal learning Download PDFInfo
- Publication number
- CN115766135A CN115766135A CN202211367168.7A CN202211367168A CN115766135A CN 115766135 A CN115766135 A CN 115766135A CN 202211367168 A CN202211367168 A CN 202211367168A CN 115766135 A CN115766135 A CN 115766135A
- Authority
- CN
- China
- Prior art keywords
- task
- data
- flow
- site
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
A network monitoring system and method facing federal study, through configuring iptables 'fire wall rule and timing data capture task through the monitoring module of communication flow of site crossing, count the time-sharing communication flow among different sites in the state of Council study, and analyze each task, flow data that users use and return to the user's front end in the form of data interface and realize visualization; the method comprises the steps that a local station task monitoring module is deployed rapidly through a docker container, a federal learning external task statistical interface is called regularly, task multi-dimensional information of a federal learning task participated by a federal learning local station is recorded, and a task data interface is provided for displaying various task information. The invention monitors the federal learning platform, counts the specific information of the federal learning task, ensures the authenticity and correctness of the monitoring task, monitors the compliance, the legality and the efficiency of the federal learning task, and facilitates the daily management and analysis of platform managers.
Description
Technical Field
The invention relates to a technology in the field of network security, in particular to a network monitoring system and a network monitoring method facing federal learning.
Background
Federal learning is a novel distributed machine learning framework, is characterized by decentralization, law and regulation satisfaction and privacy protection, and has wide prospect. However, due to the characteristics of multi-party participation and strong privacy, network monitoring and flow analysis of the federal learning platform are very difficult, and the monitoring method of the traditional machine learning platform cannot meet the requirements and is difficult to effectively perform platform supervision, task analysis and privacy protection.
Disclosure of Invention
The invention provides a network monitoring system and a method facing federal learning, aiming at the defects of poor reusability, violation of federal learning principle, high cost of monitoring flow, high vulnerability of identity authentication based on a secret key technology and poor reliability of monitoring in the prior art.
The invention is realized by the following technical scheme:
the invention relates to a network monitoring system facing federal learning, which comprises: cross site communication flow monitoring module, local site task monitoring module, wherein: the cross-site communication flow monitoring module counts time-sharing communication flows among different sites in the Federation study by configuring iptables firewall rules and timing data capturing tasks, analyzes the flow data used by each task and a user and returns the flow data to the front end of the user in a data interface form to realize visualization; the local station task monitoring module is deployed rapidly through a docker container, calls a federal learning external task statistical interface regularly, records task multi-dimensional information of a federal learning task participated by the station of the federal learning, and provides a task data interface to display various task information.
The cross-site communication traffic monitoring module comprises: flow monitoring unit, flow data capture unit, flow data analysis unit and flow data report unit, wherein: the flow monitoring unit configures iptables firewall rules according to configuration parameters so as to implement different flow statistical rules according to different local node units, and the flow data capturing unit performs timing task configuration on the cron task scheduler and executes a timing data capturing task according to the specified rules of the flow monitoring unit so as to obtain time-sharing and task-dividing flow data; the flow data analysis unit is used for counting time-sharing communication flow among different stations in the federal study to obtain time-sharing communication flow data, performing reliability analysis on the flow data obtained by the flow data capturing unit and generating a statistical report; the flow data reporting unit provides a data query interface for the client, analyzes each task, the time-interval used by the user and the task-divided flow data, returns the data to the front end of the user in a data interface form, and realizes visualization on the interface.
The local site task monitoring module comprises: the system comprises a database operation unit, a task data capturing unit and a task data reporting unit, wherein: the database operation unit creates a database table structure supporting data adding, deleting and modifying operations, the task data capturing unit calls a task data interface and captures and analyzes task data, the task data reporting unit provides a data query interface for a client and sorts the task data to obtain task multi-dimensional information of a federal learning task participated by the local site unit, and various task information is displayed through the task data interface.
The task multidimensional information includes but is not limited to: task id, task category, task state, task participant, task duration, task start time, etc.
The invention relates to a federal learning-oriented network monitoring method based on the system, which comprises the following steps:
step 1, installing system components by all local nodes participating in federal learning through a rapid deployment module, wherein the system components comprise a cross-site communication flow monitoring module and a local site task monitoring module;
step 2, the local site task monitoring module monitors the federal learning task participated by the local node in the federal learning process, and counts and analyzes the local node participated task according to the log recorded by the federal learning;
step 3, the local site task monitoring module stores the obtained data into a database storage unit;
step 4, the cross-site communication flow monitoring module configures firewall rules according to the environment and initial parameters of the local nodes;
step 5, the cross-site communication flow monitoring module utilizes the firewall and the task scheduler to count time-sharing flow data passing through the local node;
and 6, the local node obtains task statistical data and flow statistical data of the local node participating in the federal learning process through monitoring results of the local node task monitoring module and the cross-site communication flow monitoring module.
Drawings
FIG. 1 is a schematic diagram of the system of the present invention;
FIG. 2 is a schematic diagram of a cross-site communication traffic monitoring module according to the present invention;
FIG. 3 is a schematic diagram of a local site task monitoring module;
FIG. 4 is a schematic diagram of a rapid deployment module;
FIG. 5 is a schematic diagram of a database storage unit;
FIG. 6 is a diagram illustrating an effect of cross-site traffic monitoring in an embodiment;
FIG. 7 is a diagram illustrating the effect of task monitoring at a local site in the embodiment;
FIG. 8 is a flow chart of a federated learning-oriented network monitoring method.
Detailed Description
As shown in fig. 1, the present embodiment relates to a network monitoring system facing federal learning, which includes: the system comprises a cross-site communication flow monitoring module, a local site task monitoring module, a rapid deployment module and a database storage unit, wherein: the cross-site communication flow monitoring module monitors and acquires flow information through a port and outputs the flow information to the database storage unit, and meanwhile receives data from the database storage unit for statistical analysis; the local site task monitoring module acquires task information through monitoring of local nodes, stores the task information into the slave database storage unit, regularly calls the task data grabbing unit to acquire task data from the database storage unit according to flow information, carries out statistics and processing to obtain task information, and the cross-site communication flow monitoring module and the local site task monitoring module achieve local loading through the rapid deployment module.
The cross-site communication traffic monitoring module comprises: flow monitoring unit, flow data capture unit, flow data analysis unit and flow data report unit, wherein: the flow monitoring unit carries out rule configuration on the iptables firewall; the flow data capturing unit carries out timing task configuration on the cron task scheduler; the flow data analysis unit counts time-sharing communication flow among different stations in federal learning to obtain time-sharing communication flow data, performs reliability analysis on the flow data obtained by the flow data capturing unit, and generates a statistical report; the flow data reporting unit provides a data query interface for the client, and the client displays the data to the user after visualizing the data.
The rule configuration comprises:
1) Logging in a federal learning communication node through a super user;
2) Adding rules in an INPUT chain by using an iptables command, and designating a target port needing to be monitored for counting inflow flow;
3) Rules are added on the OUTPUT chain using iptables commands and the source ports that need to be monitored are specified for statistical egress traffic.
The timing task configuration comprises the following steps:
1) Setting a database connection method of a regularly executed script, guarding an iptables rule against tampering, reading flow information when executing the script, adding a timestamp and storing the timestamp into a database;
2) And writing a timed task by using a crontab command under the authority of a super user, and executing the data capture script once every minute.
The step of displaying the visualized data to the user comprises the following steps:
1) The client sends query information to the/traffic URL interface by a POST method, wherein the query information comprises the statistical start-stop time and the statistical interval;
2) The server side firstly carries out unified conversion on a time format, modifies a database query command according to a required time interval, then queries a database flow record data table to obtain flow monitoring information in a corresponding time period, acquires task information and user information of each interval, and finally returns the number of flow bytes and network packets as well as the flow consumed by each task and user according to the time format;
3) And the client converts the flow monitoring information into a statistical chart to be displayed to the user.
The local site task monitoring module comprises: the system comprises a database operation unit, a task data capturing unit and a task data reporting unit, wherein: the database operation unit creates a database table structure supporting data adding, deleting, searching and modifying operations, the task data capturing unit calls a task data interface and captures and analyzes task data, the task data reporting unit provides a data query interface for the client side, and the client side displays the task data to the user in a visualized and classified mode.
The creating comprises:
1) A traffic record data table for storing cross-site traffic information in time order;
the cross-site traffic information comprises: timestamp, number of incoming network packets, number of incoming bytes, number of outgoing network packets, and number of outgoing bytes.
2) The task recording data table is used for storing various task information performed by the site;
the task information comprises: task id, task creation time, task duration, task status, task participant information, etc.
3) And the station information recording table stores the local station information.
The local site information includes: site name, site partid, creation update time, FATE version information, site status.
The data of the grabbing analysis task comprises the following steps:
1) Acquiring task data, namely acquiring task information in a time period through a FATE Flow interface, wherein the task information comprises a completed task and an ongoing task;
2) And storing the task data, storing the acquired task data into a database, and updating the current statistical time period to ensure that the tasks are not repeatedly counted.
The visualization and classification display comprises the following steps:
1) The client sends query information to/jobURL interface by POST method, including counting start and stop date and participant id;
2) The server side performs unified conversion on a time format to generate a date list, queries a database task record data table according to submitted participant ids, acquires each task information performed by a local site, groups the task information according to task states and task categories, calculates task success rate and task average operation time, and finally returns task information of each day according to the categories;
3) And converting the task monitoring information into a statistical chart between the clients and displaying the statistical chart to the user.
The local loading comprises the following steps:
(1) and deploying a cross-site communication traffic monitoring module and a local site task monitoring module required by network monitoring into a local server, wherein the cross-site communication traffic monitoring module and the local site task monitoring module are used for counting local participating federal learning tasks and monitoring traffic of cross-site communication through local nodes.
(2) Packing a cross-site communication traffic monitoring module and a local site task monitoring module in a container, and installing a network monitoring system comprising the cross-site communication traffic monitoring module and the local site task monitoring module in a local node unit;
(3) and automatically generating configuration parameters of the local node unit and the network monitoring system according to the environments of different local node units.
The database storage unit is a matched data storage unit facing the Federal learning network monitoring system, and stores the task data and the flow data acquired by the cross-site communication flow monitoring module and the local site task monitoring module into a database of the local node unit in a specific form.
Through specific practical experiments, the method is realized on a Ubuntu 18.04 operating system platform with a 64-bit architecture, and comprises a cross-site communication flow monitoring module based on an iptables firewall and a local site task monitoring module based on a docker.
The system is rapidly deployed in the following modes: modifying the configuration file, namely writing the participant id, the site name, the database user and the password of the local site into the configuration file; bash running a flow monitoring deployment script, and adding a flow monitoring module on a federated learning system framework; bash runs a task monitoring deployment script, starts a task monitoring container by using a docker command, and adds the container to a network of the federal learning system.
The configuration of each environment of the object host in this embodiment is shown in the following table:
among the above modules, the cross-site communication traffic monitoring module based on iptables is the original creation of the invention. By using the iptables, the statistical analysis is carried out on the task flow usage aiming at the special communication scene of the federal study, and a data support is provided for platform supervision, task analysis and data safety guarantee of the federal study, so that the method has a certain effect and practicability.
The foregoing embodiments may be modified in many different ways by those skilled in the art without departing from the spirit and scope of the invention, which is defined by the appended claims and all changes that come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein.
Claims (9)
1. A federally-learned network monitoring system, comprising: cross site communication flow monitoring module, local site task monitoring module, wherein: the cross-site communication flow monitoring module counts time-sharing communication flows among different sites in the Federation study by configuring iptables firewall rules and timing data capturing tasks, analyzes the flow data used by each task and a user and returns the flow data to the front end of the user in a data interface form to realize visualization; the local station task monitoring module is deployed rapidly through a docker container, calls a federal learning external task statistical interface regularly, records task multi-dimensional information of a federal learning task participated by the station of the federal learning, and provides a task data interface to display various task information.
2. The federal learning oriented network monitoring system as claimed in claim 1, wherein the cross-site communication traffic monitoring module comprises: flow monitoring unit, flow data snatch unit, flow data analysis unit and flow data report unit, wherein: the flow monitoring unit configures iptables firewall rules according to configuration parameters so as to implement different flow statistical rules according to different local node units, and the flow data capturing unit performs timing task configuration on the cron task scheduler and executes a timing data capturing task according to the specified rules of the flow monitoring unit so as to obtain time-sharing and task-dividing flow data; the flow data analysis unit counts time-sharing communication flow among different stations in federal learning to obtain time-sharing communication flow data, performs reliability analysis on the flow data obtained by the flow data capturing unit, and generates a statistical report; the flow data reporting unit provides a data query interface for the client, analyzes each task, time-interval used by the user and task-divided flow data, and returns the data to the front end of the user in a data interface form, so that visualization is realized on the interface.
3. The federal learning oriented network monitoring system as claimed in claim 1, wherein said local site mission monitoring module comprises: the system comprises a database operation unit, a task data capturing unit and a task data reporting unit, wherein: the database operation unit creates a database table structure supporting data adding, deleting and modifying operations, the task data capturing unit calls a task data interface and captures and analyzes task data, the task data reporting unit provides a data query interface for a client and sorts the task data to obtain task multi-dimensional information of a federal learning task participated by the local site unit, and various task information is displayed through the task data interface.
4. A federal learning oriented network monitoring system as claimed in any of claims 1-3, further comprising a rapid deployment module and a database storage unit, wherein: the cross-site communication flow monitoring module monitors and acquires flow information through a port and outputs the flow information to the database storage unit, and meanwhile receives data from the database storage unit for statistical analysis; the local site task monitoring module acquires task information through monitoring of local nodes, stores the task information into the slave database storage unit, regularly calls the task data capturing unit to acquire task data from the database storage unit according to flow information, performs statistics and processing to obtain task information, and the cross-site communication flow monitoring module and the local site task monitoring module achieve local loading through the rapid deployment module.
5. A federally learned network monitor system as in claim 4, wherein said rule configuration includes:
1) Logging in a federal learning communication node through a super user;
2) Adding rules in an INPUT chain by using an iptables command, and designating a target port needing to be monitored for counting inflow flow;
3) Adding rules on an OUTPUT chain by using an iptables command, and designating a source port needing to be monitored for counting outflow;
the timing task configuration comprises the following steps:
1) Setting a database connection method of a regularly executed script, guarding that an iptables rule is not tampered, reading flow information when executing the script, adding a timestamp and storing the timestamp into a database;
2) Writing a timed task by using a crontab command under the authority of a super user, and executing the data capture script once every minute;
the creating comprises:
1) A traffic record data table for storing cross-site traffic information in time order;
the cross-site traffic information comprises: time stamp, number of network packets flowing in, number of bytes flowing in, number of network packets flowing out, and number of bytes flowing out;
2) The task recording data table is used for storing various task information performed by the site;
the task information comprises: task id, task creation time, task duration, task state, task participant information and the like;
3) A site information recording table for storing the site information;
the local site information includes: site name, site party id, creation update time, FATE version information and site state;
the data of the grabbing analysis task comprises the following steps:
1) Acquiring task data, namely acquiring task information in a time period through a FATE Flow interface, wherein the task information comprises a completed task and an ongoing task;
2) And storing the task data, storing the acquired task data into a database, and updating the current statistical time period to ensure that the tasks are not repeatedly counted.
6. The federal learning oriented network monitoring system as claimed in claim 4, wherein said visualizing the data for presentation to the user comprises:
1) The client sends query information to the/traffic URL interface by using a POST method, wherein the query information comprises statistical start-stop time and statistical intervals;
2) The server side firstly carries out unified conversion on a time format, modifies a database query command according to a required time interval, then queries a database flow record data table to obtain flow monitoring information in a corresponding time period, acquires task information and user information of each interval, and finally returns the number of flow bytes and network packets as well as the flow consumed by each task and user according to the time format;
3) And the client converts the flow monitoring information into a statistical chart to be displayed to the user.
7. A federal learning oriented network monitoring system as in claim 4, wherein said local loading comprises:
(1) deploying a cross-site communication flow monitoring module and a local site task monitoring module required by network monitoring into a local server, and counting federal learning tasks participated in locally and monitoring the flow of cross-site communication through local nodes;
(2) packing a cross-site communication traffic monitoring module and a local site task monitoring module in a container, and installing a network monitoring system comprising the cross-site communication traffic monitoring module and the local site task monitoring module in a local node unit;
(3) and automatically generating configuration parameters of the local node unit and the network monitoring system according to the environments of different local node units.
8. The system according to claim 4, wherein the database storage unit is a matching data storage unit of the system, and the database storage unit stores the task data and the traffic data acquired by the cross-site communication traffic monitoring module and the local site task monitoring module in a specific form in a database of the local node unit.
9. A network monitoring method based on the federal learning oriented network monitoring system in any one of claims 1 to 8, comprising:
step 1, all local nodes participating in federal learning install system components through a rapid deployment module, wherein the system components comprise a cross-site communication flow monitoring module and a local site task monitoring module;
step 2, the local site task monitoring module monitors the federal learning task participated by the local node in the federal learning process, and counts and analyzes the local node participated task according to the log recorded by the federal learning;
step 3, the local site task monitoring module stores the obtained data into a database storage unit;
step 4, the cross-site communication flow monitoring module configures firewall rules according to the environment and initial parameters of the local nodes;
step 5, the cross-site communication flow monitoring module utilizes the firewall and the task scheduler to count time-sharing flow data passing through the local node;
and 6, the local node obtains task statistical data and flow statistical data of the local node participating in the federal learning process through monitoring results of the local node task monitoring module and the cross-site communication flow monitoring module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211367168.7A CN115766135A (en) | 2022-11-02 | 2022-11-02 | Network monitoring system and method for federal learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211367168.7A CN115766135A (en) | 2022-11-02 | 2022-11-02 | Network monitoring system and method for federal learning |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115766135A true CN115766135A (en) | 2023-03-07 |
Family
ID=85356126
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211367168.7A Pending CN115766135A (en) | 2022-11-02 | 2022-11-02 | Network monitoring system and method for federal learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115766135A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117714217A (en) * | 2024-02-06 | 2024-03-15 | 河北数云堂智能科技有限公司 | Method and device for trusted federal intelligent security computing platform |
-
2022
- 2022-11-02 CN CN202211367168.7A patent/CN115766135A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117714217A (en) * | 2024-02-06 | 2024-03-15 | 河北数云堂智能科技有限公司 | Method and device for trusted federal intelligent security computing platform |
| CN117714217B (en) * | 2024-02-06 | 2024-05-28 | 河北数云堂智能科技有限公司 | Method and device for trusted federal intelligent security computing platform |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108197565A (en) | Target based on recognition of face seeks track method and system | |
| Gutzwiller et al. | A task analysis toward characterizing cyber-cognitive situation awareness (CCSA) in cyber defense analysts | |
| Barbosa et al. | Exploiting traffic periodicity in industrial control networks | |
| US20160191549A1 (en) | Rich metadata-based network security monitoring and analysis | |
| CN107196910A (en) | Threat early warning monitoring system, method and the deployment framework analyzed based on big data | |
| CN101197715B (en) | Method for centrally capturing mobile data service condition | |
| CN103338128A (en) | Information security management system with integrated security management and control function | |
| CN104246786A (en) | Field selection for pattern discovery | |
| CN106779485B (en) | SOA architecture-based comprehensive management system and data processing method | |
| CN106452955B (en) | A kind of detection method and system of abnormal network connection | |
| Fink et al. | Visual correlation of host processes and network traffic | |
| CN103391425B (en) | Monitoring intelligent remodeling method based on time division multiplex video analysis | |
| CN106055984A (en) | Classified management method applied to security baseline software | |
| CN110209723A (en) | A kind of equipment information collection system based on Internet of Things big data | |
| CN115766135A (en) | Network monitoring system and method for federal learning | |
| El Arass et al. | Smart SIEM: From big data logs and events to smart data alerts | |
| CN114448654B (en) | Block chain-based distributed trusted audit security evidence storing method | |
| CN114244676A (en) | Intelligent IT integrated gateway system | |
| CN111131332A (en) | Network service interconnection and flow acquisition, analysis and recording system | |
| CN104239178A (en) | Monitoring system and monitoring method | |
| Xin et al. | Fuzzy feature extraction and visualization for intrusion detection | |
| CN209086928U (en) | A kind of deployment architecture of database audit | |
| Pödör et al. | Industrial IoT techniques and solutions in wood industrial manufactures | |
| CN112465480A (en) | A real name system management system of labor affairs for building trade | |
| CN110708341A (en) | User behavior detection method and system based on remote desktop encryption network traffic mode difference |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |