CN115249149A - Data circulation system, safety management and control system and safety management and control method thereof - Google Patents
Data circulation system, safety management and control system and safety management and control method thereof Download PDFInfo
- Publication number
- CN115249149A CN115249149A CN202211146636.8A CN202211146636A CN115249149A CN 115249149 A CN115249149 A CN 115249149A CN 202211146636 A CN202211146636 A CN 202211146636A CN 115249149 A CN115249149 A CN 115249149A
- Authority
- CN
- China
- Prior art keywords
- data
- subsystem
- auditing
- safety
- control
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 53
- 230000008569 process Effects 0.000 claims abstract description 29
- 238000012545 processing Methods 0.000 claims abstract description 14
- 238000004891 communication Methods 0.000 claims abstract description 4
- 238000007726 management method Methods 0.000 claims description 81
- 238000011161 development Methods 0.000 claims description 17
- 238000004140 cleaning Methods 0.000 claims description 14
- 238000004364 calculation method Methods 0.000 claims description 13
- 230000010354 integration Effects 0.000 claims description 13
- 238000012423 maintenance Methods 0.000 claims description 12
- 238000013480 data collection Methods 0.000 claims description 8
- 238000013500 data storage Methods 0.000 claims description 6
- 230000002776 aggregation Effects 0.000 claims description 5
- 238000004220 aggregation Methods 0.000 claims description 5
- 238000001914 filtration Methods 0.000 claims description 4
- 230000003993 interaction Effects 0.000 claims description 4
- 230000000007 visual effect Effects 0.000 claims description 4
- 230000005540 biological transmission Effects 0.000 claims description 3
- 230000002427 irreversible effect Effects 0.000 claims description 3
- 238000005259 measurement Methods 0.000 claims description 3
- 230000035945 sensitivity Effects 0.000 claims description 3
- 238000012550 audit Methods 0.000 description 11
- 238000013461 design Methods 0.000 description 5
- 241001481833 Coryphaena hippurus Species 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000013499 data model Methods 0.000 description 2
- 208000003443 Unconsciousness Diseases 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/10—Office automation; Time management
- G06Q10/103—Workflow collaboration or project management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/602—Providing cryptographic facilities or services
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
- G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
- G06F21/6245—Protecting personal data, e.g. for financial or medical purposes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0631—Resource planning, allocation, distributing or scheduling for enterprises or organisations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q20/00—Payment architectures, schemes or protocols
- G06Q20/38—Payment protocols; Details thereof
- G06Q20/382—Payment protocols; Details thereof insuring higher security of transaction
- G06Q20/3829—Payment protocols; Details thereof insuring higher security of transaction involving key management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q20/00—Payment architectures, schemes or protocols
- G06Q20/38—Payment protocols; Details thereof
- G06Q20/40—Authorisation, e.g. identification of payer or payee, verification of customer or shop credentials; Review and approval of payers, e.g. check credit lines or negative lists
- G06Q20/401—Transaction verification
- G06Q20/4014—Identity check for transactions
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Human Resources & Organizations (AREA)
- Strategic Management (AREA)
- General Physics & Mathematics (AREA)
- Accounting & Taxation (AREA)
- Computer Security & Cryptography (AREA)
- Entrepreneurship & Innovation (AREA)
- General Business, Economics & Management (AREA)
- Bioethics (AREA)
- General Engineering & Computer Science (AREA)
- Quality & Reliability (AREA)
- Databases & Information Systems (AREA)
- Economics (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Operations Research (AREA)
- Data Mining & Analysis (AREA)
- Finance (AREA)
- Marketing (AREA)
- Software Systems (AREA)
- Computer Hardware Design (AREA)
- Tourism & Hospitality (AREA)
- Medical Informatics (AREA)
- Development Economics (AREA)
- Educational Administration (AREA)
- Game Theory and Decision Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data circulation system, a safety management and control system and a safety management and control method thereof, wherein the safety management and control system is in communication connection with each subsystem in the data circulation system and is used for carrying out data safety management and control on each process of data circulation, and the safety management and control system comprises: the multi-class operator and scheduling module are used for meeting the data security management and control requirements in each subsystem; the scheduling module is used for scheduling operators based on the safety control task; the safety management and control system is connected with the computing platform, the computing platform receives the operators scheduled by the scheduling module, configures computing resources, operates the corresponding operators and returns the computing results to the safety management and control system. The invention carries out unified data security management and control aiming at each business link of data circulation, and integrally grasps the data security of the whole data flow. And based on big data processing tool, can handle large batch data to can be fast, the high efficiency safety problem of management and control data transaction link.
Description
Technical Field
The invention relates to the technical field of data circulation, in particular to a data circulation system, a safety management and control system and a safety management and control method thereof.
Background
In data circulation, especially in a data transaction scene, a plurality of circulation links are involved. Since private or sensitive data may be involved in the data, each link needs to be governed in terms of data security.
In the prior art, safety management and control of data are usually only performed in a single link, the safety management and control are mainly performed by writing a java program and connecting the java program to a database in a jdbc mode, and after the data are obtained, whether the data are safe or not is checked.
For a data transaction scene comprising a plurality of circulation links, the technical means cannot integrally control data safety. Moreover, because the amount of data involved in the data transaction scenario is too large, the existing data security management and control technology cannot complete data security management and control in the whole data transaction scenario.
Disclosure of Invention
The invention provides a data circulation system, a safety management and control system thereof and a safety management and control method thereof, aiming at solving the technical problem of how to realize data safety management and control under more circulation links in a data transaction scene.
According to the safety management and control system for the data circulation system, the safety management and control system is in communication connection with a data collection subsystem, a data cleaning subsystem, a data integration subsystem, a data development subsystem, a data transaction subsystem and a data maintenance subsystem in the data circulation system and is used for performing data safety management and control on each process of data circulation, and the safety management and control system comprises:
the system comprises a data aggregation subsystem, a data cleaning subsystem, a data integration subsystem, a data development subsystem, a data transaction subsystem and a data maintenance subsystem, wherein the data aggregation subsystem is used for collecting data;
the scheduling module is used for scheduling operators based on the safety control task;
the safety management and control system is connected with the computing platform, the computing platform receives the operators scheduled by the scheduling module, configures computing resources, operates the corresponding operators and returns the computing results to the safety management and control system.
According to some embodiments of the invention, the safety management system comprises:
and the interaction module is used for receiving the calculation result and sending the calculation result to a corresponding subsystem in the data circulation system.
In some embodiments of the invention, the scheduling module employs a distributed decentralized scheduler with a visual DAG workflow task scheduling platform.
According to some embodiments of the invention, the computing platform employs Apache Spark.
In some embodiments of the invention, the plurality of classes of operators comprises:
the data process auditing operator is used for data acquisition auditing, data transmission auditing, data storage auditing and data processing auditing;
model code auditing, which is used for code vulnerability auditing, code backdoor auditing and high-risk command auditing;
sensitive content auditing, which is used for Huang Bao terrorist sensitive information auditing and negative list auditing;
and element result auditing, which is used for element irreversible auditing, element sensitivity measurement auditing, element similarity auditing, element correlation auditing and element repeatability auditing.
According to some embodiments of the invention, the safety management and control system is designed by a layered architecture, comprising:
front end display layer: adopting vue, and displaying an element ui as an interface;
and an access layer: the filtering is used for service authentication and load balancing of the front-end access back-end;
and (4) a service layer: for implementing management functions;
and (3) a data layer: for data storage.
In some embodiments of the invention, the security management system supports operations in a virtual machine or k8s container environment.
The data circulation system according to the embodiment of the invention comprises: the system comprises a data collecting subsystem, a data cleaning subsystem, a data integrating subsystem, a data developing subsystem, a data trading subsystem, a data maintenance subsystem and a safety management and control system, wherein the safety management and control system is used for data circulation.
According to the safety control method for the data circulation system provided by the embodiment of the invention, the safety control method adopts the safety control system for the data circulation system to perform data safety control on each process of data circulation, and the method comprises the following steps:
s100, registering each operator to a scheduling module;
s200, a scheduling module submits a safety control task to a computing platform;
s300, the computing platform applies for resources to operate an operator corresponding to safety control;
and S400, after the operator is executed, returning the calculation result to the safety control system.
According to some embodiments of the invention, the method further comprises:
and S500, the safety management and control system sends the calculation result to a subsystem corresponding to the data circulation system.
The invention has the following beneficial effects:
the invention carries out unified data security management and control aiming at each business link of data circulation, and can integrally grasp the data security of the whole data flow. Moreover, the invention is based on a big data processing tool, can process large batch of data, thereby quickly and efficiently managing and controlling the safety problem of the data transaction link.
Drawings
Fig. 1 is a schematic diagram of each link of data security management and control according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of an architectural design of a safety management system according to an embodiment of the invention;
FIG. 3 is a diagram illustrating a data security management method according to an embodiment of the present invention;
fig. 4 is a flowchart of a data security management method according to an embodiment of the invention.
Detailed Description
To further illustrate the technical means and effects of the present invention adopted to achieve the predetermined purposes, the present invention is described in detail below with reference to the accompanying drawings and preferred embodiments.
The description of the method flow in the present specification and the steps of the flow chart in the drawings of the present specification are not necessarily strictly performed by the step numbers, and the execution order of the method steps may be changed. Moreover, certain steps may be omitted, multiple steps may be combined into one step execution, and/or a step may be broken down into multiple step executions.
One design principle of software engineering: and (4) decoupling the functions. The invention discloses a method for managing data safety, which separates the requirements of each link of data circulation on data safety management and control and uniformly delivers the requirements to a safety management and control system for management. The data security management and control is a common module for each business link in the data circulation system, and the common module can be extracted, so that the coupling with the original business system is realized, and the purpose of decoupling is achieved.
Moreover, as the data volume is increased, the conventional processing method is more and more unconscious; large data processing tools must be contacted to process large volumes of data. The invention simultaneously solves the problems of even and big data processing.
According to the safety management and control system for the data circulation system, the safety management and control system is in communication connection with the data collection subsystem, the data cleaning subsystem, the data integration subsystem, the data development subsystem, the data transaction subsystem and the data maintenance subsystem in the data circulation system and is used for performing data safety management and control on all processes of data circulation.
It should be noted that, in the whole process of data circulation, the processing process of each subsystem of the data circulation system is involved, and the original data is converted into data elements and data products through the data aggregation subsystem, the data cleaning subsystem, the data integration subsystem, the data development subsystem, the data transaction subsystem and the data maintenance subsystem, and then the data elements and the data products are traded.
In the whole data circulation process, the safety management and control system performs data interaction with the data collection subsystem, the data cleaning subsystem, the data integration subsystem, the data development subsystem, the data transaction subsystem and the data maintenance subsystem so as to effectively manage and control the data safety of the data processing process in each subsystem.
It should be noted that, in the existing data security management and control, the above-mentioned process of the transaction from the original data to the data element to the data product and the transaction process between the data element and the data product are not involved. Moreover, because the system architecture involved in the existing data security management and control work is often simpler, and does not involve the complex architecture formed by the subsystems, the data security management and control work is realized by arranging a data security auditing function module in the corresponding subsystem.
The invention decouples the data safety control work of each subsystem of the data circulation system, designs the safety control system independent of each subsystem in the data processing process, and in the data safety control of the data circulation, the invention not only relates to the data process auditing work, but also relates to the data model establishing and training, the data element circulation in the data circulation process, therefore, the safety control system also relates to the data model code auditing, the element structure auditing and the like, and ensures the safety and reliability of the whole process of the data circulation.
The safety management and control system comprises: a multi-class operator and a scheduling module.
The multi-class operators are used for meeting the data safety control requirements in the data collection subsystem, the data cleaning subsystem, the data integration subsystem, the data development subsystem, the data transaction subsystem and the data maintenance subsystem.
In some embodiments of the present invention, as shown in FIG. 1, the extracted operators may include four classes, including: and auditing operators in the data process, model codes, sensitive contents and element results.
The data process audit operator is used for data acquisition audit, data transmission audit, data storage audit and data processing audit;
the model code audit is used for code vulnerability audit, code backdoor audit and high-risk command audit;
sensitive content auditing is used for auditing yellow riot sensitive information and negative list auditing;
and element result auditing is used for element irreversible auditing, element sensitivity measurement auditing, element similarity auditing, element correlation auditing and element repeatability auditing.
The scheduling module is used for scheduling operators based on the safety control task;
in some embodiments of the invention, the scheduling module employs a distributed decentralized scheduler with a visual DAG workflow task scheduling platform. For example, the scheduler may employ the open source based scheduling component Apache dolphin scheduler.
The safety management and control system is connected with the computing platform, the computing platform receives the operators scheduled by the scheduling module, configures computing resources, operates the corresponding operators and returns the computing results to the safety management and control system. According to some embodiments of the invention, a computing platform may employ Apache Spark.
According to some embodiments of the invention, a safety management system comprises: and the interaction module is used for receiving the calculation result and sending the calculation result to a corresponding subsystem in the data circulation system.
According to some embodiments of the present invention, as shown in fig. 3, the safety management and control system adopts a layered architecture design, including:
front end display layer: adopting vue, and displaying an element ui as an interface;
and an access layer: the filtering is used for service authentication and load balancing of the front-end access back-end;
and (4) a service layer: for implementing management functions;
and (3) a data layer: for data storage.
In some embodiments of the invention, the security management system supports operations in a virtual machine or k8s container environment.
The data circulation system according to the embodiment of the invention comprises: the system comprises a data collecting subsystem, a data cleaning subsystem, a data integration subsystem, a data development subsystem, a data transaction subsystem, a data maintenance subsystem and a safety management and control system, wherein the safety management and control system is used for data circulation.
According to the safety control method for the data circulation system in the embodiment of the present invention, the safety control method adopts the above safety control system for the data circulation system to perform data safety control on each process of data circulation, as shown in fig. 3 and 4, the method includes:
s100, registering each operator to a scheduling module;
s200, a scheduling module submits a safety control task to a computing platform;
s300, the computing platform applies for resources to operate an operator corresponding to safety control;
and S400, after the operator is executed, returning the calculation result to the safety control system.
According to some embodiments of the invention, the method further comprises:
and S500, the safety management and control system sends the calculation result to a subsystem corresponding to the data circulation system.
The invention has the following beneficial effects:
the invention carries out unified data security management and control aiming at each business link of data circulation, and can integrally grasp the data security of the whole data flow. Moreover, the invention is based on a big data processing tool, and can process a large amount of data, thereby quickly and efficiently managing and controlling the safety problem of the data transaction link.
Hereinafter, a data circulation system, a security management and control system thereof, and a security management and control method thereof according to the present invention will be described in detail with reference to the accompanying drawings. It is to be understood that the following description is only exemplary in nature and should not be taken as a specific limitation on the invention.
The safety management and control system is a technical system for managing, controlling and auditing key control points of related systems such as a data element service system, a data element operating system and the like, and mainly ensures basic safety and data safety, the auditing object is a safety control point and a basic safety tool, the auditing method is a safety strategy, and the auditing aims at providing active, intelligent and dynamic safety protection and safety auditing and ensuring the continuity, safety, reliability and high efficiency of a data element engineering system.
The technology used in the invention comprises the following steps:
1) The open source based scheduling component dolphin scheduler. The Apache Dolphin scheduler is a distributed decentralized and easily extensible visual DAG workflow task scheduling platform. The method aims to solve the complicated dependency relationship in the data processing flow, so that the scheduling system can be used in a box opening mode in the data processing flow.
2) The distributed computing component Spark. Apache Spark is a fast, general-purpose computing engine designed specifically for large-scale data processing.
3) And developing a framework springclosed. Spring Cloud is an ordered collection of a series of frameworks. The development convenience of the Spring Boot is utilized to skillfully simplify the development of infrastructure of a distributed system, such as service discovery registration, configuration center, message bus, load balancing, circuit breaker, data monitoring and the like, and the Spring Boot can be used for one-key starting and deployment in a development style.
From the whole process, each link of data security management and control is introduced:
as shown in fig. 1, in each link of data collection, data cleaning, data integration, element development, element transaction and element maintenance, 4 types of operators are extracted, and the safety control system achieves the purpose of safety control of data elements by scheduling the 4 types of operators. The 4 types of operators are specifically as follows:
and (5) auditing content operator classification:
1. and (3) data process auditing:
and (3) links: data collection, data cleaning and data integration;
object: data in the course of circulation;
the functions are as follows: the safety of the landing data in the links of data collection, data cleaning and data integration is ensured.
Example (c): storing personal information such as an identity card, a password and the like by using SM4 encryption and having a corresponding audit operator, decrypting by using an SM4 algorithm, and if the decryption succeeds, indicating that the data conforms to the encryption rule, successfully auditing; otherwise, the audit is not passed.
2. And (3) auditing model codes:
and (3) links: element development;
object: a model code;
the functions are as follows: the safety of the model codes developed by the element developers is ensured, and security loophole codes, high-risk commands and the like are prevented from being in the codes. Thereby preventing security risks such as data leakage.
Example (c): and auditing the source code of the jar package by the corresponding operator service.
3. And sensitive content auditing:
and (3) links: element development;
object: meta information of the data element;
the functions are as follows: the element developed by the element developer has some descriptive information, and whether the information content contains some sensitive content is checked through an operator;
example (c): in the description information of the elements, if there is sensitive information related to yellow, terrorism, politics and the like, the sensitive information is not operated, and auditing is carried out through a sensitive content auditing operator.
4. And (4) element result auditing:
environment: element transaction;
object: a data element result;
the functions are as follows: element result auditing is the last link of element production, and finally produced elements are audited;
example (c): for example, there are A, B two developers who can detect through a similarity operator by developing a functionally similar component; if similar elements exist, the related service party is informed to process.
As shown in fig. 2, the safety management and control system adopts a layered architecture design:
front end display layer: the layer is mainly displayed by using front-end technologies such as vue and elementui.
And an access layer: the service of the front end accessing the back end is firstly subjected to the authentication of an access layer, the filtering of the access layer such as load balancing, and the like, and the used components are robbin, feign and the like.
And (4) a service layer: the layer realizes most management functions of the safety management and control system, and development frames mainly comprise springboot frames, springclosed frames and the like.
And (3) a data layer: mysql is mainly used for storing and safely managing some data of the system.
And (3) operating environment: the safety management and control system supports operation on environments such as virtual machines, k8s containers and the like.
The operator is executed:
each operator is an independently operable unit, and the operator is used to execute the logic of data security control of each business link, and the specific execution process is shown in fig. 3 and 4:
1) Firstly, a safety management and control system registers each operator to a dispatcher;
2) Submitting the tasks to a big data platform by a dispatcher;
3) The big data platform applies for resources to operate an operator;
4) After the operator is executed, the result is transmitted back to the safety control system;
5) The safety management and control system further synchronizes the result to the service platform.
In summary, the data security function modules of each business link are extracted and managed by a system in a unified way, so that the purpose of decoupling is achieved. The big data calculation engine is adopted, and the problem that the large data size is difficult to process in the prior art is solved.
While the present invention has been described in connection with the preferred embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.
Claims (10)
1. The utility model provides a safety management and control system for data circulation system which characterized in that, safety management and control system and the data collection subsystem, data cleaning subsystem, data integration subsystem, data development subsystem, data transaction subsystem and the data maintenance subsystem in the data circulation system all communication connection for carry out data security management and control to each process of data circulation, safety management and control system includes:
the system comprises a data aggregation subsystem, a data cleaning subsystem, a data integration subsystem, a data development subsystem, a data transaction subsystem and a data maintenance subsystem, wherein the data aggregation subsystem is used for collecting data;
the scheduling module is used for scheduling operators based on the safety control task;
the safety management and control system is connected with the computing platform, the computing platform receives the operators scheduled by the scheduling module, configures computing resources, operates the corresponding operators and returns the computing results to the safety management and control system.
2. A safety management and control system for a data flow through system according to claim 1, wherein the safety management and control system comprises:
and the interaction module is used for receiving the calculation result and sending the calculation result to a corresponding subsystem in the data circulation system.
3. A security management system for a data flow system as claimed in claim 1, wherein the scheduling module employs a distributed decentralized scheduler with a visual DAG workflow task scheduling platform.
4. The security management system for a data flow system of claim 1, wherein the computing platform employs Apache Spark.
5. A security management system for a data flow through system according to claim 1, wherein the plurality of classes of operators comprises:
the data process auditing operator is used for data acquisition auditing, data transmission auditing, data storage auditing and data processing auditing;
model code auditing, which is used for code vulnerability auditing, code backdoor auditing and high-risk command auditing;
sensitive content auditing, which is used for Huang Bao terrorist sensitive information auditing and negative list auditing;
and element result auditing, which is used for element irreversible auditing, element sensitivity measurement auditing, element similarity auditing, element correlation auditing and element repetition auditing.
6. The safety management and control system for a data circulation system according to claim 1, wherein the safety management and control system is designed by a layered architecture, and comprises:
front end display layer: adopting vue, and displaying an element ui as an interface;
and an access layer: the filtering is used for the service authentication and the load balance of the front-end access back-end;
and (4) a service layer: for implementing management functions;
and (3) a data layer: for data storage.
7. A security management system for a data flow through system according to claim 6, wherein the security management system supports operations in a virtual machine or k8s container environment.
8. A data circulation system, comprising: the system comprises a data collecting subsystem, a data cleaning subsystem, a data integration subsystem, a data development subsystem, a data transaction subsystem, a data maintenance subsystem and a safety management and control system, wherein the safety management and control system is used for data circulation according to any one of claims 1 to 7.
9. A safety control method for a data circulation system, wherein the safety control method adopts a safety control system for the data circulation system according to any one of claims 1 to 7 to perform data safety control on each process of data circulation, and the method comprises the following steps:
s100, registering each operator to a scheduling module;
s200, a scheduling module submits a safety control task to a computing platform;
s300, the computing platform applies for resources to operate an operator corresponding to safety control;
and S400, after the operator is executed, returning the calculation result to the safety control system.
10. A method of security management for a data flow system according to claim 9, the method further comprising:
and S500, the safety management and control system sends the calculation result to a subsystem corresponding to the data circulation system.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211146636.8A CN115249149A (en) | 2022-09-21 | 2022-09-21 | Data circulation system, safety management and control system and safety management and control method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211146636.8A CN115249149A (en) | 2022-09-21 | 2022-09-21 | Data circulation system, safety management and control system and safety management and control method thereof |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115249149A true CN115249149A (en) | 2022-10-28 |
Family
ID=83699603
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211146636.8A Pending CN115249149A (en) | 2022-09-21 | 2022-09-21 | Data circulation system, safety management and control system and safety management and control method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115249149A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160330219A1 (en) * | 2015-05-04 | 2016-11-10 | Syed Kamran Hasan | Method and device for managing security in a computer network |
| CN106598000A (en) * | 2016-07-01 | 2017-04-26 | 北京华科合创科技发展有限公司 | Production scheduling and safety management and control system |
| CN111756801A (en) * | 2020-05-22 | 2020-10-09 | 江南大学 | Method and system for processing intelligent manufacturing big data |
| CN113553381A (en) * | 2021-07-28 | 2021-10-26 | 中建材信息技术股份有限公司 | Distributed data management system based on novel pipeline scheduling algorithm |
| CN114328688A (en) * | 2021-12-27 | 2022-04-12 | 国网河北省电力有限公司信息通信分公司 | Management and control platform for electric power energy big data |
-
2022
- 2022-09-21 CN CN202211146636.8A patent/CN115249149A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160330219A1 (en) * | 2015-05-04 | 2016-11-10 | Syed Kamran Hasan | Method and device for managing security in a computer network |
| CN106598000A (en) * | 2016-07-01 | 2017-04-26 | 北京华科合创科技发展有限公司 | Production scheduling and safety management and control system |
| CN111756801A (en) * | 2020-05-22 | 2020-10-09 | 江南大学 | Method and system for processing intelligent manufacturing big data |
| CN113553381A (en) * | 2021-07-28 | 2021-10-26 | 中建材信息技术股份有限公司 | Distributed data management system based on novel pipeline scheduling algorithm |
| CN114328688A (en) * | 2021-12-27 | 2022-04-12 | 国网河北省电力有限公司信息通信分公司 | Management and control platform for electric power energy big data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105283852B (en) | A kind of method and system of fuzzy tracking data | |
| CN110458559B (en) | Transaction data processing method, device, server and storage medium | |
| CN107368365A (en) | Cloud platform automatic O＆M method, system, equipment and storage medium | |
| CN108064379A (en) | The query engine fetched for remote endpoint information | |
| CN105574082A (en) | Storm based stream processing method and system | |
| EP2977899A2 (en) | Integrating execution of computing analytics within a mapreduce processing environment | |
| US20220179711A1 (en) | Method For Platform-Based Scheduling Of Job Flow | |
| CN113961245A (en) | Security protection system, method and medium based on micro-service application | |
| CN111381940B (en) | Distributed data processing method and device | |
| Ahmad et al. | IoT-fog-based healthcare 4.0 system using blockchain technology | |
| CN110062926A (en) | Device driver telemetering | |
| CN112313627A (en) | Mapping mechanism of events to serverless function workflow instances | |
| US10009220B2 (en) | In-vehicle information system and information processing method thereof | |
| CN115865398A (en) | Method and system for constructing digital service security native system | |
| CN114817022A (en) | Railway electronic payment platform test method, system, equipment and storage medium | |
| Hsaini et al. | FSM modeling of testing security policies for mapreduce frameworks | |
| CN113836237A (en) | Method and device for auditing data operation of database | |
| CN115249149A (en) | Data circulation system, safety management and control system and safety management and control method thereof | |
| CN110611694B (en) | Data processing center based on virtualized master-slave container | |
| US20090031308A1 (en) | Method And Apparatus For Executing Multiple Simulations on a Supercomputer | |
| CN111161052A (en) | Bank operation data processing method and device | |
| CN111447273A (en) | Cloud processing system and data processing method based on cloud processing system | |
| US20210286785A1 (en) | Graph-based application performance optimization platform for cloud computing environment | |
| CN114579405A (en) | Data processing method, data processing apparatus, electronic device, and storage medium | |
| CN114329450A (en) | Data security processing method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20221028 |
|
| RJ01 | Rejection of invention patent application after publication |