CN116208666B - Processing method and device supporting multi-source data center joint security calculation data - Google Patents
Processing method and device supporting multi-source data center joint security calculation data Download PDFInfo
- Publication number
- CN116208666B CN116208666B CN202310500330.6A CN202310500330A CN116208666B CN 116208666 B CN116208666 B CN 116208666B CN 202310500330 A CN202310500330 A CN 202310500330A CN 116208666 B CN116208666 B CN 116208666B
- Authority
- CN
- China
- Prior art keywords
- joint
- node
- data center
- proxy node
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/602—Providing cryptographic facilities or services
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/04—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks
- H04L63/0428—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks wherein the data content is protected, e.g. by encrypting or encapsulating the payload
- H04L63/0442—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks wherein the data content is protected, e.g. by encrypting or encapsulating the payload wherein the sending and receiving network entities apply asymmetric encryption, i.e. different keys for encryption and decryption
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/08—Network architectures or network communication protocols for network security for authentication of entities
- H04L63/0823—Network architectures or network communication protocols for network security for authentication of entities using certificates
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/5017—Task decomposition
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention discloses a method and a device for supporting multi-source data center combined safe calculation data processing, comprising the following steps: acquiring joint calculation task information crossing a multi-source data center; the joint calculation task information is based on SQL specification; analyzing the joint calculation task information and sending corresponding calculation tasks to each proxy node; acquiring data of a corresponding data center through each proxy node, performing joint calculation according to the acquired data, and outputting a calculation result of a joint calculation task; the method solves the problem that connection interaction with other multi-source data can not be realized, and improves the value of the multi-source data.
Description
Technical Field
The invention relates to the technical field of servers, in particular to a method and a device for supporting multi-source data center combined safe calculation data processing.
Background
With the development of big data technology, various storage services such as MySQL (relational database), clickHouse (analytic database), HBase (distributed, array-oriented open source database), hive (data warehouse tool), etc. appear, so that data generated by each data center business system are scattered on different storage services, and each data is just like a plurality of islands cannot interact with other data in a connection manner, thus greatly reducing the data value.
The traditional solution is that distributed query engines such as Spark SQL (structured data processing engine) or prest (open source database query engine) can solve the joint computation of multiple data sources, but joint computation of data crossing multiple sources data centers is not provided, connection interaction with other data crossing multiple sources cannot be realized, and data value cannot be improved. Another solution is to extract and aggregate all the data through the data center, which causes waste of storage resources, and in order to ensure that data synchronization can be performed, it is necessary to ensure that the network between the data center service and each data source can be communicated, and great cost is required to construct a basic network environment.
Accordingly, there is a need in the art for improvement.
Disclosure of Invention
The invention aims to solve the technical problem that the method and the device for processing the data support multi-source data center joint security calculation are used for solving the problem that connection interaction with other multi-source data can not be realized.
The technical scheme adopted for solving the technical problems is as follows:
in a first aspect, the present invention provides a method for supporting multi-source data center joint security computing data processing, including:
acquiring joint calculation task information crossing a multi-source data center; the joint calculation task information is based on SQL specification;
analyzing the joint calculation task information and sending corresponding calculation tasks to each proxy node;
and acquiring data of a corresponding data center through each proxy node, performing joint calculation according to the acquired data, and outputting a calculation result of the joint calculation task.
In one implementation, the acquiring the joint computing task information across the multi-source data center includes, before:
the IP and port information of the service are stored in a service database through a node registration module, and a unique number is generated for each proxy node.
In one implementation, the acquiring the joint computing task information across the multi-source data center further includes:
generating a certificate required by encrypting and decrypting data between the proxy node and the central node through the openssl, and returning the certificate to the corresponding proxy node;
generating an RSA key pair, and encrypting the unique codes of all proxy nodes through the public key of the RSA key pair;
acquiring all machine IPs contained in the central node cluster, and returning the machine IPs to each proxy node in the form of an IP list; the IP list is used for generating a white list at each proxy node.
In one implementation, the acquiring the joint computing task information across the multi-source data center further includes:
and periodically updating certificates corresponding to each proxy node and encrypted node codes according to the request of each proxy node.
In one implementation, the acquiring joint computing task information across a multi-source data center includes:
acquiring the joint calculation task information;
and analyzing the table resources in the joint calculation task information through an SQL analysis module, and authenticating the authority of the user node according to the analyzed table resources.
In one implementation, the parsing the joint computing task information, sending a corresponding computing task to each proxy node, and generating a key and authentication information for encrypted transmission for each proxy node, includes:
splitting the joint calculation task into a plurality of subtasks through an SQL analysis module;
and distributing a plurality of subtasks to corresponding proxy nodes.
In one implementation, the splitting the joint computing task into a plurality of subtasks by the SQL parsing module includes:
acquiring a read lock of a corresponding proxy node from the Zookeeper cluster;
and submitting the corresponding subtasks according to the reading lock acquisition result, and releasing the reading lock after the corresponding subtasks are operated.
In one implementation manner, the acquiring, by each proxy node, the data of the corresponding data center, and performing joint calculation according to the acquired data, includes:
acquiring node authentication information fed back by each proxy node; the node authentication information is the result of encrypting the unique codes of the proxy nodes by the public key in the RSA key pair;
performing agent node authentication according to the node authentication information fed back by each agent node;
and acquiring data of the corresponding data center according to the authentication result, and performing joint calculation according to the acquired data.
In one implementation manner, the proxy node authentication according to the node authentication information fed back by each proxy node includes:
decrypting by a private key in the RSA key pair;
if the decryption fails, rejecting the task request of the corresponding user node;
if the decryption is successful, judging whether the decrypted code is the code of the corresponding proxy node;
if the decrypted code is the code of the corresponding proxy node, the proxy node authentication is judged to be successful.
In one implementation manner, the acquiring, by each proxy node, the data of the corresponding data center, performing joint calculation according to the acquired data, and outputting a calculation result of the joint calculation task, and then includes:
acquiring a write lock of a proxy node to be updated from a Zookeeper cluster;
and updating the key of the corresponding proxy node according to the acquired write lock, and releasing the write lock.
In a second aspect, the present invention provides a data processing apparatus supporting multi-source data center federated secure computing, comprising:
the task management module is used for acquiring joint calculation task information of a cross-multi-source data center; the joint calculation task information is based on SQL specification;
the node management module is used for analyzing the joint calculation task information and sending corresponding calculation tasks to each proxy node;
and the joint calculation module is used for acquiring the data of the corresponding data center through each proxy node, carrying out joint calculation according to the acquired data and outputting the calculation result of the joint calculation task.
In a third aspect, the present invention provides a terminal comprising: the system comprises a processor and a memory, wherein the memory stores a multi-source data center combined secure computing data processing program, and the multi-source data center combined secure computing data processing program is used for realizing the operation of the multi-source data center combined secure computing data processing method.
In a fourth aspect, the present invention also provides a computer readable storage medium storing a supporting multi-source data center joint security computing data processing program for implementing the operations of the supporting multi-source data center joint security computing data processing method according to the first aspect when executed by a processor.
The technical scheme adopted by the invention has the following effects:
the invention provides a safe joint calculation method supporting cross-multi-source data centers, which can realize joint calculation of the cross-data centers and the cross-data sources, obtains data of each data center by proxy nodes, gathers the data to the center nodes for calculation, and ensures the safety in the data calculation process by key encryption transmission calculation results and application-approval mechanisms, wherein each proxy node only receives requests initiated by the center nodes by checking user credentials, thereby solving the problem that connection interaction with other cross-multi-source data cannot be realized, and improving the value of the cross-multi-source data.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to the structures shown in these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a method of processing data that supports multi-source data center federated security computing in one implementation of the invention.
FIG. 2 is a schematic diagram of the overall architecture in one implementation of the invention.
FIG. 3 is a schematic diagram of task management in one implementation of the invention.
FIG. 4 is a schematic diagram of user and proxy node authority authentication in one implementation of the invention.
Fig. 5 is a functional schematic of a terminal in one implementation of the invention.
The achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more clear and clear, the present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
Exemplary method
Traditional solutions have been through distributed query engines such as Spark SQL (structured data processing engine) or prest (open source database query engine) which can address the joint computation of multiple data sources, but do not provide joint computation across multiple source data center data. Another solution is to extract and aggregate all the data through the data center, which causes waste of storage resources, and in order to ensure that data synchronization can be performed, it is necessary to ensure that the network between the data center service and each data source can be communicated, and great cost is required to construct a basic network environment.
Aiming at the problems, the embodiment of the invention provides a safe support cross-multisource data center combined computing method, which can realize cross-data center and cross-data source combined computing, and the proxy node acquires data of each data center and gathers the data to the center node for computing, and the safety in the data computing process is ensured by key encryption transmission computing results and an application-approval mechanism, and each proxy node only receives requests initiated by the center node by checking user credentials, so that the problems that data center data are independent and cannot be effectively associated with each other and the safety of a scene is combined in the cross-multisource data center are solved.
As shown in fig. 1, an embodiment of the present invention provides a method for supporting multi-source data center joint security computing data processing, including the following steps:
step S100, acquiring joint calculation task information of a cross-multi-source data center.
In this embodiment, the method for supporting multi-source data center joint security computing data processing is applied to a terminal, where the terminal includes but is not limited to: a computer, etc.
As shown in fig. 2, the overall architecture of the present embodiment includes a proxy node and a central node; the proxy node is responsible for the operation of internal data of the multi-source data center, and the proxy node realizes the operation based on a Presto program, specifically: the Presto program is internally provided with a plurality of connectors of data sources, each connector is responsible for interacting with different databases, metadata is acquired by using JDBC (database connection program), table data is acquired by executing SQL (database query statement) sentences, and therefore the operation process of the internal data of the multi-source data center is realized.
Because each proxy node is in butt joint with one data center data, the center node is responsible for interaction with the proxy node, so that joint calculation of the data center and the data center data can be realized; the joint calculation task is a task realized through a pre-written SQL statement, and because the central node is provided with an SQL analysis and SQL execution engine, the SQL analysis engine is called through the pre-written SQL statement, which data needs to be acquired from which nodes is analyzed BY the SQL analysis engine, and after the data is acquired, SQL operations such as JOIN (relational query), UNION (association), GROUP BY (grouping) and the like are carried out on the data BY the SQL execution engine.
In this embodiment, each proxy node needs to register at the central node; the proxy node needs to apply a key when registering with the central node, which key can be used for encryption during data transmission. The user writes the joint calculation task through the unified SQL standard, submits the joint calculation task to the central node, the central node analyzes the SQL, the central node judges whether the user applies for the resource or not by checking the table and the field information related to the task, and if the user does not apply for the resource, the user does not have permission, the request is refused.
In particular, the present embodiment relates to the following components and modules:
agent node: based on Presto realization, the method is used for acquiring data of different databases in a data center;
center node: acquiring data of different data centers through proxy nodes and performing joint calculation;
and the node management module: managing information of the proxy node, and generating a key and authentication information for encrypted transmission for the proxy node;
metadata management module: the metadata information of each data center is synchronized, the metadata information of each data center comprises library, table and field information, and the synchronization process of the metadata information of each data center is as follows: after adding a data source, a data synchronization operation is automatically triggered, and then the data synchronization operation is triggered through a timing task. Connectors of various data sources are arranged in each proxy node, and the connectors can acquire metadata of the different data sources in a JDBC mode and synchronize the metadata to the central node; in the metadata management module, a user needs to apply for required table resources, and the table resources can be used only after approval by a node administrator;
the task management module: the method is responsible for managing the cross-multisource data center joint calculation task, a user writes the joint calculation task based on the SQL specification of prest, and the central node analyzes the SQL to split into a plurality of subtasks and distributes the subtasks to the proxy node for execution; the center node analysis process specifically comprises the following steps: and (3) obtaining which fields of a table of which node needs to be acquired after analysis is completed through an SQL parser built in Presto, performing which function operations, converting into corresponding query sentences, and delivering the query sentences to service execution of the proxy node.
User and node authority authentication module: and acquiring table resources contained in the user joint calculation task, and rejecting the request if the user has no access. Authentication service is provided for the proxy node, authentication information is carried when the central node issues a calculation task to the proxy node, and if the authentication information is illegal, the proxy node refuses the request.
The node management module, the metadata management module, the task management module and the user and node authority authentication module are modules arranged in a central node.
The following is the use flow of the system: node management (i.e. registering proxy node), metadata management (i.e. metadata synchronization, resource application, approval), task management (i.e. initiating task), user, node authority authentication (i.e. verifying user authority), return data, detailed description of the specific implementation of each module.
Specifically, in one implementation of the present embodiment, the step S100 includes the following steps before:
and S001, storing the IP and port information of the service in a service database through a node registration module, and generating a unique number for each proxy node.
In this embodiment, in the node management module, node registration is first required.
When registering the nodes, the proxy node is realized based on prest, the node registration module stores the IP and port information of prest service in a service database, and the system generates a unique number for each proxy node during registration to distinguish different proxy nodes.
Specifically, in one implementation of the present embodiment, the following steps are further included before step S100:
step S002, generating a certificate required by encrypting and decrypting data between the proxy node and the central node through the openssl, and returning the certificate to the corresponding proxy node;
step S003, an RSA key pair is generated, and unique codes of all proxy nodes are encrypted through a public key in the RSA key pair;
step S004, all the machine IPs contained in the central node cluster are acquired and returned to each proxy node in the form of an IP list.
In this embodiment, in the node management module, the key required for data transmission is secondarily generated.
In the process of generating the key, first, certificates required for encrypting and decrypting data are returned to the proxy node through the proxy node of opensl (software library package of open source code) and the central node.
Next, an RSA (algorithm for encryption and digital signature) key pair (RSA key pair includes a public key and a private key) is generated, the generated proxy node is uniquely encoded and encrypted by the public key, and the encrypted unique code is stored by the central node, and can be used for proxy node authentication in a subsequent data transmission process.
And finally, acquiring all the machine IPs contained in the central node cluster, returning an IP list to the proxy node, and adding the IP list into a white list by the proxy node, wherein the white list can be used for client node authentication in the subsequent data transmission process.
Specifically, in one implementation of the present embodiment, the following steps are further included before step S100:
step S005, periodically updating certificates corresponding to each proxy node and encrypted node codes according to the request of each proxy node.
In this embodiment, in the node management module, the key needs to be updated periodically at last.
In the process of updating the secret key regularly, each proxy node has a timing task because of leakage risk of the certificate and the encrypted node code, the central node is requested regularly, the certificate and the encrypted node code are updated, and the updating process of the certificate and the encrypted node code is as follows: the certificate and the encrypted node code are stored on a server where the node is located through the proxy node, and the updating can be completed through replacing the corresponding file content; the running task is affected during updating, and a solution is provided in the key updating and task mutex initiating part in the embodiment.
In this embodiment, metadata management is performed after node management, which includes: metadata synchronization, user application of resources, and node administrator approval.
Metadata synchronization: after the node registration is completed, metadata information of the access data is acquired through the proxy node, and the metadata information is stored in a service library, wherein the metadata information comprises: database, table, field information, metadata synchronization process is as described above. Metadata has a unique identification, such as: under the proxy node noda there is a MySQL connection named edu, which contains a school library and a student table containing two fields, id and name, these two fields are represented by nodea.edu.school.student.id and nodea.edu.school.student.name.
The user applies for resources: the user applies for the required table resources according to the service requirement, judges which node the table belongs to according to the unique identifier of each table, and sends an application record to the node administrator; since each proxy node has a unique code, for example: datacenter01, then the student table under the schema library of connection name mysql for that proxy node is uniquely identified as datacenter01.Mysql. Schema.
Node administrator approval: after receiving the request of the user resource application, the node administrator performs approval, and a resource authorization record is added after the approval is passed.
In this embodiment, after metadata management, tasks initiated by the user are processed through task management.
Specifically, in one implementation of the present embodiment, step S100 includes the steps of:
step S101, acquiring the joint calculation task information;
and step S102, analyzing the table resources in the joint calculation task information through an SQL analysis module, and authenticating the authority of the user node according to the analyzed table resources.
In the task management module, a task is initiated first: the user writes the joint calculation task based on the Presto SQL specification, the central node splits the joint calculation task into a plurality of subtasks by calling the Presto SQL analysis module service, the subtasks are distributed to each proxy node, and the proxy node acquires the data in the data center and returns the data after encryption by the key generated by the central node management module.
In order to solve the problem that the running task of the node is affected during the updating of the key, a mutual exclusion mode of the updating of the key and the initiation of the task is designed in the embodiment, so that the node management module cannot update the key during the execution of the joint calculation task.
As shown in fig. 1, in an implementation manner of the embodiment of the present invention, the method for supporting multi-source data center joint security computing data processing further includes the following steps:
and step 200, analyzing the joint calculation task information and sending corresponding calculation tasks to each proxy node.
In this embodiment, in the process of analyzing the joint computing task by the task management module, the user and the proxy node are authenticated by the user and the node authority authentication module respectively.
Specifically, in one implementation of the present embodiment, step S200 includes the steps of:
step S201, splitting the joint calculation task into a plurality of subtasks through an SQL analysis module;
step S202, distributing a plurality of subtasks to corresponding proxy nodes.
In this embodiment, user authority authentication: after the user submits the joint analysis task, the task management module judges whether the user applies for the resource or not through analyzing table resources contained in the task by SQL, and if not, the request is refused.
As shown in fig. 3, after performing SQL analysis and user authentication, only the task processing is performed by using a method of updating a key and initiating mutual exclusion of the task, and the key update cannot be performed.
Specifically, in one implementation of the present embodiment, step S201 includes the following steps:
step S201a, acquiring a read lock of a corresponding proxy node from a Zookeeper cluster;
step S201b, submitting the corresponding subtasks according to the read lock acquisition result, and releasing the read lock after the corresponding subtasks are operated.
In this embodiment, the portion of the key update and the initiating task mutex includes: when a common user initiates a joint calculation task, a task management module analyzes proxy nodes related to the task through an SQL analysis function, then tries to acquire read locks of the nodes from a Zookeeper (distributed application service software), the task can be submitted after the read locks are successfully acquired, and the read locks can be released after the operation is completed, so that the task processing can be performed, and the secret key updating can not be performed.
Specifically, the process of acquiring the read locks of the nodes from the Zookeeper is as follows: creating a temporary sequence node of ip-type-id on the ZooKeeper, wherein the type of the read lock is R, the type of the write lock is W, id starts from 0000000001, judging whether all predecessor nodes of the node do not exist W type nodes when acquiring the read lock, and acquiring the read lock successfully if the W type nodes do not exist. If the acquisition fails, monitoring the state of the precursor node, and entering a waiting state until the write lock of the precursor node is released.
As shown in fig. 1, in an implementation manner of the embodiment of the present invention, the method for supporting multi-source data center joint security computing data processing further includes the following steps:
step S300, acquiring data of a corresponding data center through each proxy node, performing joint calculation according to the acquired data, and outputting a calculation result of the joint calculation task.
In this embodiment, the central node has user authority authentication, and the proxy node also needs to perform authentication, and the specific authentication mode of the proxy node is as follows.
Specifically, in one implementation of the present embodiment, step S300 includes the steps of:
step S301, node authentication information fed back by each proxy node is obtained; the node authentication information is the result of encrypting the unique code of each proxy node by the public key in the RSA key pair;
step S302, performing agent node authentication according to the node authentication information fed back by each agent node;
in one implementation of this embodiment, step S302 includes the steps of:
step S302a, decrypting the private key in the RSA key pair;
step S302b, if decryption fails, rejecting the task request of the corresponding user node;
step S302c, if the decryption is successful, judging whether the decrypted code is the code of the corresponding proxy node;
in step S302d, if the decrypted code is the code of the corresponding proxy node, it is determined that the proxy node authentication is successful.
In this embodiment, secondary development is performed on prest, and a permission authentication plug-in is added, as shown in fig. 4, after receiving a request, a proxy node obtains an IP address of the requesting client, further determines whether the IP exists in an IP whitelist in a node management module, and if not, considers that the client request is not from a central node, and refuses the request.
If the IP is legal, the proxy node acquires the authentication information in the request parameters, sends the authentication information to the central node for authentication, the central node firstly decrypts the authentication information through a private key generated in the node management module, if the authentication information is illegal, the request is refused, after the decryption is successful, whether the decrypted code is the code of the node is judged, and the authentication can be successful only if the authentication information and the authentication information are equal.
Step S303, obtaining the data of the corresponding data center according to the authentication result, and carrying out joint calculation according to the obtained data.
In this embodiment, after the authentication of the user and the node authority authentication module is successful, that is, after the authentication of both the user authentication and the proxy node authentication is successful, the central node acquires the data of the corresponding data center, decrypts the data, performs joint calculation, and returns and outputs a joint calculation result; the output joint calculation result is sent to the user side in the form of a two-dimensional data table, wherein the two-dimensional data table comprises field names and specific data.
In one implementation manner of the embodiment of the present invention, the method for supporting multi-source data center joint security computing data processing further includes the following steps:
step S400, obtaining a write lock of a proxy node to be updated from a Zookeeper cluster;
step S500, updating the key of the corresponding proxy node according to the acquired write lock, and releasing the write lock.
In this embodiment, after executing the joint calculation task, the central node may update the key of each proxy node according to the update request sent by each proxy node, where the process follows the key update and initiate task mutual exclusion manner.
Another part of the mutual exclusion of the key updating and the initiating task comprises: when the proxy node updates the key, the write lock of the node needs to be acquired from the Zookeeper cluster, and if the joint calculation task involves the resource of the node at the moment, the acquisition of the write lock is blocked until the joint calculation task is executed. Similarly, when the read lock of the node is to be acquired by joint calculation after the write lock is acquired, the key updating is required to be completed, and the success can be acquired after the write lock is released.
Specifically, the process of acquiring the write locks of the nodes from the Zookeeper is as follows: creating a temporary sequence node on the ZooKeeper, wherein when the id value of the current node is minimum, the acquisition of the write lock is successful, if the acquisition fails, the temporary sequence node enters a waiting state, and when all the precursor nodes are released, the acquisition is successful.
In the embodiment, a data joint calculation crossing a multi-source data center is designed, a data authority authentication mechanism is added, and the security of data encryption is ensured through key encryption.
In the embodiment, the scheme realizes the joint query between the multi-source data center and the multi-data sources, the proxy node acquires the data in the data center, the IP of each data element is not exposed, and the proxy node has an authentication mechanism to ensure the data security; meanwhile, the agent node acquires a key during registration, data can be encrypted through the key during subsequent interaction with the central node, and the key is automatically updated periodically; and the agent node has authentication, the authentication information is automatically updated periodically, and the task initiated by the user performs authority verification of the table resource, so that the security problem of the joint calculation scene of the data sources of the cross-data center is ensured on the premise of solving the data island.
The following technical effects are achieved through the technical scheme:
the embodiment provides a safe support cross-multisource data center combined computing method, which can realize cross-data center and cross-data source combined computing, and the proxy node obtains data of each data center and gathers the data to the center node for computing, and the security in the data computing process is ensured through key encryption transmission computing results and an application-approval mechanism, and each proxy node only receives requests initiated by the center node through verification of user credentials, so that the problem that data center data are independent and cannot be effectively associated with data island and the security problem of a cross-multisource data source combined computing scene are solved.
Exemplary apparatus
Based on the above embodiment, the present invention further provides a data processing apparatus supporting multi-source data center joint security computation, including:
the task management module is used for acquiring joint calculation task information of a cross-multi-source data center; the joint calculation task information is based on SQL specification;
the node management module is used for analyzing the joint calculation task information and sending corresponding calculation tasks to each proxy node;
and the joint calculation module is used for acquiring the data of the corresponding data center through each proxy node, carrying out joint calculation according to the acquired data and outputting the calculation result of the joint calculation task.
Based on the above embodiment, the present invention also provides a terminal, and a functional block diagram thereof may be shown in fig. 5.
The terminal comprises: the system comprises a processor, a memory, an interface, a display screen and a communication module which are connected through a system bus; wherein the processor of the terminal is configured to provide computing and control capabilities; the memory of the terminal comprises a storage medium and an internal memory; the storage medium stores an operating system and a computer program; the internal memory provides an environment for the operation of the operating system and computer programs in the storage medium; the interface is used for connecting external equipment such as mobile terminals, computers and other equipment; the display screen is used for displaying corresponding information; the communication module is used for communicating with a cloud server or a mobile terminal.
The computer program is configured to implement an operation supporting a multi-source data center federated secure computing data processing method when executed by a processor.
It will be appreciated by those skilled in the art that the functional block diagram shown in fig. 5 is merely a block diagram of some of the structures associated with the present inventive arrangements and is not limiting of the terminal to which the present inventive arrangements may be applied, and that a particular terminal may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.
In one embodiment, a terminal is provided, including: the system comprises a processor and a memory, wherein the memory stores a data processing program supporting multi-source data center joint security calculation, and the data processing program supporting multi-source data center joint security calculation is used for realizing the operation of the data processing method supporting multi-source data center joint security calculation when being executed by the processor.
In one embodiment, a computer readable storage medium is provided, wherein the computer readable storage medium stores a support multi-source data center joint security computing data processing program for implementing the operations of the support multi-source data center joint security computing data processing method as above when the support multi-source data center joint security computing data processing program is executed by a processor.
Those skilled in the art will appreciate that implementing all or part of the above-described methods may be accomplished by way of a computer program comprising instructions for the relevant hardware, the computer program being stored on a non-volatile storage medium, the computer program when executed comprising the steps of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in embodiments provided herein may include non-volatile and/or volatile memory.
In summary, the invention provides a method and a device for supporting multi-source data center joint security calculation data processing, wherein the method comprises the following steps: acquiring joint calculation task information crossing a multi-source data center; the joint calculation task information is based on SQL specification; analyzing the joint calculation task information and sending corresponding calculation tasks to each proxy node; acquiring data of a corresponding data center through each proxy node, performing joint calculation according to the acquired data, and outputting a calculation result of a joint calculation task; the invention solves the problems that the data of the data center is independent and cannot be effectively associated with the data island and the security problem of the scene is jointly calculated by crossing data sources of the data center.
It is to be understood that the invention is not limited in its application to the examples described above, but is capable of modification and variation in light of the above teachings by those skilled in the art, and that all such modifications and variations are intended to be included within the scope of the appended claims.
Claims (12)
1. A method for supporting multi-source data center joint security computing data processing, comprising:
acquiring joint calculation task information crossing a multi-source data center; the joint calculation task information is based on SQL specification;
analyzing the joint calculation task information and sending corresponding calculation tasks to each proxy node;
acquiring data of a corresponding data center through each proxy node, performing joint calculation according to the acquired data, and outputting a calculation result of a joint calculation task;
the acquiring the joint calculation task information of the cross-multi-source data center comprises the following steps:
generating a certificate required by encrypting and decrypting data between the proxy node and the central node through the openssl, and returning the certificate to the corresponding proxy node;
generating an RSA key pair, and encrypting the unique codes of all proxy nodes through the public key of the RSA key pair;
the method comprises the steps of acquiring data of a corresponding data center through each proxy node, carrying out joint calculation according to the acquired data, outputting a calculation result of a joint calculation task, and then comprising the following steps:
acquiring a write lock of a proxy node to be updated from a Zookeeper cluster;
and updating the key of the corresponding proxy node according to the acquired write lock, and releasing the write lock.
2. The method for supporting multiple source data center joint security computing data processing according to claim 1, wherein the acquiring joint computing task information across multiple source data centers previously comprises:
the IP and port information of the service are stored in a service database through a node registration module, and a unique number is generated for each proxy node.
3. The method for supporting multiple source data center joint security computing data processing according to claim 1, wherein the acquiring joint computing task information across multiple source data centers further comprises:
acquiring all machine IPs contained in the central node cluster, and returning the machine IPs to each proxy node in the form of an IP list; the IP list is used for generating a white list at each proxy node.
4. The method for supporting multiple source data center joint security computing data processing according to claim 1, wherein the acquiring joint computing task information across multiple source data centers further comprises:
and periodically updating certificates corresponding to each proxy node and encrypted node codes according to the request of each proxy node.
5. The method for supporting multiple source data center joint security computing data processing according to claim 1, wherein the obtaining joint computing task information across multiple source data centers comprises:
acquiring the joint calculation task information;
and analyzing the table resources in the joint calculation task information through an SQL analysis module, and authenticating the authority of the user node according to the analyzed table resources.
6. The method for supporting multiple source data center joint security computing data processing according to claim 1, wherein the parsing the joint computing task information and sending the corresponding computing task to each proxy node includes:
splitting the joint calculation task into a plurality of subtasks through an SQL analysis module;
and distributing a plurality of subtasks to corresponding proxy nodes.
7. The method for supporting multiple source data center joint security computing data processing according to claim 6, wherein the splitting the joint computing task into a plurality of sub-tasks by the SQL parsing module comprises:
acquiring a distributed read lock;
and submitting the corresponding subtasks according to the reading lock acquisition result, and releasing the reading lock after the corresponding subtasks are operated.
8. The method for supporting multi-source data center joint security calculation data processing according to claim 1, wherein the acquiring data of a corresponding data center by each proxy node, performing joint calculation according to the acquired data, comprises:
acquiring node authentication information fed back by each proxy node; the node authentication information is the result of encrypting the unique codes of the proxy nodes by the public key in the RSA key pair;
performing agent node authentication according to the node authentication information fed back by each agent node;
and acquiring data of the corresponding data center according to the authentication result, and performing joint calculation according to the acquired data.
9. The method for supporting multiple source data center joint security computing data processing according to claim 8, wherein the performing proxy node authentication according to the node authentication information fed back by each proxy node comprises:
decrypting by a private key in the RSA key pair;
if the decryption fails, rejecting the task request of the corresponding user node;
if the decryption is successful, judging whether the decrypted code is the code of the corresponding proxy node;
if the decrypted code is the code of the corresponding proxy node, the proxy node authentication is judged to be successful.
10. A data processing apparatus for supporting multi-source data center federated security computing, comprising:
the task management module is used for acquiring joint calculation task information of a cross-multi-source data center; the joint calculation task information is based on SQL specification;
the node management module is used for analyzing the joint calculation task information and sending corresponding calculation tasks to each proxy node;
the joint calculation module is used for acquiring data of a corresponding data center through each proxy node, carrying out joint calculation according to the acquired data and outputting a calculation result of a joint calculation task;
the acquiring the joint calculation task information of the cross-multi-source data center comprises the following steps:
generating a certificate required by encrypting and decrypting data between the proxy node and the central node through the openssl, and returning the certificate to the corresponding proxy node;
generating an RSA key pair, and encrypting the unique codes of all proxy nodes through the public key of the RSA key pair;
the method comprises the steps of acquiring data of a corresponding data center through each proxy node, carrying out joint calculation according to the acquired data, outputting a calculation result of a joint calculation task, and then comprising the following steps:
acquiring a write lock of a proxy node to be updated from a Zookeeper cluster;
and updating the key of the corresponding proxy node according to the acquired write lock, and releasing the write lock.
11. A terminal, comprising: a processor and a memory storing a multi-source data center-supporting joint security computing data processing program that, when executed by the processor, is operable to implement the multi-source data center-supporting joint security computing data processing method of any of claims 1-9.
12. A computer readable storage medium storing a multi-source data center joint security computing enabled data processing program which, when executed by a processor, is operable to implement the multi-source data center joint security enabled data processing method of any of claims 1-9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310500330.6A CN116208666B (en) | 2023-05-06 | 2023-05-06 | Processing method and device supporting multi-source data center joint security calculation data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310500330.6A CN116208666B (en) | 2023-05-06 | 2023-05-06 | Processing method and device supporting multi-source data center joint security calculation data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116208666A CN116208666A (en) | 2023-06-02 |
| CN116208666B true CN116208666B (en) | 2023-07-25 |
Family
ID=86513334
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310500330.6A Active CN116208666B (en) | 2023-05-06 | 2023-05-06 | Processing method and device supporting multi-source data center joint security calculation data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116208666B (en) |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114419507A (en) * | 2022-01-18 | 2022-04-29 | 中国石油大学(华东) | Internet factory operation diagnosis method and system based on federal learning |
Family Cites Families (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180136979A1 (en) * | 2016-06-06 | 2018-05-17 | Sitting Man, Llc | Offer-based computing enviroments |
| CN106202324B (en) * | 2016-06-30 | 2020-10-30 | 北京奇虎科技有限公司 | Data processing method and device for real-time computing platform |
| US10977260B2 (en) * | 2016-09-26 | 2021-04-13 | Splunk Inc. | Task distribution in an execution node of a distributed execution environment |
| US10452444B1 (en) * | 2017-10-19 | 2019-10-22 | Pure Storage, Inc. | Storage system with compute resources and shared storage resources |
| CN109710413B (en) * | 2018-12-29 | 2020-09-08 | 重庆誉存大数据科技有限公司 | Integral calculation method of rule engine system of semi-structured text data |
| CN111858739A (en) * | 2020-08-04 | 2020-10-30 | 浪潮卓数大数据产业发展有限公司 | Mapreduce-based data aggregation method and system |
| CN113407600B (en) * | 2021-08-18 | 2021-11-23 | 浩鲸云计算科技股份有限公司 | Enhanced real-time calculation method for dynamically synchronizing multi-source large table data in real time |
| CN113868295A (en) * | 2021-09-18 | 2021-12-31 | 支付宝(杭州)信息技术有限公司 | Data query method and device and multi-party security database |
| CN115130814B (en) * | 2022-05-10 | 2023-05-02 | 中南大学 | Privacy computing method and system for longitudinal data fusion |
| CN115913656A (en) * | 2022-10-28 | 2023-04-04 | 交通银行股份有限公司 | Bank cross-border data platform, task processing method and deployment method |
-
2023
- 2023-05-06 CN CN202310500330.6A patent/CN116208666B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114419507A (en) * | 2022-01-18 | 2022-04-29 | 中国石油大学(华东) | Internet factory operation diagnosis method and system based on federal learning |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116208666A (en) | 2023-06-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7168759B2 (en) | Systems and Methods for Supporting SQL-Based Rich Queries in Hyperledger Fabric Blockchain | |
| EP3688957B1 (en) | Managing a blockchain cloud service | |
| CN113711536B (en) | Extracting data from a blockchain network | |
| Bhaskaran et al. | Double-blind consent-driven data sharing on blockchain | |
| JP2022508011A (en) | Data processing methods, devices, electronic devices and computer programs based on blockchain networks | |
| US9503447B2 (en) | Secure communication between processes in cloud | |
| CN111144881A (en) | Selective access to asset transfer data | |
| US11863677B2 (en) | Security token validation | |
| US10911538B2 (en) | Management of and persistent storage for nodes in a secure cluster | |
| US11546425B2 (en) | Systems and methods of providing ledger as a service | |
| US20230370265A1 (en) | Method, Apparatus and Device for Constructing Token for Cloud Platform Resource Access Control | |
| US11822538B2 (en) | Systems and methods of transaction identification generation for transaction-based environment | |
| JP2022545683A (en) | Blockchain database management system | |
| Abraham et al. | Qualified eID derivation into a distributed ledger based IdM system | |
| CN109088858B (en) | Medical system and method based on authority management | |
| CN113612770A (en) | Cross-domain secure interaction method, system, terminal and storage medium | |
| US11418342B2 (en) | System and methods for data exchange using a distributed ledger | |
| CN110910110A (en) | Data processing method and device and computer storage medium | |
| CN111698198B (en) | Secret generation and share distribution | |
| CN116208666B (en) | Processing method and device supporting multi-source data center joint security calculation data | |
| CN115622812A (en) | Digital identity verification method and system based on block chain intelligent contract | |
| CN117040930B (en) | Resource processing method, device, product, equipment and medium of block chain network | |
| US20240143730A1 (en) | Multi-factor authentication using blockchain | |
| Velthuis | New authentication mechanism using certificates for big data analytic tools | |
| CN117255081A (en) | Data processing method and device based on block chain system, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |