CN114143039A - Global multistage unified safe data transmission method and server cluster - Google Patents
Global multistage unified safe data transmission method and server cluster Download PDFInfo
- Publication number
- CN114143039A CN114143039A CN202111307940.1A CN202111307940A CN114143039A CN 114143039 A CN114143039 A CN 114143039A CN 202111307940 A CN202111307940 A CN 202111307940A CN 114143039 A CN114143039 A CN 114143039A
- Authority
- CN
- China
- Prior art keywords
- data transmission
- task
- data
- streaming
- circulation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000005540 biological transmission Effects 0.000 title claims abstract description 89
- 238000000034 method Methods 0.000 title claims abstract description 38
- 230000007246 mechanism Effects 0.000 claims abstract description 14
- 238000012546 transfer Methods 0.000 claims description 20
- 238000001514 detection method Methods 0.000 claims description 8
- 238000012795 verification Methods 0.000 claims description 8
- 238000004590 computer program Methods 0.000 claims description 7
- 239000011159 matrix material Substances 0.000 claims description 5
- 230000002457 bidirectional effect Effects 0.000 abstract description 6
- 238000010586 diagram Methods 0.000 description 8
- 238000013461 design Methods 0.000 description 7
- 230000008901 benefit Effects 0.000 description 6
- 230000008569 process Effects 0.000 description 4
- 230000010076 replication Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000007689 inspection Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000007792 addition Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 239000000969 carrier Substances 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000011144 upstream manufacturing Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/10—Network architectures or network communication protocols for network security for controlling access to devices or network resources
- H04L63/101—Access control lists [ACL]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
- G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
- H04L9/32—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols including means for verifying the identity or authority of a user of the system or for message authentication, e.g. authorization, entity authentication, data integrity or data verification, non-repudiation, key authentication or verification of credentials
- H04L9/321—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols including means for verifying the identity or authority of a user of the system or for message authentication, e.g. authorization, entity authentication, data integrity or data verification, non-repudiation, key authentication or verification of credentials involving a third party or a trusted authority
- H04L9/3213—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols including means for verifying the identity or authority of a user of the system or for message authentication, e.g. authorization, entity authentication, data integrity or data verification, non-repudiation, key authentication or verification of credentials involving a third party or a trusted authority using tickets or tokens, e.g. Kerberos
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
- H04L9/32—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols including means for verifying the identity or authority of a user of the system or for message authentication, e.g. authorization, entity authentication, data integrity or data verification, non-repudiation, key authentication or verification of credentials
- H04L9/3247—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols including means for verifying the identity or authority of a user of the system or for message authentication, e.g. authorization, entity authentication, data integrity or data verification, non-repudiation, key authentication or verification of credentials involving digital signatures
Landscapes
- Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Hardware Design (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Databases & Information Systems (AREA)
- Bioethics (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a global multi-level unified safe data transmission method and a server cluster, which create a flow task by creating a message Service (SDK), dynamically recombining cluster nodes for cascade transmission through a cascade data transmission structure of a forest topology structure and customizing a flow relation, then performing data transmission on the created flow task through the SDK, and ensuring the safety of data transmission among multi-level multi-center clusters through ACL authority control and Token mechanisms in the data transmission process, thereby realizing the dynamic customization of reliable bidirectional data flow among the multi-level clusters and finally improving the data flow efficiency among the multi-level clusters.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a global multi-level unified and safe data transmission method and a server cluster.
Background
When data transmission is performed in a multi-center and multi-level complex computer cluster environment, for example, as shown in fig. 1, a data center is deployed in a cluster 0, lower-level computer devices of the data center are respectively deployed in different computer clusters, for example, a computer device 1 is deployed on the cluster 1, and a computer device 2 is deployed on the cluster 2, if data transmission is performed between the data center and lower-level devices thereof at this time, data streaming needs to be performed between the clusters in a synchronous manner, that is, when the data center needs to send data to the device 1, the cluster 0 needs to synchronously stream the data to the cluster 1, and similarly, when the device 2 needs to send data to the data center, the cluster 2 needs to synchronously stream the data to the cluster 0, and the principle of data cascading among the multi-level and multi-center clusters is also the same.
The existing mainstream message middleware technology comprises Kafka, pulser, rocktmq and the like, all of which have implementation schemes for providing different message cross-cluster synchronization functions, and focuses on providing a technical scheme for full data synchronization among computer clusters distributed in different regions, so as to ensure data consistency among regions, such as a Geo Replication cross-cluster message Replication scheme of pulser, a MirrorMaker cross-cluster message Replication scheme of Kafka and the like. However, the existing cross-cluster message synchronization technology cannot provide a service for data transmission through dynamically recombining a multi-level and multi-center computer cluster, so that reliable bidirectional data transfer among dynamically customized multi-level clusters cannot be realized, and the data transfer efficiency among the multi-level clusters is reduced.
Disclosure of Invention
The invention provides a global multi-level unified safe data transmission method and a server cluster, which aim to solve the problem that reliable bidirectional data transfer between multi-level clusters cannot be efficiently carried out in the prior art.
In a first aspect, the present invention provides a global multi-level unified secure data transmission method, where the method includes: developing a message Service (SDK) to provide messaging capability between message clients; designing a cascade data transmission structure based on forest topology, dynamically recombining cluster nodes of cascade transmission according to the cascade transmission structure, customizing a circulation relation, and creating a circulation task; performing data transmission on the created circulation task based on the message service SDK; meanwhile, the safety of data transmission among the multi-stage and multi-center clusters is guaranteed through ACL authority control and Token mechanism.
Optionally, the method further comprises: and dynamically recombining the cascade transmission nodes based on the user indication and customizing the stream forwarding relation.
Optionally, the plurality of circulation tasks include a plurality of circulation tasks, and a corresponding relationship between the circulation tasks is established by enabling multiple threads according to each circulation task and a circulation relationship thereof, so that one service supports customization of the plurality of circulation tasks.
And establishing a message service client by using multiple threads according to the streaming task cluster node and the streaming relation thereof by accessing the message service SDK, and establishing the corresponding relation between the streaming tasks so as to realize cross-cluster data transmission of the streaming tasks.
Before data transmission, detecting whether a loop exists in a streaming task, wherein the loop detection comprises single-task loop detection and multi-task loop detection, the single-task loop detection is used for detecting whether data in two streaming directions are repeated, namely if the data are in the reported streaming direction and the delivered streaming direction, the streaming task exists in a loop path; the multi-task loopback check is based on all task flow relationships and flow data.
The method comprises the steps of constructing a directed graph of all tasks in a adjacency matrix mode, and detecting whether a loop path exists in the directed graph or not based on depth-first traversal DFS.
The Token mechanism adopts the client IP encrypted by MD5 as Token identification for identity authentication.
The ACL authority control comprises two parts of authority analysis and authority verification;
the permission resolution comprises: the server analyzes the request of the client to obtain an attribute field needing authentication;
the permission check comprises the following steps: checking whether a global IP full list is hit, checking whether a user IP full list is hit, checking the signatures of the Access Key and the SecretKey, and checking whether the authority required by the request is matched with the authority owned by the user.
In a second aspect, the present invention provides a plurality of server clusters, and data transmission is performed between the plurality of server clusters by using any of the above methods.
In a third aspect, the present invention provides a computer-readable storage medium, which stores a signal-mapped computer program, and when the computer program is executed by at least one processor, the computer program implements the steps of any one of the global multi-level unified secure data transmission methods described above.
The invention has the following beneficial effects:
the invention provides data transceiving capacity by developing and designing a message service SDK and designs a cascade data transmission structure. Dynamically recombining cluster nodes of cascade transmission according to the cascade data transmission structure and customizing a data flow relation to establish a flow task; then, data transmission is carried out on the created circulation task through a message Service (SDK); and the safety of data transmission among the multi-level multi-center clusters is guaranteed through ACL authority control and Token mechanism, so that reliable bidirectional data transfer among the multi-level clusters is dynamically customized, and the data transfer efficiency among the multi-level clusters is finally improved.
The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:
FIG. 1 is a schematic diagram of data transmission between existing multi-level, multi-center computer clusters;
fig. 2 is a schematic flowchart of a global multi-level unified secure data transmission method according to an embodiment of the present invention;
FIG. 3 is a diagram illustrating an internal interface of a message service SDK according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a forest topology-based cascade transmission structure design according to an embodiment of the present invention;
fig. 5 is a flowchart illustrating a global multi-level multi-center data transmission method according to an embodiment of the present invention;
fig. 6 is a schematic diagram of data transmission based on the data streaming service according to an embodiment of the present invention;
fig. 7 is a schematic diagram of a data flow service across cluster nodes according to an embodiment of the present invention;
FIG. 8 is a schematic diagram illustrating an implementation principle of a depth-first traversal directed graph according to an embodiment of the present invention;
fig. 9 is a flowchart of an implementation of data transmission between cascaded clusters based on a message service SDK according to an embodiment of the present invention;
fig. 10 is a schematic flowchart of the process of performing client authentication based on Token mechanism according to the embodiment of the present invention;
fig. 11 is a logic diagram illustrating access control of a message service to a client request right through an ACL according to an embodiment of the present invention.
Detailed Description
Aiming at the problem that reliable bidirectional data transfer among multi-level clusters cannot be efficiently carried out in the prior art, cluster nodes which are in cascade transmission are dynamically recombined through a cascade data transmission structure of a forest topological structure, a transfer relation is customized, a transfer task is created, then the created transfer task is subjected to data transmission through a message Service (SDK), and the safety of the data transmission among the multi-level and multi-center clusters is guaranteed through Access Control List (ACL) authority control and Token mechanism in the data transmission process, so that the reliable bidirectional data transfer among the multi-level clusters is dynamically customized, and the data transfer efficiency among the multi-level clusters is finally improved. The present invention will be described in further detail below with reference to the drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the invention.
A first embodiment of the present invention provides a global multi-level unified secure data transmission method, referring to fig. 2, the method includes:
s201, creating a message Service (SDK) to provide message receiving and sending capability between message clients;
in particular, embodiments of the present invention design and develop a message service SDK that provides messaging capabilities between message clients. Where a messaging client represents a program, whether a service or a client program, that uses a messaging service for messaging. The message service provides an access SDK, and the schematic diagram of the internal interface of the message service SDK is shown in fig. 3;
s202, designing a cascade data transmission structure of a forest topological structure, dynamically recombining cluster nodes according to a cascade data transmission interface and customizing a data flow relation, and creating a flow task; and carrying out data transmission on the created circulation task based on the message service SDK, and ensuring the safety of data transmission among the multi-stage and multi-center clusters through ACL authority control and Token mechanism in the data transmission process.
That is, according to the requirement characteristics of data transmission among the multi-level and multi-center clusters, the embodiment of the present invention designs a cascading data transmission architecture based on a forest topology structure to improve the efficiency and reliability of data transmission, dynamically recombines cluster nodes of cascading transmission and customizes a flow relationship through the cascading data transmission structure, creates a flow task, and realizes data transmission of the flow task based on the message service SDK.
Specifically, the embodiment of the invention designs a cascading data transmission structure based on forest topology, as shown in fig. 4. Compared with other topological structures such as a star structure and a ring structure, the forest topological structure is considered to be a hierarchical multi-center centralized control type network interface, and the forest topological structure has the advantages of being high in reliability, low in data transmission delay and the like. Therefore, aiming at the requirement characteristics of data transmission among the multi-stage multi-center clusters, in order to ensure the reliability and high efficiency of data transmission in the environment, the embodiment of the invention designs a cascade transmission structure based on forest topology;
in addition, the embodiment of the invention also reconstructs parent domain cluster nodes and sub-domain cluster nodes dynamically according to the cascade transmission structure, customizes data flow transfer relation, and comprises reporting and issuing two transfer directions and creating a transfer task; in addition, one circulation task can customize two circulation directions simultaneously. The data are reported when the subdomain cluster transfers to the parent domain cluster, and the data are issued when the parent domain cluster transfers to the subdomain cluster;
by the creation of the circulation task, the cross-inter-cluster data flow circulation service provided by the invention simultaneously supports the customization of a plurality of circulation tasks;
in an embodiment of the present invention, according to the created flow task, before data flow is performed between the parent domain and the child domain cluster nodes, performing a loopback check process on the flow task, including:
a) the loop check of a single task, before the single task carries out data circulation, the single task checks whether data in two circulation directions are repeated or not, and if the data are repeated, the task has a loop path;
b) and performing loop check on a plurality of tasks, wherein the multi-task stores the data content by adopting a directed graph data structure represented by an adjacent matrix before data circulation, and performs loop check processing on the plurality of circulation tasks by adopting a depth-first search directed graph loop algorithm.
In the embodiment of the invention, based on the message service SDK, according to the reporting or issuing relation of the circulation task, a message service client is established through multithreading, and a message receiving and sending interface provided by the SDK is called to realize the data circulation among the multi-level clusters.
In specific implementation, in order to guarantee the safety of data flow transfer among the multi-stage multi-center clusters, the safety and controllability of the data flow are guaranteed by using a Token mechanism and ACL authority control;
in the embodiment of the invention, the client IP address encrypted by MD5 is used as Token to carry out client identity authentication; and after the ACL verification is started by the server, performing authority analysis and authority verification on the client request, and performing access control on the resource level.
The method according to an embodiment of the invention will be explained and illustrated in detail below with reference to fig. 5-11:
a) the message middleware is a middleware which uses messages as carriers to carry out communication, realizes a large amount of data exchange among different applications by using an efficient and reliable message mechanism, and is an important supporting transmission component in a distributed application system. In large distributed systems, message middleware typically employs a communication model of message queues. Message middleware of current mainstream includes rockmq, Kafaka, RabbitMQ, and the like.
b) The common topological structures of the forest structures mainly comprise star structures, ring structures, forest structures and the like. The star structure is formed by connecting all workstations in a star mode into a network, the network only has one central node, and other nodes are directly connected with the central node, so that the star structure has the advantages of simple structure, convenience in management and small network delay, has obvious defects, is highly dependent on the central node, can cause the breakdown of the whole data transmission structure once the central node is broken down, and has low reliability; the ring structure is a closed ring formed by connecting a plurality of nodes in a network end to end through point-to-point links, and has no central node, and only an upstream node needs to receive data and send the data to a downstream node during data transmission, but the most defect is that the whole structure is crashed when any one node crashes or any one link crashes depending on the reliability of all the nodes; when the number of nodes in the ring is too many, data is generally transmitted from one node to another node for multiple times, which inevitably causes higher delay;
compared with a star structure and a ring structure, the forest structure has more advantages. a) The system is a hierarchical centralized control type network, and can take upper nodes as authoritative data check nodes so as to ensure the correctness of data; b) if the forest structure is properly established, the average time complexity of forwarding data from one node to the other nodes is O (logn), and the forest structure is relatively better; c) even if the upper node crashes, the subtree can also independently perform data forwarding through networking, so that the crash of the whole structure is avoided, and the method is relatively reliable. Therefore, the forest structure is selected to be more suitable for a multi-level and multi-center data transmission scheme.
d) The Token mechanism: the Token is a string of character strings generated by the server and used as an identifier (Token) for the client to request, when the client requests, the server generates a Token, and the identity of the client is verified through the Token; there are two common ways to verify identity using Token: first, use device number/device mac address as Token id; second, a session value is used as Token identification.
c) ACL: access Control Lists (ACLs), which are packet filtering based Access Control techniques, can filter data packets on an interface according to a set condition, and allow the data packets to pass through or drop. ACL can relate to concepts of users, resources, rights, roles and the like, and by means of the ACL, the access of the users to the network can be effectively controlled, so that the network security is guaranteed to the maximum extent.
e) Directed graph: a graph is a special data structure, each data element can be arbitrarily associated with another, and a typical graph structure is composed of vertices (data elements) and edges (lines connecting the vertices). A directed graph is a graph in which each edge has directionality.
Fig. 5 is a flow chart of the multilevel multi-center secure unified data transmission method. As shown in fig. 5, the multi-level multi-center secure unified data transmission method of the present invention includes the following steps:
in step 501, a data stream forwarding relationship (including reporting and issuing) is customized and a forwarding task is created by dynamically recombining a parent domain cluster node and a sub-domain cluster node which are in cascade transmission;
wherein, according to some embodiments of the invention, as shown in fig. 6, one streaming task may customize a bi-directional streaming relationship; in addition, the data flow service supports simultaneous customization of a plurality of flow tasks, and the corresponding relation between the flow tasks is established by starting multithreading according to each flow task and the flow relation thereof, so that one service supports the customization of the plurality of flow tasks;
then, in step 502, a multi-thread message service client is created according to the created circulation task and the message service SDK according to the task circulation relationship, so as to implement cross-cluster data transmission of the circulation task. This step 502 comprises the following sub-steps:
checking whether a loop path exists or not according to the created circulation task, wherein the check comprises loop check of a single task and loop check of a plurality of tasks;
the loopback check of a single circulation task is to check whether the data in two circulation directions are repeated, namely if the data is in the reported circulation direction and the delivered circulation direction, the circulation task has a loopback path;
performing loopback inspection on a plurality of looped tasks, constructing a directed graph of all tasks in an adjacent matrix mode according to all task circulation relations and circulation data and improving retrieval efficiency, and detecting whether a loopback path exists in the directed graph based on depth-first traversal (DFS), wherein the loopback inspection is shown in FIGS. 7-8;
wherein, in the adjacency matrix, the rows represent parent domain clusters (source addresses) and the columns represent child domain clusters (destination addresses); DFS has the following steps:
a) assuming that all the vertexes in the initial state are not visited, starting from each vertex v, visiting the vertex first;
b) sequentially starting from each non-accessed adjacent point of v, performing depth-first traversal on the graph until all vertexes communicated with the v path in the graph are accessed;
c) if other vertexes are not accessed in the graph at the moment, selecting one vertex which is not accessed as a starting point, and performing depth-first traversal again;
d) the above process is repeated until all vertices in the graph have been visited.
According to the circulation tasks and the circulation relation thereof, based on the message service SDK, establishing a message service client through multithreading, establishing the corresponding relation between the circulation tasks, and realizing the cross-cluster data transmission of the circulation tasks, as shown in FIG. 9;
the message service is an efficient, reliable, safe, convenient and extensible distributed message service, provides message transmission capability between message clients, currently supports access SDK of Java language and C + + language, takes message service SDK of Java version as an example, and takes the class of "ac.nci.xt4b.messageclient.client" as the interface definition of the message clients, and the main interface definition and function description are as follows:
the class of "ac.nci.xt4b.messageclient.impl.clusterMqclient" is realized by an interface of a message client.
In addition, according to the embodiment of the present invention, in order to enable the multi-level multi-center unified data transmission technical solution to have security controllability, security of data transmission is guaranteed through a Token mechanism and ACL authority control, as shown in fig. 10 to 11.
As shown in fig. 10, when the message client sends a connection service request, the IP address encrypted by MD5 is used as a parameter and transmitted to the server, the server obtains the client IP address according to the connection information of the message client and performs MD5 encryption as Token, when the message client requests a connection service, the Token transmitted by the client and the Token obtained by the server are compared, if the connection service is the same, the connection service passes verification, the client succeeds in connecting the service, and if the connection service is not the same, the connection service is rejected.
ACL rights control provides mainly resource level user access control for message services. The message service injects an Access Key and a SecretKey signature into the client through RPCHook, when the service end starts ACL verification and the message client after the connection service succeeds requests the service end to send/subscribe the message, the service end verifies the authority possessed by the Access Key in the client request, and if the verification fails, the exception is thrown. In addition, the ACL main flow mainly includes two parts, namely, right resolution and right verification, and the specific logic is as shown in fig. 11. The specific process mainly comprises the following steps: the server analyzes the request of the client to obtain an attribute field needing authentication; the server side carries out a series of checks on the authority, including: checking whether a global IP full list is hit; checking whether the IP full list of the user is hit; checking the signatures of Access Key and SecretKey; checking whether the rights required by the request match the rights owned by the user, etc.
Generally speaking, the embodiment of the invention designs and develops a message service SDK providing message receiving and sending capability; a cascade data transmission path based on a forest topological structure is designed; according to the cascade transmission structure, creating a transfer task by dynamically recombining a parent domain cluster and a sub-domain cluster and customizing a data transfer relation (including reporting and transferring); through the circulation task, according to a circulation relation, unified data circulation among the cascade clusters is realized based on the message service SDK; the security in the data transmission process among the cross-clusters is guaranteed through a Token mechanism and ACL authority control; meanwhile, in the data transmission process, one task is supported, the two-way circulation direction can be customized simultaneously, in addition, the loop check processing is carried out on all circulation tasks based on the directed graph loop algorithm of the depth-first retrieval, so that the safe and controllable multi-stage and multi-center unified data circulation is realized, and effective and reliable service is provided for the data transmission among the multi-stage and multi-center clusters.
A second embodiment of the present invention provides a plurality of server clusters, and data transmission is performed among the plurality of server clusters by using any one of the global multi-level unified secure data transmission methods in the first embodiment of the present invention. The relevant content of the embodiments of the present invention can be understood by referring to the first embodiment of the present invention, and will not be discussed in detail herein.
A third embodiment of the present invention provides a computer-readable storage medium, which stores a signal-mapped computer program, and when the computer program is executed by at least one processor, the computer program implements the global multi-level unified secure data transmission method according to any one of the first embodiments of the present invention. The relevant content of the embodiments of the present invention can be understood by referring to the first embodiment of the present invention, and will not be discussed in detail herein.
Although the preferred embodiments of the present invention have been disclosed for illustrative purposes, those skilled in the art will appreciate that various modifications, additions and substitutions are possible, and the scope of the invention should not be limited to the embodiments described above.
Claims (10)
1. A global multi-level unified secure data transmission method is characterized by comprising the following steps:
creating a message Service (SDK) to provide messaging capability between message clients;
dynamically recombining cluster nodes of cascade transmission through a cascade data transmission structure of a forest topological structure, customizing a flow relation, creating a flow task, performing data transmission on the created flow task through a message Service (SDK), and ensuring the safety of data transmission among the multi-stage and multi-center clusters through ACL authority control and Token mechanism in the data transmission process.
2. The method of claim 1, further comprising:
and dynamically recombining the cascade transmission nodes based on the user indication and customizing the stream forwarding relation.
3. The method of claim 1,
the method comprises the steps that a plurality of circulation tasks are included, and the corresponding relation among the circulation tasks is established by starting multiple threads according to the circulation tasks and the circulation relation thereof, so that one service can support the customization of the plurality of circulation tasks.
4. The method of claim 1,
and establishing a message service client by using multiple threads according to the streaming task cluster node and the streaming relation thereof by accessing the message service SDK, and establishing the corresponding relation between the streaming tasks so as to realize cross-cluster data transmission of the streaming tasks.
5. The method of claim 1,
before data transmission, detecting whether a loop exists in a streaming task, wherein the loop detection comprises single-task loop detection and multi-task loop detection, the single-task loop detection is used for detecting whether data in two streaming directions are repeated, namely if the data are in the reported streaming direction and the delivered streaming direction, the streaming task exists in a loop path; the multi-task loopback check is based on all task flow relationships and flow data.
6. The method of claim 5,
the method comprises the steps of constructing a directed graph of all tasks in a adjacency matrix mode, and detecting whether a loop path exists in the directed graph or not based on depth-first traversal DFS.
7. The method according to any one of claims 1 to 6,
the Token mechanism adopts the client IP encrypted by MD5 as Token identification for identity authentication.
8. The method according to any one of claims 1 to 6,
the ACL authority control comprises two parts of authority analysis and authority verification, wherein the authority analysis comprises the following steps: the server analyzes the request of the client to obtain an attribute field needing authentication; the permission check comprises the following steps: checking whether a global IP full list is hit, checking whether a user IP full list is hit, checking the signatures of the Access Key and the SecretKey, and checking whether the authority required by the request is matched with the authority owned by the user.
9. A server cluster, comprising: the server cluster is a plurality of server clusters, and data transmission is carried out among the server clusters by adopting the global multi-level unified safe data transmission method as claimed in any one of claims 1 to 8.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the steps of the global multi-level unified secure data transfer method according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111307940.1A CN114143039B (en) | 2021-11-05 | 2021-11-05 | Global multistage unified secure data transmission method and server cluster |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111307940.1A CN114143039B (en) | 2021-11-05 | 2021-11-05 | Global multistage unified secure data transmission method and server cluster |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114143039A true CN114143039A (en) | 2022-03-04 |
| CN114143039B CN114143039B (en) | 2024-04-16 |
Family
ID=80392398
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111307940.1A Active CN114143039B (en) | 2021-11-05 | 2021-11-05 | Global multistage unified secure data transmission method and server cluster |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114143039B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101252506A (en) * | 2007-12-29 | 2008-08-27 | 中国建设银行股份有限公司 | Data transmission system |
| CN110611621A (en) * | 2019-09-26 | 2019-12-24 | 上海依图网络科技有限公司 | Tree-structured multi-cluster routing control method and cluster forest |
| CN111865935A (en) * | 2020-06-30 | 2020-10-30 | 北京天融信网络安全技术有限公司 | Data transmission system |
| CN112235285A (en) * | 2020-10-12 | 2021-01-15 | 山东健康医疗大数据有限公司 | Method and system for user authentication mode and service based on non-session |
| US11165859B1 (en) * | 2021-04-09 | 2021-11-02 | Area 1 Security, Inc. | Scaling stateful services hosted in a cluster of server nodes |
-
2021
- 2021-11-05 CN CN202111307940.1A patent/CN114143039B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101252506A (en) * | 2007-12-29 | 2008-08-27 | 中国建设银行股份有限公司 | Data transmission system |
| CN110611621A (en) * | 2019-09-26 | 2019-12-24 | 上海依图网络科技有限公司 | Tree-structured multi-cluster routing control method and cluster forest |
| CN111865935A (en) * | 2020-06-30 | 2020-10-30 | 北京天融信网络安全技术有限公司 | Data transmission system |
| CN112235285A (en) * | 2020-10-12 | 2021-01-15 | 山东健康医疗大数据有限公司 | Method and system for user authentication mode and service based on non-session |
| US11165859B1 (en) * | 2021-04-09 | 2021-11-02 | Area 1 Security, Inc. | Scaling stateful services hosted in a cluster of server nodes |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114143039B (en) | 2024-04-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Correia et al. | Byzantine consensus in asynchronous message-passing systems: a survey | |
| Van Renesse et al. | Astrolabe: A robust and scalable technology for distributed system monitoring, management, and data mining | |
| Hu et al. | Anomaly detection system in secure cloud computing environment | |
| Martin et al. | A framework for dynamic byzantine storage | |
| US11095476B2 (en) | Spanning tree protocol enabled n-node link aggregation system | |
| Neves et al. | Solving vector consensus with a wormhole | |
| Alvisi et al. | How robust are gossip-based communication protocols? | |
| Biswas et al. | A novel leader election algorithm based on resources for ring networks | |
| Newport | Consensus with an abstract MAC layer | |
| Rodrigues et al. | A distributed k-mutual exclusion algorithm based on autonomic spanning trees | |
| Tu et al. | Nearcast: A locality-aware P2P live streaming approach for distance education | |
| Herault et al. | A model for large scale self-stabilization | |
| CN114143039B (en) | Global multistage unified secure data transmission method and server cluster | |
| de Oliveira et al. | An adaptive security management model for emergency networks | |
| Shao et al. | Accelerating bgp configuration verification through reducing cycles in smt constraints | |
| US10397099B2 (en) | Spanning tree protocol enabled N-node link aggregation system | |
| Mehrabian et al. | Randomized rumor spreading in poorly connected small‐world networks | |
| Reiter et al. | Distributed construction of a fault-tolerant network from a tree | |
| Li et al. | DSL: dynamic and self‐learning schedule method of multiple controllers in SDN | |
| Kumar et al. | HT‐Paxos: High Throughput State‐Machine Replication Protocol for Large Clustered Data Centers | |
| Zheng et al. | Research on SDN-based mimic server defense technology | |
| Lembke et al. | Consistent and secure network updates made practical | |
| da Silva et al. | A secure and distributed control plane for software defined networks | |
| Vigfusson | Affinity in distributed systems | |
| Ying et al. | LOPE: A Low‐Overhead Payment Verification Method for Blockchains |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |