CN114666348B - Method for quickly constructing distributed system based on python language - Google Patents
Method for quickly constructing distributed system based on python language Download PDFInfo
- Publication number
- CN114666348B CN114666348B CN202210574788.1A CN202210574788A CN114666348B CN 114666348 B CN114666348 B CN 114666348B CN 202210574788 A CN202210574788 A CN 202210574788A CN 114666348 B CN114666348 B CN 114666348B
- Authority
- CN
- China
- Prior art keywords
- node
- distributed system
- cluster
- nodes
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Devices For Executing Special Programs (AREA)
- Multi Processors (AREA)
Abstract
The application particularly relates to a method for quickly constructing a distributed system based on a python language. The method comprises the following steps: associating the node and the IP port information and forming associated information; configuring the associated information in the distributed system; when a node is started, the distributed system sends a message out along the node, and other nodes learn about the starting of the node; and if the new node is added to the distributed system, the new node is added to any node to form a first cluster with synchronous information, wherein the associated information is configured in the distributed system, and information is sent based on the distributed system to synchronize the information flow of each node, and if the new node is added to the distributed system, the new node is added to any node to form the first cluster with synchronous information, so that the distributed system is quickly built based on each node, other single-machine projects are quickly expanded by a distributed framework of the distributed system, and signal synchronization is carried out based on the distributed system.
Description
Technical Field
The application relates to the technical field of distributed systems, in particular to a method for quickly constructing a distributed system based on python language.
Background
With the rapid development of cloud computing technology and the requirement of digital transformation of enterprise assets, the demand of various cloud service software is greatly increased.
In a distributed system, a set of independent computers appear to a user as a unified whole, as if it were a system. There are many distributed products such as hadoop hdfs (distributed storage system), spark (distributed computing system), storm (distributed streaming computing system), elasticsearch (distributed search system), kafka (distributed publish-subscribe message system), etc. but due to the complex theory of distribution, it is very difficult to construct a distributed framework from scratch, and there is no general distributed framework in the python field for a while.
Disclosure of Invention
In order to solve the problem that a distributed framework is difficult to establish, the application provides a method for quickly establishing a distributed system based on a python language, the associated information is configured in the distributed system, information is sent based on the distributed system to synchronize information flow of each node, if a new node is added to the distributed system, the new node is added to any one node to form a first cluster with information synchronization, so that the distributed system is quickly established based on each node, other single-machine projects can be quickly expanded by the distributed framework of the distributed system, and signal synchronization is carried out based on the distributed system.
The method is realized by adopting the following technical scheme:
a method for quickly constructing a distributed system based on a python language is applied to construction of a universal distributed framework, and comprises the following steps:
associating the node and the IP port information and forming associated information;
configuring the association information within a distributed system;
when one node is started, the distributed system sends out messages along the node, and other nodes learn about the starting of the node;
and if the new node is added into the distributed system, adding the new node into any one node to form a first cluster with synchronous information.
Optionally, the distributed system is formed based on mutual correspondence between a plurality of the nodes and under the creation of python language.
Optionally, when each undetermined node joins the first cluster, the undetermined node reports its own IP port information to the first cluster, and transmits the IP port information to each node in the first cluster through the first cluster.
Optionally, each node of the first cluster receives the IP port information, and brings the undetermined node into the first cluster to form a second cluster.
Optionally, information synchronization is performed between nodes of the same cluster based on the packet, and the start signal sent by any one of the nodes is synchronized with other nodes.
Optionally, the new node enters any one of the nodes, and adjusts the configuration of the first cluster; and the node receives the IP port information of the new node and constructs signal transmission with the node of the same cluster.
Optionally, the nodes are docked based on the same distributed system, and the nodes perform information synchronization.
Optionally, the self-defined service logic data is performed based on the node; and sharing the service logic data to each node in the cluster.
The technical scheme provided by the application has the following beneficial effects:
according to the method for quickly constructing the distributed system based on the python language, the associated information is configured in the distributed system, information is sent based on the distributed system to synchronize information flow of each node, if a new node is added to the distributed system, the new node is added to any one node to form a first cluster with information synchronization, so that the distributed system is quickly constructed based on each node, other single-machine projects can be quickly expanded by a distributed framework of the distributed system, and signal synchronization is carried out based on the distributed system.
These and other aspects of the present application will be more readily apparent from the following description of the embodiments. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the description of the exemplary embodiments or the related art will be briefly introduced below, which are used to provide further understanding of the present application and constitute a part of the specification, and together with the embodiments of the present application, serve to explain the present application and not to limit the present application. In the drawings:
fig. 1 is a flowchart of a method for quickly constructing a distributed system based on python language according to the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of and not restrictive on the broad application.
In some of the flows described in the specification and claims of this application and in the above-described figures, a number of operations are included that occur in a particular order, but it should be clearly understood that these operations may be performed out of order or in parallel as they occur herein, the number of operations, e.g., 101, 102, etc., merely being used to distinguish between various operations, and the number itself does not represent any order of performance. Additionally, the flows may include more or fewer operations, and the operations may be performed sequentially or in parallel. It should be noted that, the descriptions of "first", "second", etc. in this document are used for distinguishing different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit the types of "first" and "second".
The technical solutions in the exemplary embodiments of the present application will be clearly and completely described below with reference to the drawings in the exemplary embodiments of the present application, and it is obvious that the described exemplary embodiments are only a part of the embodiments of the present application, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
In a distributed system, a set of independent computers appear to a user as a unified whole, as if it were a system. There are many distributed products such as hadoop hdfs (distributed storage system), spark (distributed computing system), storm (distributed streaming computing system), elasticsearch (distributed search system), kafka (distributed publish-subscribe message system), etc. but due to the complex theory of distribution, it is very difficult to construct a distributed framework from scratch, and there is no general distributed framework in the python field for a while.
In addition, to realize a distributed system, a distributed consistency algorithm is realized first, and the existing distributed consistency algorithms include Paxos, raft, ZAB and Gossip. Where the raft calculation is the most understandable and most applicable consensus algorithm in comparison.
The technical scheme of the application is further explained by combining the specific embodiment as follows:
in order to solve the problem that a distributed framework is difficult to establish, the application provides a method for quickly establishing a distributed system based on a python language, the associated information is configured in the distributed system, information is sent based on the distributed system to synchronize information flow of each node, if a new node is added to the distributed system, the new node is added to any one node to form a first cluster with information synchronization, so that the distributed system is quickly established based on each node, other single-machine projects can be quickly expanded by the distributed framework of the distributed system, and signal synchronization is carried out based on the distributed system.
Referring to fig. 1, an embodiment of the present application provides a method for quickly building a distributed system based on a python language, which is applied to building a universal distributed framework, and the method for quickly building a distributed system based on a python language includes:
and S10, associating the node information with the IP port information, and forming association information.
And S20, configuring the associated information in a distributed system.
And S30, when one node is started, the distributed system sends out the message along the node, and other nodes learn about the starting of the node.
And S40, if a new node is added into the distributed system, adding the new node into any one node to form a first cluster with synchronous information.
In the embodiment of the application, the node and the IP port information are associated, and the association information is formed.
The nodes and the IP port information have corresponding relations and are in one-to-one correspondence, and the association information is formed based on the association of the nodes and the IP port information, so that the nodes and the IP port information are embedded into the distributed system based on the association information, and the information in each node in the distributed system is synchronized.
In one embodiment of the application, the distributed system is formed based on mutual correspondence between a plurality of the nodes and under creation of python language, wherein mutual correspondence between a plurality of the nodes and as a part of the distributed system, at this time, a plurality of the nodes form the distributed system under creation of python language, and construction of channels between the respective nodes is performed based on python language.
In an embodiment of the present application, when each undetermined node joins the first cluster, the undetermined node reports its own IP port information to the first cluster, and transmits the IP port information to each node in the first cluster through the first cluster.
The first cluster is used as an aggregate of all undetermined nodes, and information sharing is carried out on the undetermined nodes so as to classify and process the undetermined nodes.
In an embodiment of the present application, each node waiting for the first cluster receives the IP port information, and includes the undetermined node in the first cluster to form a second cluster;
and the undetermined nodes enter the first cluster, any node in the first cluster is acted, and the addition of the undetermined nodes to the first cluster is confirmed under the interaction of the undetermined nodes and any node, so that a second cluster is formed.
In addition, information synchronization is carried out among all nodes in the same cluster based on messages, and the starting signal sent by any node is synchronized with other nodes.
In some embodiments, the new node enters any of the nodes and adjusts the configuration of the first cluster; and the node receives the IP port information of the new node and constructs signal transmission with the node of the same cluster.
The self-built distributed module realizes all functions of raft distributed consistency, a user starts to realize synchronization among nodes after introducing the module and configuring ip and port information of each node, and the self-built distributed module has the functions of searching other nodes, automatically selecting a master and dynamically adding or quitting the nodes. When a certain module node is started, the node automatically sends a message to search other nodes arranged in the configuration, if a confirmation message is received, the existence of the node is confirmed to form a distributed system, in addition, a general distributed framework is constructed, other single-machine projects are rapidly expanded, and the distributed system is formed.
In some embodiments, the interfacing of the nodes is performed based on the same distributed system, and the nodes perform information synchronization.
After the distributed system is started, the states of all nodes of the distributed system, such as leader nodes and follower nodes, information about which nodes exit the cluster and the like, can be obtained through an internal function. When the follower node exits or the leader is replaced, the logic can be checked through the overview information, so that the time for manually developing and processing the logic is saved. The user can process the logic of the user according to the state information of the node.
At this time, the node a sends a message indicating whether to start up to other nodes in the configuration, if the other nodes are in the start-up state, confirmation information is returned, and after receiving the confirmation information, the node a confirms the existence of the other nodes, and can check the information of the other nodes from the node a. Similarly, since other nodes will perform the same operation, the cluster in normal operation can query the same result from which started node looks at the information of the nodes in the cluster, that is, so-called synchronization.
In addition, after the distributed system is started, the states of all nodes of the distributed system, such as leader nodes and follower nodes, information about which nodes exit the cluster, and the like, can be obtained through an internal function. When the follower node exits or leader replacement occurs, the logic can be checked through overview information, and time for manually developing and processing the logic is saved. The user can process the logic of the user according to the state information of the node.
In some embodiments, custom business logic data is conducted based on the node; and sharing the service logic data to each node in the cluster.
In this case, the user can synchronize customized variable data through a shared channel of the cluster. Some self-defined business logic data can be synchronized to each node of the cluster through this channel. At this time, the shared channel is actually shared data, for example, a variable a is added to the node a in the cluster, although the operation is only performed on the node a in the operation, after the operation is completed, the node a notifies the operation of adding the variable a to other nodes in the cluster, and after the other nodes receive a command, the other nodes also add the variable a to their own node, which is the shared data in a certain sense.
Additionally, the distributed modules may synchronize information of nodes and shared variables. However, the user needs to know that the synchronization of the distributed system information is delayed, that is, the data is given at a certain node and can be immediately acquired at other nodes. This rule needs to be known in use to avoid the creation of certain pits in development.
According to the method for quickly constructing the distributed system based on the python language, the associated information is configured in the distributed system, information is sent based on the distributed system to synchronize the information flow of each node, if a new node is added to the distributed system, the new node is added to any node to form a first cluster with information synchronization, so that the distributed system is quickly constructed based on each node, other single-machine projects are quickly expanded by a distributed framework of the distributed system, and signal synchronization is carried out based on the distributed system.
It should be understood that although the steps are described above in a certain order, the steps are not necessarily performed in the order described. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, some steps of this embodiment may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of performing the steps or stages is not necessarily sequential, but may be performed alternately or alternately with other steps or at least a part of the steps or stages in other steps.
In an embodiment, there is further provided a computer device including at least one processor, and a memory communicatively connected to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions being executable by the at least one processor to cause the at least one processor to execute the method for quickly building a distributed system based on python language, the processor executing the instructions to implement the steps in the method embodiments:
associating the node and the IP port information and forming associated information;
configuring the association information within a distributed system;
when one node is started, the distributed system sends out messages along the node, and other nodes learn about the starting of the node;
and if the new node is added into the distributed system, adding the new node into any one node to form a first cluster with synchronous information.
In one embodiment, a computer-readable storage medium is provided, which stores computer instructions for causing the computer to execute the method for quickly building a distributed system based on python language, and includes the following steps:
associating the node and the IP port information and forming associated information;
configuring the association information within a distributed system;
when one node is started, the distributed system sends a report out along the node, and other nodes learn about the starting of the node;
and if the new node is added into the distributed system, adding the new node into any one node to form a first cluster with synchronous information.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program represented by computer instructions and stored in a non-volatile computer-readable storage medium, and the computer program can include the processes of the embodiments of the methods described above when executed. Any reference to memory, storage, database or other medium used in the embodiments provided herein can include at least one of non-volatile and volatile memory.
Non-volatile memory may include read-only memory, magnetic tape, floppy disk, flash memory, optical memory, or the like. Volatile memory may include random access memory or external cache memory. By way of illustration, and not limitation, RAM may take many forms, such as static random access memory, dynamic random access memory, and the like.
To sum up, the technical scheme that this application provided has following advantage:
according to the method for quickly constructing the distributed system based on the python language, the associated information is configured in the distributed system, information is sent based on the distributed system to synchronize information flow of each node, if a new node is added to the distributed system, the new node is added to any one node to form a first cluster with information synchronization, so that the distributed system is quickly constructed based on each node, other single-machine projects can be quickly expanded by a distributed framework of the distributed system, and signal synchronization is carried out based on the distributed system.
The above description is only exemplary of the present application and should not be taken as limiting the present application, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.
Claims (4)
1. A method for quickly building a distributed system based on a python language is applied to building a universal distributed framework, and comprises the following steps:
associating the node and the IP port information and forming associated information;
configuring the association information within a distributed system;
when one node is started, the distributed system sends a report out along the node, and other nodes learn about the starting of the node;
if a new node is added into the distributed system, adding the new node into any one node to form a first cluster with synchronous information;
forming the distributed system based on mutual correspondence among a plurality of the nodes and under the creation of python language;
when each undetermined node joins the first cluster, reporting the IP port information of the undetermined node to the first cluster by the undetermined node, and transmitting the IP port information to each node in the first cluster through the first cluster;
when each node of the first cluster receives the IP port information, bringing the undetermined node into the first cluster to form a second cluster; the method comprises the steps that undetermined nodes enter a first cluster, any node in the first cluster is acted, and the addition of the undetermined nodes to the first cluster is confirmed under the interaction of the undetermined nodes and any node, so that a second cluster is formed;
and carrying out information synchronization among the nodes of the same cluster based on the messages, and synchronizing the starting signal sent by any one node with other nodes.
2. The python language based method for quickly building a distributed system as recited in claim 1, wherein the new node enters any of the nodes and adjusts the configuration of the first cluster; and the node receives the IP port information of the new node and constructs signal transmission with the node of the same cluster.
3. The method for quickly building a distributed system based on python language according to claim 2, wherein the docking of the nodes is performed based on the same distributed system, and the nodes perform information synchronization.
4. The python language based method for quickly building a distributed system as recited in claim 3, wherein custom business logic data is performed based on the nodes; and sharing the service logic data to each node in the cluster.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210574788.1A CN114666348B (en) | 2022-05-25 | 2022-05-25 | Method for quickly constructing distributed system based on python language |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210574788.1A CN114666348B (en) | 2022-05-25 | 2022-05-25 | Method for quickly constructing distributed system based on python language |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114666348A CN114666348A (en) | 2022-06-24 |
| CN114666348B true CN114666348B (en) | 2022-11-11 |
Family
ID=82038316
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210574788.1A Active CN114666348B (en) | 2022-05-25 | 2022-05-25 | Method for quickly constructing distributed system based on python language |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114666348B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102710554A (en) * | 2012-06-25 | 2012-10-03 | 深圳中兴网信科技有限公司 | Distributed message system and service status detection method thereof |
| CN106960060A (en) * | 2017-04-10 | 2017-07-18 | 聚好看科技股份有限公司 | The management method and device of a kind of data-base cluster |
| CN110392080A (en) * | 2018-04-20 | 2019-10-29 | 杭州海康威视系统技术有限公司 | A kind of dispositions method based on distributed system, device and distributed system |
| CN110661841A (en) * | 2019-08-06 | 2020-01-07 | 江阴逐日信息科技有限公司 | Data consistency method for distributed service discovery cluster in micro-service architecture |
| CN110677347A (en) * | 2019-08-19 | 2020-01-10 | 荣邦科技有限公司 | Method for registering and discovering services of micro-services |
| CN111209012A (en) * | 2019-12-31 | 2020-05-29 | 中国电力科学研究院有限公司 | Linux system-oriented software agent end automatic deployment method |
| CN112328685A (en) * | 2020-11-05 | 2021-02-05 | 浪潮云信息技术股份公司 | Full-peer distributed database data synchronization method |
| CN113596168A (en) * | 2021-08-02 | 2021-11-02 | 中国民生银行股份有限公司 | Block chain alliance chain-based verification method and device |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10430240B2 (en) * | 2015-10-13 | 2019-10-01 | Palantir Technologies Inc. | Fault-tolerant and highly-available configuration of distributed services |
| CN105515759B (en) * | 2015-11-27 | 2018-11-09 | 国网信息通信产业集团有限公司 | A kind of micro services register method and system |
-
2022
- 2022-05-25 CN CN202210574788.1A patent/CN114666348B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102710554A (en) * | 2012-06-25 | 2012-10-03 | 深圳中兴网信科技有限公司 | Distributed message system and service status detection method thereof |
| CN106960060A (en) * | 2017-04-10 | 2017-07-18 | 聚好看科技股份有限公司 | The management method and device of a kind of data-base cluster |
| CN110392080A (en) * | 2018-04-20 | 2019-10-29 | 杭州海康威视系统技术有限公司 | A kind of dispositions method based on distributed system, device and distributed system |
| CN110661841A (en) * | 2019-08-06 | 2020-01-07 | 江阴逐日信息科技有限公司 | Data consistency method for distributed service discovery cluster in micro-service architecture |
| CN110677347A (en) * | 2019-08-19 | 2020-01-10 | 荣邦科技有限公司 | Method for registering and discovering services of micro-services |
| CN111209012A (en) * | 2019-12-31 | 2020-05-29 | 中国电力科学研究院有限公司 | Linux system-oriented software agent end automatic deployment method |
| CN112328685A (en) * | 2020-11-05 | 2021-02-05 | 浪潮云信息技术股份公司 | Full-peer distributed database data synchronization method |
| CN113596168A (en) * | 2021-08-02 | 2021-11-02 | 中国民生银行股份有限公司 | Block chain alliance chain-based verification method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114666348A (en) | 2022-06-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11736371B2 (en) | Heartbeat information sending method and apparatus, and heartbeat sending node | |
| Govorskii et al. | Mathematical modeling of inhomogeneous traffic in a heterogeneous integrated corporate data control system | |
| KR101871383B1 (en) | Method and system for using a recursive event listener on a node in hierarchical data structure | |
| EP4072097A1 (en) | Method and device for dynamically adding consensus node in blockchain | |
| Hong | A distributed, asynchronous and incremental algorithm for nonconvex optimization: An admm based approach | |
| US9424260B2 (en) | Techniques for data assignment from an external distributed file system to a database management system | |
| CN114238495A (en) | Method and device for switching main cluster and standby cluster of database, computer equipment and storage medium | |
| CN117950850A (en) | Data transmission method, device, electronic equipment and computer readable medium | |
| CN111258840A (en) | Cluster node management method and device and cluster | |
| CN114666348B (en) | Method for quickly constructing distributed system based on python language | |
| KR102031589B1 (en) | Methods and systems for processing relationship chains, and storage media | |
| CN111183622B (en) | Block chain system, information sharing method and related equipment | |
| CN107729421A (en) | The execution method, apparatus and storage medium of storing process | |
| JP2023065558A (en) | Method of responding to operation, apparatus of responding to operation, electronic device, and storage medium | |
| CN113691403B (en) | Topology node configuration method, related device and computer program product | |
| CN113965538B (en) | Equipment state message processing method, device and storage medium | |
| EP3435626B1 (en) | Table connection method in distributed database system, and distributed database system | |
| CN115277707A (en) | Service processing method, device, electronic equipment and storage medium | |
| CN115242839A (en) | Internet of things cascading data transmission method and device, electronic equipment and storage medium | |
| CN111858956B (en) | Knowledge graph construction method, knowledge graph construction device, network equipment and storage medium | |
| CN114722398A (en) | Intelligent contract anomaly detection method, block chain network, equipment and storage medium | |
| CN112541038A (en) | Time series data management method, system, computing device and storage medium | |
| CN111274042A (en) | FPGA heterogeneous acceleration system, synchronous scheduling method thereof and related equipment | |
| US20240264960A1 (en) | Managing a workspace mesh | |
| CN115168366B (en) | Data processing method, data processing device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |