CN115269615A

CN115269615A - Block chain consensus algorithm implementation method and device based on streaming processing

Info

Publication number: CN115269615A
Application number: CN202210641627.XA
Authority: CN
Inventors: 王晓亮; 宁志伟
Original assignee: Beijing Xita Technology Co ltd
Current assignee: Beijing Xita Technology Co ltd
Priority date: 2022-06-07
Filing date: 2022-06-07
Publication date: 2022-11-01

Abstract

The embodiment of the specification discloses a block chain consensus algorithm implementation method and device based on streaming processing, wherein in the method, a consensus module for executing the block chain consensus algorithm comprises an external event, a consensus engine, middleware and a data source; corresponding to the implementation of the consensus algorithm, comprising: receiving the external event and distributing the external event to the consensus engine according to a message type, the consensus engine comprising a processing unit corresponding to the message type; the consensus engine reads the latest data provided by the data source in real time and submits the external event to the processing unit corresponding to the consensus engine for calculation based on the latest data and the corresponding message type; and calling the middleware to write the processing data into the data source in real time. The invention introduces stream processing, splits the realization of the consensus module into corresponding functional modules, and has the characteristics of strong universality, strong expansibility, high performance, strong maintainability and the like.

Description

Block chain consensus algorithm implementation method and device based on streaming processing

Technical Field

The present disclosure relates to the field of computer software technologies, and in particular, to a method and an apparatus for implementing a blockchain consensus algorithm based on streaming processing, an electronic device, and a storage medium.

Background

The consensus algorithm is a series of processes and rules generated for implementing a distributed coherency protocol, and commonly used consensus algorithms include conventional block chain consensus algorithms such as PoW (Proof of Work), poS (Proof of stamp), and the like, and also include a terminator consensus engine framework. The consensus module serves as a core module of the blockchain system and plays a crucial role in the blockchain system. However, due to the objective existence of high implementation complexity of the consensus algorithm, the code amount of the consensus algorithm is updated and iterated; meanwhile, since the data of the consensus module during operation is stored in the disk of its operating environment, its performance is limited by the disk IO. How to carry out reasonable frame design on the realization process of the consensus algorithm, enhance the flexibility and the universality of the consensus module, and improve the efficiency and the expandability of the consensus calculation is a technical problem to be solved urgently.

Disclosure of Invention

An object of the embodiments of the present specification is to provide a method, an apparatus, an electronic device, and a storage medium for implementing a blockchain consensus algorithm based on streaming processing.

In order to solve the above technical problem, the embodiments of the present specification are implemented as follows:

in a first aspect, a method for implementing a blockchain consensus algorithm based on streaming processing is provided, where a consensus module for executing the blockchain consensus algorithm includes an external event, a consensus engine, middleware, and a data source; corresponding to the implementation of the consensus algorithm, comprising:

receiving the external event and distributing the external event to the consensus engine according to a message type, the consensus engine comprising a processing unit corresponding to the message type;

the consensus engine reads the latest data provided by the data source in real time and submits the external event to the processing unit corresponding to the consensus engine for calculation based on the latest data and the corresponding message type;

and calling the middleware to write the processing data into the data source in real time.

In a second aspect, a block chain consensus algorithm implementation device based on streaming processing is provided, in which a consensus module for executing a block chain consensus algorithm includes an external event, a consensus engine, middleware, and a data source; corresponding to the implementation of consensus algorithms, comprising:

an external event receiving module for receiving the external event and distributing the external event to the consensus engine according to a message type, wherein the consensus engine comprises a processing unit corresponding to the message type;

a consensus engine calculation module, configured to read, by the consensus engine, the latest data provided by the data source, and submit the external event to the consensus engine for calculation corresponding to the processing unit based on the latest data and the corresponding message type;

and the data updating module is used for calling the middleware to write the processing data into the data source in real time.

In a third aspect, an electronic device is provided, including: a processor; and

a memory arranged to store computer executable instructions that, when executed, cause the processor to perform the method of the first aspect.

In a fourth aspect, a computer-readable storage medium is presented, storing one or more programs which, when executed by an electronic device comprising a plurality of application programs, cause the electronic device to perform the method of the first aspect.

The specification can achieve at least the following technical effects:

the scheme of the invention introduces stream processing, and splits the realization of the consensus module into corresponding functional modules such as an external event, a consensus engine, a middleware, a data source and the like, the consensus engine can realize the substitution of the consensus engine only by partially adapting, and the corresponding data reading and writing are processed by the middleware and the data source, so that the invention has the characteristics of strong universality, strong expansibility, high performance, strong maintainability and the like.

Drawings

In order to more clearly illustrate the embodiments of the present specification or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only some embodiments described in the present specification, and for those skilled in the art, other drawings can be obtained according to the drawings without any creative effort.

Fig. 1 is a block chain consensus algorithm implementation method framework diagram based on streaming processing according to an embodiment of the present disclosure.

Fig. 2 is a schematic diagram of a block chain consensus algorithm based on streaming processing according to an embodiment of the present disclosure.

Fig. 3 is a second schematic diagram of a block chain consensus algorithm based on streaming processing according to an embodiment of the present disclosure.

Fig. 4 is a third schematic diagram of an implementation method of a block chain consensus algorithm based on streaming processing according to an embodiment of the present disclosure.

Fig. 5 is a schematic diagram of an apparatus for implementing a blockchain consensus algorithm based on streaming processing according to an embodiment of the present disclosure.

Fig. 6 is a second schematic diagram of an implementation apparatus of a block chain consensus algorithm based on streaming processing according to an embodiment of the present disclosure.

Fig. 7 is a schematic structural diagram of an electronic device provided in an embodiment of the present specification.

Detailed Description

In order to make those skilled in the art better understand the technical solutions in the present specification, the technical solutions in the embodiments of the present specification will be clearly and completely described below with reference to the drawings in the embodiments of the present specification, and it is obvious that the described embodiments are only a part of the embodiments of the present specification, and not all of the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present specification without making any creative effort shall fall within the protection scope of the present specification.

The following describes a block chain consensus algorithm implementation scheme based on streaming processing, which is described in detail by specific examples.

Key terms

Block chain consensus: in the blockchain, nodes communicate with each other through data communication, and any node can be added into the blockchain system as long as the data format conforms to a certain blockchain convention. That is, each node can use an arbitrarily written program to attack the blockchain to gain benefit. In order to ensure the operation safety of the blockchain system, the blockchain nodes must be made as decentralized as possible, because in this case, a node is overturned by its accounting node if it is checked for false accounts, and therefore, the essence of the blockchain consensus mechanism is to make the blockchain nodes as random and decentralized as possible. Common blockchain consensus mechanisms include: (1) PoW (proof of workload); (2) PoS (entitlement certification) (3) DPoS (delegation entitlement certification); (4) PBFT (practical byzantine fault-tolerant algorithm); (5) POL (validation pool), etc.; in addition, there are POW + POS and POP mechanisms.

Flow processing: the method is a process for continuously processing, aggregating and analyzing unbounded data sets. Streaming assumes that the potential value of the data is the freshness of the data, requiring processing to get the results as soon as possible. In this manner, data arrives in a streaming manner. During the continuous arrival of data, only a small portion of the streaming data is stored in a limited memory, since the stream carries a large amount of data. Streaming is used for online applications and typically operates on the order of seconds or milliseconds. The current mainstream stream processing components include: from, spark Streaming, kafKa, flume, flink, S3, local file, etc. The streaming processing is gradually becoming a mainstream tool of big data processing technology due to its capability of processing mass data rapidly and efficiently, and the following are some common application scenarios: algorithm trading and stock market monitoring; intrusion, surveillance and fraud detection; a smart grid; a geo-fence; vehicle and wildlife tracking; and (5) traffic monitoring. In general, stream processing is very useful for scenarios where we detect problems and effectively improve the timeliness of the results. Furthermore, stream processing plays a key role in data-driven companies.

Enddermint: is an open source project of Terdermint company and is a variant of PBFT algorithm, the relationship between Terdermint and PBFT is similar to that of Raft and Paxos, and Terdermint is a simplified version of PBFT. The Terdermint algorithm randomly selects some nodes as validators verification nodes, and then selects one of the validators as a promoter node. The promoter node starts to monitor and collect all transactions of the whole network, and after a few minutes, a new block is assembled and broadcasted to the whole network, and the new block is the promoter block. After all validator nodes in the whole network receive the proxy block, all transactions in the block are read, verification is carried out one by one, if no problem exists, a pre-vote voting message is sent out to indicate that the block is approved, a positive vote is cast, if illegal transactions in the block are found, an objection vote is cast, the voting messages are broadcasted to all validator nodes, so each validator node sends out a voting message and collects the voting messages of others, and when the collected number of the approved votes exceeds 2/3, a pre-commit voting message is sent out, which is the voting of the second phase, and each node also monitors and collects the pre-commit voting message. When the pre-commit agreement ticket number collected by one validator node exceeds 2/3, the block is agreed by most people, the block can be written into a local block chain with great care, and the commit is completed after the block is added to the tail. Meanwhile, the block height is increased by 1, the node index of the promoter is also increased by 1, and a new block is proposed after the next round is entered.

Example one

Fig. 1 is a block chain consensus algorithm implementation method framework diagram based on streaming processing according to an embodiment of the present invention. As shown in fig. 1, the overall process of the scheme of the present invention can be described as follows: (1) The external event driving is that external events are detected in a circulating mode, and corresponding events are distributed to a method corresponding to the consensus engine for decision making; (2) Event decision, namely judging by the consensus engine according to the real-time updated data source as a state variable, writing the data source into the message middleware, and driving the data source to update in real time; (3) And returning the decision result, namely returning the decision result to the corresponding client. It should be noted that, the external event here refers to a read-write data event from the outside in the consensus module, which drives the operation of the consensus module; the middleware is middleware used in streaming processing, such as Kafka, S3, local file, etc.; the data source is a data source in data streaming processing, is used for storing block chain consensus module data and provides read-write capability; the consensus engine is an engine which is realized according to a consensus algorithm, receives and transmits data and processes the data, but does not contain data storage. Meanwhile, the data source processing also needs a structured query language; and a consensus node Leader in the consensus algorithm, which is responsible for processing external events.

For this purpose, fig. 2 shows a block chain consensus algorithm implementation method based on streaming processing according to the present invention. The consensus module for executing the blockchain consensus algorithm comprises an external event, a consensus engine, middleware and a data source; corresponding to the implementation of the consensus algorithm, comprising:

step 101: receiving the external event, and distributing the external event to the consensus engine according to a message type, the consensus engine comprising a processing unit corresponding to the message type.

Optionally, the block chain consensus algorithm at least comprises a terminermint consensus algorithm. The embodiment will also be described by taking the endrmintint consensus algorithm as an example, but it should be understood that any block chain consensus algorithm that can be implemented by using a streaming processing method is within the scope of the present invention.

Optionally, the message type includes at least one of Noop, propofol, prevote, presmit, newView, leaderpivote, or leaderpommute. The external event is an abstract summary of the event processing message, but it should be understood that any read-write data event from outside the consensus module, which can drive the operation of the consensus module, is within the scope of the present invention.

Step 102: and the consensus engine reads the latest data provided by the data source in real time and submits the external event to the processing unit corresponding to the consensus engine for calculation based on the latest data and the corresponding message type.

Optionally, as shown in fig. 3, a specific process of distributing the external event to the consensus engine according to the message type includes:

step 301: and updating the parameter configuration of the consensus module according to the latest data provided by the data source.

Step 302: checking the validity of the block according to the message type of the external event.

Step 103: and calling the middleware to write the processing data into the data source in real time.

In one embodiment of the invention, a Terdermint consensus algorithm is selected, local file is used as a middleware, material is used as a data source for streaming processing, and SQL is used for reading the data source. First, a proto file as above is defined for a message from a network to accept an external event and to be handed over to a consensus engine for further processing according to a message type, and technical details are as follows.

Secondly, defining data source such as trail, managing the life cycle of the data source, providing a write _ json method in a local file mode, writing data into the data source, and providing real-time latest data by utilizing the capability of the data source for updating the data in real time, wherein the technical details are as follows.

If other consensus algorithms are adopted to realize the consensus function, events from the network need to be processed according to proto files, and data reading and writing and data source life cycle management are realized according to the trail.

As shown in fig. 4, in another embodiment, a block chain consensus algorithm implementation method based on streaming processing is provided, further including:

step 104: and when the node consensus voting number exceeds the set proportion of the total number of the verification nodes, updating the node state, and calling the middleware to write the node state into the data source after broadcasting to the block link point.

The corresponding processing procedure can also be specifically designed for different message types. In an embodiment of the present invention, a Proposal type is taken as an example, and a streaming processing process of the specific work is as follows: deserializing a byte array of the received message into a program-readable structure, and inquiring whether the message is received by a data source through parameters of the structure and SQL; querying a data source through SQL to determine whether a sender of the message is a Leader or not and whether a block corresponding to the message exists or not; if the first two steps of verification pass, the message is written into a data source through the middleware, and the data can be updated in real time so as to be convenient for other queries to use; querying the data source through SQL to determine whether the block state is normal, and writing the block state into the data source through middleware; writing the corresponding relation between the block and the block state into a data source through the middleware; if the voting number exceeds 2/3 of the total number of the verification nodes, clearing data, entering the next step, and otherwise, ending the processing; broadcasting information, changing the current node state, and writing the information into a data source through middleware; and returning a processing result.

Example two

Fig. 5 is a schematic structural diagram of an apparatus 500 for implementing a blockchain consensus algorithm based on streaming processing according to an embodiment of the present disclosure. Referring to fig. 5, in an embodiment, an apparatus for implementing a blockchain consensus algorithm based on streaming processing, a consensus module for performing the blockchain consensus algorithm includes an external event, a consensus engine, middleware, and a data source; corresponding to the implementation of the consensus algorithm, comprising:

an external event receiving module 501, configured to receive the external event and distribute the external event to the consensus engine according to a message type, where the consensus engine includes a processing unit corresponding to the message type;

a consensus engine calculation module 502, configured to read the latest data provided by the data source by the consensus engine, and submit the external event to the consensus engine corresponding to the processing unit for calculation based on the latest data and the corresponding message type;

and a data updating module 503, configured to invoke the middleware to write the processing data into the data source in real time.

As shown in fig. 6, in another embodiment, an apparatus 500 for implementing a blockchain consensus algorithm based on streaming processing is provided, further including:

a consensus feedback module 504, configured to update the node status when the number of node consensus votes exceeds a set ratio of the total number of verification nodes, and call the middleware to write the node status into the data source after broadcasting to the block link point.

It should be understood that the apparatus for implementing a blockchain consensus algorithm based on streaming processing according to the embodiment of the present specification may further perform the method performed by the apparatus (or device) for implementing a blockchain consensus algorithm based on streaming processing in fig. 1 to 4, and implement the functions of the apparatus (or device) for implementing a blockchain consensus algorithm based on streaming processing in the examples shown in fig. 1 to 4, which are not described herein again.

EXAMPLE III

Fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present specification. Referring to fig. 7, at a hardware level, the electronic device includes a processor, and optionally further includes an internal bus, a network interface, and a memory. The Memory may include a Memory, such as a Random-Access Memory (RAM), and may further include a non-volatile Memory, such as at least 1 disk Memory. Of course, the electronic device may also include hardware required for other services.

The processor, the network interface, and the memory may be connected to each other via an internal bus, which may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one double-headed arrow is shown in FIG. 7, but this does not indicate only one bus or one type of bus.

And the memory is used for storing programs. In particular, the program may include program code comprising computer operating instructions. The memory may include both memory and non-volatile storage and provides instructions and data to the processor.

The processor reads the corresponding computer program from the nonvolatile memory into the memory and then runs the computer program to form the shared resource access control device on the logic level. The processor is used for executing the program stored in the memory and is specifically used for executing the following operations:

the consensus module for executing the blockchain consensus algorithm comprises an external event, a consensus engine, middleware and a data source; corresponding to the implementation of consensus algorithms, comprising:

The block chain consensus algorithm implementation method based on streaming processing disclosed in the embodiments of fig. 1 to 4 of the present specification can be applied to or implemented by a processor. The processor may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in a processor or instructions in the form of software. The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), application Specific Integrated Circuits (ASICs), field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components. The various methods, steps and logic blocks disclosed in the embodiments of the present specification may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the embodiments of the present specification may be embodied directly in a hardware decoding processor, or in a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in a memory, and a processor reads information in the memory and completes the steps of the method in combination with hardware of the processor.

Of course, besides the software implementation, the electronic device of the embodiment of the present disclosure does not exclude other implementations, such as a logic device or a combination of software and hardware, and the like, that is, the execution subject of the following processing flow is not limited to each logic unit, and may also be hardware or a logic device.

Example four

Embodiments of the present specification also propose a computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a portable electronic device comprising a plurality of application programs, are capable of causing the portable electronic device to perform the method of the embodiments shown in fig. 1 to 4, and in particular to perform the method of:

the consensus module for executing the blockchain consensus algorithm comprises an external event, a consensus engine, middleware and a data source; corresponding to the implementation of the consensus algorithm, comprising:

In short, the above description is only a preferred embodiment of the present disclosure, and is not intended to limit the scope of the present disclosure. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present specification shall be included in the protection scope of the present specification.

The systems, devices, modules or units illustrated in the above embodiments may be implemented by a computer chip or an entity, or by a product with certain functions. One typical implementation device is a computer. In particular, the computer may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising a … …" does not exclude the presence of another identical element in a process, method, article, or apparatus that comprises the element.

The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.

Claims

1. A block chain consensus algorithm implementation method based on streaming processing is characterized in that a consensus module for executing the block chain consensus algorithm comprises an external event, a consensus engine, middleware and a data source; corresponding to the implementation of the consensus algorithm, comprising:

2. The method of claim 1, wherein the blockchain consensus algorithm comprises at least a terminrmindermint consensus algorithm.

3. The method of claim 1, wherein the message type comprises at least one of Noop, propofol, prevote, precommit, newView, leader Precommit, or leader Precommit.

4. The method of claim 1, wherein distributing the external event to the consensus engine according to the message type comprises: updating the parameter configuration of the consensus module according to the latest data provided by the data source; and checking the validity of the block according to the message type of the external event.

5. The method of claim 1, further comprising updating the node status when the number of node consensus votes exceeds a set percentage of a total number of verified nodes, and invoking the middleware to write the node status to the data source after broadcasting to a block link node.

6. A block chain consensus algorithm implementation device based on streaming processing is characterized in that a consensus module for executing a block chain consensus algorithm comprises an external event, a consensus engine, middleware and a data source; corresponding to the implementation of the consensus algorithm, comprising:

7. The apparatus of claim 6, in which the blockchain consensus algorithm comprises at least a Terdermint consensus algorithm.

8. The apparatus of claim 6, wherein the message type comprises at least one of Noop, proposal, prevote, precommit, newView, leaderPrevote, or LeaderPrecommit.

9. The apparatus of claim 6, wherein distributing the external event to the consensus engine according to the message type comprises: updating the parameter configuration of the consensus module according to the latest data provided by the data source; and checking the validity of the block according to the message type of the external event.

10. The apparatus of claim 6, further comprising a consensus feedback module for updating the node status when the number of node consensus votes exceeds a set percentage of the total number of verified nodes, and invoking the middleware to write the node status to the data source after broadcasting to a block link point.

11. An electronic device, comprising:

a processor; and

a memory arranged to store computer executable instructions that, when executed, cause the processor to perform the method of any of claims 1 to 5.

12. A computer readable storage medium, characterized in that the computer readable storage medium stores one or more programs that, when executed by an electronic device comprising a plurality of application programs, cause the electronic device to perform the method of any of claims 1 to 5.