CN117235122A

CN117235122A - Data processing method, apparatus, device, readable storage medium, and program product

Info

Publication number: CN117235122A
Application number: CN202210631400.7A
Authority: CN
Inventors: 陈永清
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2022-06-06
Filing date: 2022-06-06
Publication date: 2023-12-15

Abstract

The embodiment of the application provides a data processing method, a device, equipment, a readable storage medium and a program product, which relate to the field of blockchain, and application scenes comprise but are not limited to a scene of reading and writing a cache of a blockchain. The method comprises the following steps: acquiring a target key of a target block or a block chain; writing a plurality of key value pairs in a target block into a block cache of a memory of a block chain; or reading a target value corresponding to the target key from the block cache; the block cache is a first-level hash table, the first-level hash table comprises a second-level hash table corresponding to the block height of each block in at least two blocks of a block chain and the block height of each block, the second-level hash table corresponding to the block height of each block in at least two blocks of the block chain and the block height of each block form a key value pair, the second-level hash table comprises a plurality of third-level hash tables, and each key value pair in all key value pairs of each block is stored in any one of the third-level hash tables.

Description

Data processing method, apparatus, device, readable storage medium, and program product

Technical Field

The present application relates to the field of computer technology, and in particular, to a data processing method, apparatus, device, readable storage medium, and program product.

Background

Blockchains are novel application modes of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanisms, encryption algorithms, and the like. The block of the block chain comprises a plurality of KV (Key-Value) pairs, and the prior art directly writes the KV in the block into a cache of the block chain; for example, there are 10 blocks in a blockchain, each block has 1 ten thousand transactions, and there are 100 KV in each transaction, then the 10 blocks have 10≡7 KV in total, and since 10≡6 KV is written to the cache for one block at a time, only 10≡6 KV can be written to the cache in series, so that the update time of the cache is as high as O (n≡3), the lock holding time is long, and the efficiency of writing one block to the cache at a time is low. The KV reading from the blockchain cache is inefficient due to the long time to hold the lock.

Disclosure of Invention

The application provides a data processing method, a device, equipment, a computer readable storage medium and a computer program product, which aim at the defects of the prior mode and are used for solving the problem of how to improve the cache efficiency of a read-write blockchain.

Acquiring a target key of a target block or a block chain;

writing a plurality of key value pairs in a target block into a block cache of a memory of a block chain; or reading a target value corresponding to the target key from the block cache;

wherein the target key and the target value form a key value pair; the block cache is a first-level hash table, the first-level hash table comprises a second-level hash table corresponding to the block height of each block in at least two blocks of a block chain and the block height of each block, the second-level hash table corresponding to the block height of each block in at least two blocks of the block chain and the block height of each block form key value pairs, the second-level hash table comprises a plurality of third-level hash tables, and each key value pair in all key value pairs of each block is stored in any one of the third-level hash tables.

In one embodiment, writing a plurality of key-value pairs in a target block to a block cache of memory of a blockchain includes:

acquiring a new second-level hash table;

writing a plurality of key value pairs in the target block into a new second-level hash table;

the new second level hash table is written to the blockcache of the blockchain memory.

In one embodiment, obtaining a new second level hash table includes:

Based on the object pool in the memory of the block chain, a new second-level hash table is obtained.

In one embodiment, writing a plurality of key-value pairs in a target block to a new second-level hash table includes:

writing a plurality of key value pairs in the target block into a plurality of third-level hash tables corresponding to the new second-level hash tables through a hash function; each of the plurality of key-value pairs is stored within any one of a plurality of third-level hash tables corresponding to the new second-level hash table.

In one embodiment, writing the new second level hash table to a blockcache of a blockchain memory includes:

locking the first-stage hash table, and writing the new second-stage hash table and the block height of the target block into the first-stage hash table; the new second-level hash table and the block height of the target block form a key value pair;

the first level hash table is released.

In one embodiment, writing the new second level hash table and the block height of the target block into the first level hash table comprises:

and deleting the second-level hash table corresponding to the minimum block height in the block heights of at least two blocks, and writing the new second-level hash table and the block height of the target block into the first-level hash table.

In one embodiment, reading a target value corresponding to a target key from a tile cache includes:

locking the block cache, and sequencing the block heights of at least two blocks to obtain an ordered number group, wherein the ordered number group comprises key value pairs of each block in the at least two blocks, and the block heights of the at least two blocks are sequenced from large to small in the ordered number group;

based on the ordered array, reading a target value corresponding to the target key from the block cache;

and releasing the lock on the block cache.

In one embodiment, reading a target value corresponding to a target key from a tile cache based on an ordered array, comprises:

determining a first block height corresponding to the target key based on the ordered number group;

determining a second-level hash table corresponding to the height of the first block;

and reading a target value corresponding to the target key from the second-level hash table corresponding to the first block height.

In one embodiment, determining a first tile height corresponding to the target key based on the ordered set comprises:

determining at least two keys identical to the target key from the ordered array;

and determining the block height which is sequenced to be the first block height corresponding to the target key in the block heights which are respectively corresponding to the at least two keys.

In a second aspect, the present application provides a data processing apparatus comprising:

the first processing module is used for acquiring a target key of a target block or a block chain;

the second processing module is used for writing a plurality of key value pairs in the target block into the block cache of the memory of the blockchain; or reading a target value corresponding to the target key from the block cache;

In a third aspect, the present application provides an electronic device, comprising: a processor, a memory, and a bus;

a bus for connecting the processor and the memory;

a memory for storing operation instructions;

And a processor for executing the data processing method according to the first aspect of the present application by calling an operation instruction.

In a fourth aspect, the present application provides a computer readable storage medium storing a computer program for executing the data processing method of the first aspect of the present application.

In a fifth aspect, the present application provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the data processing method of the first aspect of the application.

The technical scheme provided by the embodiment of the application has at least the following beneficial effects:

acquiring a target key of a target block or a block chain; writing a plurality of key value pairs in a target block into a block cache of a memory of a block chain; or reading a target value corresponding to the target key from the block cache; wherein the target key and the target value form a key value pair; the block cache is a first-level hash table, the first-level hash table comprises a second-level hash table corresponding to the block height of each block in at least two blocks of a block chain and the block height of each block, the second-level hash table corresponding to the block height of each block in at least two blocks of the block chain and the block height of each block form key value pairs, the second-level hash table comprises a plurality of third-level hash tables, and each key value pair in all key value pairs of each block is stored in any one of the third-level hash tables; therefore, based on the cache multi-level architecture, namely the first-level hash table, the second-level hash table and the third-level hash table, the write cache or the read cache is carried out, a plurality of key value pairs in the target block can be quickly written into the cache, or target values corresponding to the target keys can be quickly read from the cache, and the preemption of locks is reduced, namely the holding time of the locks is reduced, so that the efficiency of writing the cache and the efficiency of reading the cache are improved.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings that are required to be used in the description of the embodiments of the present application will be briefly described below.

FIG. 1 is a schematic diagram of an alternative architecture of a distributed system for use in a blockchain system in accordance with embodiments of the present application;

FIG. 2 is a schematic diagram of an alternative block structure according to an embodiment of the present application;

FIG. 3 is a schematic diagram of a data processing system according to an embodiment of the present application;

FIG. 4 is a schematic flow chart of a data processing method according to an embodiment of the present application;

FIG. 5 is a schematic diagram of data processing according to an embodiment of the present application;

FIG. 6 is a flowchart illustrating another data processing method according to an embodiment of the present application;

FIG. 7 is a schematic diagram of data processing according to an embodiment of the present application;

FIG. 8 is a schematic diagram of a data processing apparatus according to an embodiment of the present application;

fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

Embodiments of the present application are described below with reference to the drawings in the present application. It should be understood that the embodiments described below with reference to the drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application, and the technical solutions of the embodiments of the present application are not limited.

As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless expressly stated otherwise, as understood by those skilled in the art. It will be further understood that the terms "comprises" and "comprising," when used in this specification, specify the presence of stated features, information, data, steps, operations, elements, and/or components, but do not preclude the presence or addition of other features, information, data, steps, operations, elements, components, and/or groups thereof, all of which may be included in the present specification. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may be present. Further, "connected" or "coupled" as used herein may include wirelessly connected or wirelessly coupled. The term "and/or" as used herein indicates at least one of the items defined by the term, e.g. "a and/or B" indicates implementation as "a", or as "B", or as "a and B".

It will be appreciated that in the specific embodiments of the present application, where data processing related data is involved, user approval or consent is required when the above embodiments of the present application are applied to specific products or technologies, and the collection, use and processing of the related data is required to comply with relevant laws and regulations and standards of the relevant country and region.

For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the embodiments of the present application will be described in further detail with reference to the accompanying drawings.

The embodiment of the application provides a data processing method provided by a data processing system, and relates to the fields of block chain technology, storage, cloud technology and the like. For example, the cache of the read-write blockchain and the like in the embodiment of the application can be performed through blockchain technology.

The system according to the embodiment of the present application may be a distributed system formed by connecting a client and a plurality of nodes (any form of computing device in an access network, such as a server and a user terminal) through a network communication.

Taking a distributed system as an example of a blockchain system, referring To fig. 1, fig. 1 is a schematic diagram of an alternative architecture of a distributed system 100 applied To a blockchain system according To an embodiment of the present application, where the architecture is formed by a plurality of nodes (arbitrary computing devices in an access network, such as servers and user terminals) and clients, and a Peer-To-Peer (P2P, peer To Peer) network is formed between the nodes, where the P2P protocol is an application layer protocol running on top of a transmission control protocol (TCP, transmission Control Protocol) protocol. In a distributed system, any machine, such as a server, a terminal, may join to become a node, including a hardware layer, an intermediate layer, an operating system layer, and an application layer.

Referring to the functionality of each node in the blockchain system shown in fig. 1, the functions involved include:

1) The routing, the node has basic functions for supporting communication between nodes.

Besides the routing function, the node can also have the following functions:

2) The application is used for being deployed in a block chain to realize specific service according to actual service requirements, recording data related to the realization function to form recorded data, carrying a digital signature in the recorded data to represent the source of task data, sending the recorded data to other nodes in the block chain system, and adding the recorded data into a temporary block when the source and the integrity of the recorded data are verified by the other nodes.

For example, the services implemented by the application include:

2.1 Wallet for providing electronic money transactions, including initiating a transaction (i.e., sending a transaction record of the current transaction to other nodes in the blockchain system, the other nodes, after verification, storing record data of the transaction in a temporary block of the blockchain in response to acknowledging that the transaction is valid; of course, the wallet also supports inquiry of remaining electronic money in the electronic money address;

2.2 The shared account book is used for providing the functions of storing, inquiring, modifying and the like of account data, sending record data of the operation on the account data to other nodes in the blockchain system, and after the other nodes verify to be effective, storing the record data into a temporary block as a response for acknowledging that the account data is effective, and also sending confirmation to the node initiating the operation.

2.3 A computerized agreement that can execute the terms of a contract, implemented by code deployed on a shared ledger for execution when certain conditions are met, for completing automated transactions based on actual business demand codes, such as querying the physical distribution status of the goods purchased by the buyer, transferring the electronic money of the buyer to the merchant's address after the buyer signs for the goods; of course, the smart contract is not limited to executing the contract for the transaction, and may execute a contract that processes the received information.

3) The blockchain comprises a series of blocks (blocks) which are connected with each other according to the generated sequence time, the new blocks are not removed once being added into the blockchain, and record data submitted by nodes in the blockchain system are recorded in the blocks.

Referring to fig. 2, fig. 2 is an optional Block Structure (Block Structure) provided in an embodiment of the present application, where each Block includes a hash value of a transaction record stored in the Block (hash value of the Block) and a hash value of a previous Block, and each Block is connected by the hash value to form a Block chain. In addition, the block may include information such as a time stamp at the time of block generation. The Blockchain (Blockchain), which is essentially a de-centralized database, is a string of data blocks that are generated in association using cryptographic methods, each of which contains associated information that is used to verify the validity (anti-counterfeiting) of its information and to generate the next block.

In order to better understand and describe the schemes of the embodiments of the present application, some technical terms related to the embodiments of the present application are briefly described below.

Go language: the Go language is a statically strong type and compiled language; the grammar of the Go language is similar to C, and the grammar functions of the Go language comprise memory security, GC (garbage collection), structural morphology, CSP-style concurrent computation and the like.

Hash table: a Hash table (Hash table) is also called a Hash table, and is a data structure that is directly accessed according to a Key Value (Key Value); that is, the hash table accesses records by mapping key values to a location in the table to speed up the lookup; this mapping function is called a hash function and the array in which the records are stored is called a hash table.

Object pool sync.pool: a basic package in the Go language is used for caching objects in some Go languages, and the objects can be multiplexed so as to reduce the creation of the objects, thereby reducing the cost consumption of GC (garbage collection) and improving the performance.

Hash function: after hash function operation, one string is converted into another string or integer data.

Time complexity: the time complexity of the algorithm is a function, and the time complexity qualitatively describes the running time of the algorithm; this is a function of the length of the string representing the value of the algorithm input; the time complexity is usually expressed by a large O symbol, and the low-order term and the leader coefficient of the function are not included; using this approach, the time complexity can be said to be asymptotic, i.e., consider the case when the input value size approaches infinity; for example, O (N≡3) represents N times N.

Leveldb: leveldb is a very efficient KV database, version 1.2 of Leveldb is able to support billions (billion) of data volumes, and at this level of quantity has very high performance.

WAL: WAL (Write Ahead Log) is a common means in database systems to guarantee the atomicity and durability of data operations; the WAL in Leveldb is to prevent the data from being lost due to downtime after the data is written into the memory, i.e., the data must be written into the disk before the data is written into the memory.

MVCC: MVCC (Multi-Version Concurrency Control ) is a method of concurrency control, typically in a database management system, to achieve concurrent access to a database; realizing transaction memory in a programming language; the MVCC maintains multiple versions of a data such that read and write operations do not conflict.

The scheme provided by the embodiment of the application relates to a block chain technology and a cloud technology, and the technical scheme of the application is described in detail by a specific embodiment. The following embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.

In order to better understand the scheme provided by the embodiment of the present application, the scheme is described below in connection with a specific application scenario.

In one embodiment, fig. 3 is a schematic diagram of a data processing system to which the embodiment of the present application is applied, and it can be understood that the data processing method provided by the embodiment of the present application may be applied, but is not limited to, to the application scenario shown in fig. 3.

In this example, as shown in FIG. 3, the architecture of the data processing system in this example may include, but is not limited to, a client 10, a server 20, a network 30, and a database 40. The user terminal 10 and the server terminal 20 can interact through the network 30, and the user terminal 10 and the database 40 can interact through the network 30; the client 10 may be a terminal, a server, etc., the server 20 may be a server, and a blockchain exists in the server 20. The client 10 sends the target block to the server 20 through the network 30, the server 20 obtains the target block, and the server 20 writes a plurality of key value pairs in the target block into a block cache of a memory of the blockchain; the client 10 sends the target key of the blockchain to the server 20 through the network 30, the server 20 acquires the target key of the blockchain, the server 20 reads the target value corresponding to the target key from the blockcache, and the server 20 sends the target key and the target value to the client 10; wherein the target key and the target value form a key value pair; the block cache is a first-level hash table, the first-level hash table comprises a second-level hash table corresponding to the block height of each block in at least two blocks of a block chain and the block height of each block, the second-level hash table corresponding to the block height of each block in at least two blocks of the block chain and the block height of each block form key value pairs, the second-level hash table comprises a plurality of third-level hash tables, and each key value pair in all key value pairs of each block is stored in any one of the third-level hash tables. The client 10 stores the target block, the target key, and the target value in the database 40.

It will be appreciated that the above is only an example, and the present embodiment is not limited thereto.

The terminal may be a smart phone (such as an Android phone, an iOS phone, etc.), a phone simulator, a tablet computer, a notebook computer, a digital broadcast receiver, a MID (Mobile Internet Devices, mobile internet device), a PDA (personal digital assistant), etc. The server may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server or a server cluster for providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs (Content Delivery Network, content delivery networks), basic cloud computing services such as big data and artificial intelligent platforms, and the like.

Cloud computing (clouding) is a computing model that distributes computing tasks across a large pool of computers, enabling various application systems to acquire computing power, storage space, and information services as needed. The network that provides the resources is referred to as the "cloud". Resources in the cloud are infinitely expandable in the sense of users, and can be acquired at any time, used as needed, expanded at any time and paid for use as needed.

As a basic capability provider of cloud computing, a cloud computing resource pool (cloud platform for short, generally referred to as IaaS (Infrastructure as a Service, infrastructure as a service) platform) is established, in which multiple types of virtual resources are deployed for external clients to select for use.

According to the logic function division, a PaaS (Platform as a Service ) layer can be deployed on an IaaS (Infrastructure as a Service ) layer, and a SaaS (Software as a Service, software as a service) layer can be deployed above the PaaS layer, or the SaaS can be directly deployed on the IaaS. PaaS is a platform on which software runs, such as a database, web container, etc. SaaS is a wide variety of business software such as web portals, sms mass senders, etc. Generally, saaS and PaaS are upper layers relative to IaaS.

The artificial intelligence cloud Service is also commonly referred to as AIaaS (AIas a Service, chinese is "AI as Service"). The service mode of the artificial intelligent platform is the mainstream at present, and particularly, the AIaaS platform can split several common AI services and provide independent or packaged services at the cloud. This service mode is similar to an AI theme mall: all developers can access one or more artificial intelligence services provided by the use platform through an API interface, and partial deep developers can also use an AI framework and AI infrastructure provided by the platform to deploy and operate and maintain self-proprietary cloud artificial intelligence services.

The network may include, but is not limited to: a wired network, a wireless network, wherein the wired network comprises: local area networks, metropolitan area networks, and wide area networks, the wireless network comprising: bluetooth, wi-Fi, and other networks implementing wireless communications. And in particular, the method can be determined based on actual application scene requirements, and is not limited herein.

Referring to fig. 4, fig. 4 is a schematic flow chart of a data processing method provided in the embodiment of the present application, where the method may be performed by any electronic device, for example, a server, and as an optional implementation manner, the method may be performed by the server, and for convenience of description, in the following description of some optional embodiments, a description will be given taking the server as an implementation body of the method. As shown in fig. 4, the data processing method provided by the embodiment of the application includes the following steps:

s201, acquiring a target key of a target block or a block chain.

Specifically, the target block may be a block in a blockchain located in the server; the target block may also be a block sent to the server by the user terminal; one block includes a plurality of KV (Key-Value pairs), and both Key (Key) and Value types are string strings; for example, if a block has 1 ten thousand transactions, each transaction has 100 KV, then the block includes a total of 10≡6 KV. The target Key of the blockchain may be a Key present in at least one block of the blockchain.

S202, writing a plurality of key value pairs in a target block into a block cache of a memory of a block chain; or reading a target value corresponding to the target key from the block cache.

Specifically, the block cache may be a data structure that sets key-value pairs into the memory of the blockchain, i.e., the block cache is a memory structure; the chunk cache may be a data structure in the Go language, i.e., the chunk cache is a hash table. The block cache may implement fast read or write key-value pairs. The plurality of key-value pairs in the target block may be all key-value pairs in the target block, and all key-value pairs in the target block are written into the block cache of the memory of the blockchain. The target Value corresponding to the target key may be a Value present in at least one block of the blockchain. The target value is read from the tile cache based on the target key.

Specifically, as shown in fig. 5, the block cache (blockCache) is a first-level hash table, where the first-level hash table includes a block height (blockHeight) of each block of at least two blocks of the blockchain and a second-level hash table (convergencymap) corresponding to the block height of each block, for example, at least two blocks are 10 blocks, and the block heights of the 10 blocks are respectively a block height 0, a block height 1, a block height 2, a block height 3, a block height 4, a block height 5, a block height 6, a block height 7, a block height 8, and a block height 9, and the block heights of the 10 blocks correspond to 10 different second-level hash tables, that is, the block height of each block corresponds to one second-level hash table. The block height (blockHeight) of each block and the second level hash table (ConcurrencyMap) corresponding to the block height of each block in at least two blocks of the blockchain form a Key-Value pair (Key: blockHeight, value: concurrencyMap), i.e., key is blockHeight, value is ConcurrencyMap. The blockCache can be expressed as map [ uint64] ConcurrencyMap, the map [ uint64] ConcurrencyMap indicates that the type Key (blockHeight) is the int64 type, and Value is ConcurrencyMap.

The second level hash table (ConcurrencyMap) includes a plurality of third level hash tables (Maps), each key pair of the total key pairs of each chunk being stored within any one of the plurality of third level hash tables. A second level hash table stores all Key value pairs in a block, maps a Key (string type) of any Key value pair in the block by using a hash function, such as a fnv function or a fnv function, maps the Key to a third level hash table in a plurality of third level hash tables, and the value range of an index number (index) of the third level hash table may be 0-N, where the Key may be a string type, and N is a positive integer, such as N is 31; the total number of the plurality of third level hash tables may be n+1, for example, if N is 31, one second level hash table includes 32 third level hash tables, all key value pairs in the block may be allocated to the 32 third level hash tables, the key value pairs allocated between the respective third level hash tables in the 32 third level hash tables may be different, the 32 third level hash tables store all key value pairs in the block, and each of the 32 third level hash tables may store a plurality of key value pairs in the block. The key pairs in the chunk may also be unassigned into a portion of the third-level hash tables of the plurality of third-level hash tables, i.e., there are no key pairs in a portion of the third-level hash tables.

In the embodiment of the application, the target key of the target block or the block chain is acquired; writing a plurality of key value pairs in a target block into a block cache of a memory of a block chain; or reading a target value corresponding to the target key from the block cache; wherein the target key and the target value form a key value pair; the block cache is a first-level hash table, the first-level hash table comprises a second-level hash table corresponding to the block height of each block in at least two blocks of a block chain and the block height of each block, the second-level hash table corresponding to the block height of each block in at least two blocks of the block chain and the block height of each block form key value pairs, the second-level hash table comprises a plurality of third-level hash tables, and each key value pair in all key value pairs of each block is stored in any one of the third-level hash tables; therefore, based on the cache multi-level architecture, namely the first-level hash table, the second-level hash table and the third-level hash table, the write cache or the read cache is carried out, a plurality of key value pairs in the target block can be quickly written into the cache, or target values corresponding to the target keys can be quickly read from the cache, and the preemption of locks is reduced, namely the holding time of the locks is reduced, so that the efficiency of writing the cache and the efficiency of reading the cache are improved.

In one embodiment, the block cache for writing a plurality of key-value pairs in a target block to the memory of the blockchain includes steps A1-A3:

step A1, a new second-level hash table is acquired.

Specifically, to write multiple key value pairs in a target block into the block cache of the memory of the blockchain, a second-level hash table (ConcurrencyMap) is first acquired, i.e., a new second-level hash table, which corresponds to the target block.

In one embodiment, obtaining a new second level hash table includes:

Specifically, creating a new second-level hash table (ConcurrencyMap) in real time based on an object pool in a memory of a blockchain; a new second level hash table may also be invoked from the object pool in the memory of the blockchain. For example, the object pool in memory of the blockchain may be the object pool sync.

And step A2, writing a plurality of key value pairs in the target block into a new second-level hash table.

Specifically, a plurality of key value pairs in the target block are written into a new second-level hash table (ConcurrencyMap) through a function ConcurrencyMapA. Set; the new second-level hash table establishes a mapping relationship between the Key (Key) of each Key value pair in the target block and one third-level hash table (Map) in the plurality of third-level hash tables through a hash function, such as fnv function or fnv function, that is, the Key (Key) is mapped to one third-level hash table in the plurality of third-level hash tables, so that the plurality of Key value pairs in the target block can be written into the plurality of third-level hash tables included in the new second-level hash table.

Specifically, the hash function may be a function of fnv, fnv36, fnv, or the like. The total number of the plurality of third-level hash tables corresponding to the new second-level hash table, that is, the total number of the plurality of third-level hash tables included in the new second-level hash table may be M, where M is a positive integer, for example, M is 32, 64, etc.; m is 32, the 32 third level hash tables can be distinguished by index numbers (index), the index numbers of the 32 third level hash tables are 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31 and 32, respectively; all key-value pairs in the target block may be uniformly distributed into the 32 third-level hash tables, and all key-value pairs in the target block may also be distributed into some of the 32 third-level hash tables, for example, 30 third-level hash tables, and 2 third-level hash tables other than the 30 third-level hash tables in the 32 third-level hash tables may not be distributed with key-value pairs.

And step A3, writing the new second-level hash table into a block cache of the memory of the block chain.

Specifically, the block cache is a first level hash table, the first level hash table includes a limited number of second level hash tables, for example, the first level hash table includes 10 second level hash tables; thus, writing a new second level hash table to the blockcache of the blockchain memory, i.e., writing a new second level hash table to the first level hash table, the new second level hash table will replace one of the second level hash tables.

the first level hash table is released.

Specifically, the process of writing the new second level hash table to the blockcache of the blockchain memory includes: locking the first-stage hash table, and writing the new second-stage hash table and the block height of the target block into the first-stage hash table; the new second-level hash table and the block height of the target block form a key value pair; the first level hash table is released. The lock holding time in the process is very short, and only one simple hashset operation (the block heights of the new second-level hash table and the target block are written into the first-level hash table), so the time complexity of the process is O (1), the number of times of O (1) representing operation is 1, the process operation is fast, the new second-level hash table can be quickly written into the block cache of the memory of the blockchain, the lock preemption is reduced, namely, the lock holding time is reduced, and the cache writing efficiency is improved.

Specifically, the block height of a block may refer to the position of the block in the blockchain, with smaller block heights indicating longer storage times for data (key-value pairs) in the block; for example, the block heights of the two blocks are 2 and 3, respectively, the data in the block with the block height of 2 is 2021 stored in month 1, and the data in the block with the block height of 3 is 2021 stored in month 3, that is, the data in the block with the block height of 3 is updated on the basis of the data in the block with the block height of 2.

For example, the at least two blocks are 10 blocks, and the block heights of the 10 blocks are respectively block height 0, block height 1, block height 2, block height 3, block height 4, block height 5, block height 6, block height 7, block height 8 and block height 9, wherein the block heights of the 10 blocks correspond to 10 different second-level hash tables, namely, the block height of each block corresponds to one second-level hash table; the block height 0 is the smallest block height among the 10 blocks, the second-level hash table corresponding to the block height 0 and the block height 0 is deleted, and the new second-level hash table and the block height of the target block are written into the first-level hash table.

and releasing the lock on the block cache.

Specifically, when all keys in a block cache (blockCache) are read, the block cache is locked, wherein all keys are the block heights (blockHeight) of at least two blocks, and each Key in all keys is the block height of one block. Reading all keys in a block cache (block cache) to obtain an array block height Arr comprising the block heights of at least two blocks; for example, at least two blocks are 10 blocks, the 10 blocks corresponding to 10 block heights, wherein one block corresponds to one block height, the array block height arr includes the block heights of the 10 blocks, i.e., the array block height arr includes the 10 block heights, but the 10 block heights included by the array block height arr are unordered. Because only 10 blocks are high, the holding time of the lock is very short, and the preemption of the lock is reduced, namely the holding time of the lock is reduced, so that the efficiency of reading and caching is improved.

Sequencing 10 blocks in the array block height Arr from large to small to obtain a sequenced ordered array sortblock height Arr; wherein the ordering may be a fast ordering. For example, at least two blocks are 10 blocks, the block heights of the 10 blocks are rapidly ordered from large to small, and the block heights in the ordered set sortblockheight arr are sequentially ordered from large to small: block height 9, block height 8, block height 7, block height 6, block height 5, block height 4, block height 3, block height 2, block height 1 and block height 0.

According to MVCC, traversing the ordered number group sortblockheight arr from large to small according to the block height, for example traversing the ordered number group sortblockheight arr from block height 9, block height 8, block height 7, block height 6, block height 5, block height 4, block height 3, block height 2, block height 1, block height 0 in sequence, if determining that the second level hash table (concurrencylmap) corresponding to block height 8, the second level hash table corresponding to block height 6 and the second level hash table corresponding to block height 2 all have the same Key (Key) as the target Key; since the block height 8 is the maximum block height among the block heights 8, 6 and 2, a Value (Value) corresponding to the same Key (Key) as the target Key in the block height 8 is determined as the target Value corresponding to the target Key; the second level hash table (concurrencypap) corresponding to the block height 8, for example, concurrencyppa, may read the value corresponding to the target key (inputKey), i.e., the target value, from the concurrencyppa by calling the function concurrencyppa. Get ("inputKey") in memory.

It should be noted that the key value pairs included in two or more blocks may be identical, i.e., the key value pairs are multi-version. For example, the second-level hash table corresponding to the block height 8, the second-level hash table corresponding to the block height 6, and the second-level hash table corresponding to the block height 2 all have the same Key (Key) as the target Key, that is, the Key value pair consisting of the target Key and the target value is multi-version; the block height 8 is the maximum block height among the block height 8, the block height 6 and the block height 2, and the value corresponding to the target key, i.e., the target value, in the second-level hash table corresponding to the block height 8 is read.

Specifically, according to MVCC, traversing the ordered number group sortblockheight arr from large to small according to the order of block height, for example traversing the ordered number group sortblockheight arr from block height 9, block height 8, block height 7, block height 6, block height 5, block height 4, block height 3, block height 2, block height 1, block height 0 in sequence, if determining that the second level hash table (concurrencylmap) corresponding to block height 8, the second level hash table corresponding to block height 6 and the second level hash table corresponding to block height 2 all have the same Key (Key) as the target Key; since block height 8 is the largest block height among block height 8, block height 6, and block height 2, block height 8 is the first block height; therefore, a Value (Value) corresponding to the same Key (Key) as the target Key in the block height 8 is determined as the target Value corresponding to the target Key; the second level hash table (concurrencypap) corresponding to the block height 8, for example, is a concurrencyppa, that is, the concurrencyppa is a second level hash table corresponding to the first block height, and the value corresponding to the target key (inputKey), that is, the target value, may be read from the concurrencyppa by calling a function concurrencyppa. Get ("inputKey") in the memory.

Specifically, for example, the ordered number group sortblockheight arr is traversed sequentially from block height 9, block height 8, block height 7, block height 6, block height 5, block height 4, block height 3, block height 2, block height 1, and block height 0, and if it is determined that the second level hash table (condurencymap) corresponding to block height 8, the second level hash table corresponding to block height 6, and the second level hash table corresponding to block height 2 all have the same Key (Key) as the target Key; since block height 8 is the largest block height among block height 8, block height 6, and block height 2, block height 8 is the first block height.

The application of the embodiment of the application has at least the following beneficial effects:

based on a cache multi-level architecture, namely a first-level hash table, a second-level hash table and a third-level hash table, write cache or read cache is carried out, so that a plurality of key value pairs in a target block can be written into the cache or target values corresponding to the target keys can be read from the cache, preemption of locks is reduced, namely the holding time of the locks is reduced, and the efficiency of writing the cache and the efficiency of reading the cache are improved.

In order to better understand the method provided by the embodiment of the present application, the scheme of the embodiment of the present application is further described below with reference to examples of specific application scenarios.

The method provided by the embodiment of the application can be applied to a blockchain scene, such as a cache of a read-write blockchain, and the like.

In one embodiment, as shown in FIG. 6, in a blockchain, a block has many KV (key-value pairs) that are written to the block Cache to provide read acceleration and write acceleration when flush Cache to db (database). The write acceleration is capable of rapidly writing KV from the cache to db; if there is no block Cache, the data needs to be read again from the WAL and then written to the db, so that a write operation is slow.

The process of writing a block to db includes: writing KV in a block into a pre-write log (WAL), writing KV in the block into a block Cache (block Cache), and finally writing KV into db, such as Leveldb, through the block Cache; the writing amount of KV is large, for example, 10000 transactions are performed in one block, 100 KV are performed in 1 transaction, and the total KV is 10000 x 100=10≡6. During the write process, there is also a read that occurs, i.e., there are other operations that need to read KV from the block Cache.

In one embodiment, benchMark test benchmarks for adding a block to cache are shown in table 1, benchmarks for deleting a block from cache are shown in table 2, and benchmarks for concurrence Get (reading one KV from cache) and Set (writing all KV in a block to cache) are shown in table 3.

Table 1: adding a block to the BenchMark of the cache

The block cache is a block cache in the method provided by the embodiment of the application; addLock-counts represents: in the preset time, the blocks can be added to the cache for times, for example, 5290111 times, i.e. 5290111 blocks are added to the cache. AddLock-ops represent the time required for each operation, e.g., 208ns/op represents 208ns (nanoseconds) for each operation.

As shown in table 1, for AddBlock operations (adding a block to the cache), the performance of the block cache is best, e.g., 24600 times the performance of the block cache is mapCache and 55565 times the performance of the block cache is syncMapCache.

Table 2: benchMark deleting block from cache

As shown in Table 2, for DelBlock operation (deleting one block from cache), the performance of the block cache is best, with each operation requiring 24.4ns, e.g., 75448 times the performance of the block cache as mapC ache, 80666 times the performance of the block cache as syncMapCache.

Table 3: benchMark with concurrent Get and Set

Cache type	counts	ops	Performance pair base
				blockCache	103902	300354ns/op	1
mapCache	402	3597079ns/op	1/11
				syncMapCache	291	4468245ns/op	1/14

As shown in table 3, for Set (AddBlock) and Get operations performed in a concurrent mix, each operation (including Set and Get, i.e., read and write) is averaged, and only 300354ns is required for the block cache, which is 11 times higher than mapCache, which is 14 times higher than syncMapCache.

In a specific application scenario embodiment, for example, a cache scenario of a read-write blockchain, referring to fig. 7, a process flow of a data processing method is shown, where the method may be performed by any electronic device, for example, a server, and as an alternative implementation manner, the method may be performed by the server, and for convenience of description, in the following description of some alternative embodiments, a description will be given taking the server as an implementation body of the method as an example. As shown in fig. 7, the processing flow of the data processing method provided by the embodiment of the application includes the following steps:

s701, constructing a block cache of a multi-level architecture in a memory of a block chain.

Specifically, the block cache of the multi-level architecture may be a block cache of a three-level architecture, for example, as shown in fig. 5, including a first level hash table (blockCache), a second level hash table (conducincymap), and a third level hash table (Map); the block cache (blockCache) is a first-level hash table, the first-level hash table comprises a block height (blockHeight) of each block in at least two blocks of a blockchain and a second-level hash table (ConcurrencyMap) corresponding to the block height of each block, and the block height (blockHeight) of each block in at least two blocks of the blockchain and the second-level hash table (ConcurrencyMap) corresponding to the block height of each block form a Key Value pair (Key: blockHeight, value: concurrencyMap), namely Key is blockHeight, and Value is ConcurrencyMap. The second level hash table (ConcurrencyMap) includes a plurality of third level hash tables (Maps), each key pair of the total key pairs of each chunk being stored within any one of the plurality of third level hash tables. A second level hash table stores all Key-value pairs in a block, maps keys (string types) of any Key-value pair in the block using a hash function, such as fnv function or fnv function, and maps the keys to one of a plurality of third level hash tables.

It should be noted that, the ConcurrencyMap uses fnv function or fnv function to map, and as the fnv function or fnv function can ensure that the randomness is very strong and the collision probability is very low, it can ensure that in the concurrent update of ConcurrencyMap, the lock conflict is very low, and if there are 64 threads (each thread corresponds to one third level hash table, i.e. 64 threads correspond to 64 third level hash tables), it can ensure that 64 computations are completed simultaneously in parallel and there is no lock contention. By reducing the competition of the locks, the holding time of the locks is reduced, thereby ensuring the high efficiency of reading and writing.

S702, writing all key value pairs in the block A into a pre-write log.

Specifically, as shown in fig. 6, downtime will cause a key pair (KV) in the block cache to be lost, and to prevent the key pair from being lost, all key pairs in block a may be written to the pre-write log (WAL) first.

S703, writing all key value pairs in the block A into the block cache.

Specifically, all key value pairs in one tile (tile a, the target tile) are written to the tile cache (blockCache). Creating a second level hash table (ConcurrencyMap) in real time based on the object pool in the memory of the blockchain; a second level hash table may also be invoked from a pool of objects in the memory of the blockchain. For example, the object pool may be an object pool sync. Locking the first-level hash table, writing all key value pairs in the block into the second-level hash table, and releasing the lock of the first-level hash table; locking the first-level hash table, writing the second-level hash table into a block cache, and releasing the lock of the first-level hash table.

It should be noted that, writing a block (block) into the block cache (block cache) updates the time complexity corresponding to the first-level hash table to be O (1), so the time for holding the lock on the first-level hash table is very short, i.e. the holding time of the lock is reduced, thereby improving the efficiency of writing the cache.

S704, reading the key value pair from the block cache.

Specifically, locking a block cache, and sequencing the block heights of 10 blocks in a block chain to obtain an ordered array, wherein the ordered array comprises key value pairs of each block in the 10 blocks, and the block heights of the 10 blocks are sequenced from large to small in the ordered array; based on the ordered array, reading a target value corresponding to the target key from the block cache; releasing the lock of the block cache; the key value pair formed by the target key and the target value is the key value pair read from the block cache.

It should be noted that, during the reading process, only the block cache (block cache) needs to be locked or released, and since only 10 elements (the block heights of 10 blocks) exist in the block cache, the 10 elements are traversed very fast at a time. In most cases, the write and read are lock-free contentions, and even if the write and read are lock-free contentions, the lock contention time is very short; by reducing the competition of the locks, the holding time of the locks is reduced, thereby ensuring the high efficiency of reading and writing.

S705, writing all key value pairs of the block A in the block cache into a database.

Specifically, as shown in fig. 5, all key pairs for block a in the block cache are written to db, e.g., leveldb.

based on the multi-level architecture of the block cache, all key value pairs in the block can be written into the block cache or the key value pairs can be read from the cache quickly, and the holding time of the lock is reduced by reducing the competition of the lock, so that the efficiency of writing the cache and the efficiency of reading the cache are improved.

The embodiment of the application also provides a data processing device, and a schematic structural diagram of the data processing device is shown in fig. 8, and the data processing device 90 comprises a first processing module 901 and a second processing module 902.

A first processing module 901, configured to obtain a target key of a target block or a blockchain;

a second processing module 902, configured to write a plurality of key value pairs in a target block into a block cache of a memory of the blockchain; or reading a target value corresponding to the target key from the block cache;

In one embodiment, the second processing module 902 is specifically configured to:

acquiring a new second-level hash table;

the first level hash table is released.

And deleting the second-level hash table corresponding to the minimum block height in the block heights of the at least two blocks, and writing the new second-level hash table and the block height of the target block into the first-level hash table.

and releasing the lock on the block cache.

The embodiment of the application also provides an electronic device, a schematic structural diagram of which is shown in fig. 9, and an electronic device 4000 shown in fig. 9 includes: a processor 4001 and a memory 4003. Wherein the processor 4001 is coupled to the memory 4003, such as via a bus 4002. Optionally, the electronic device 4000 may further comprise a transceiver 4004, the transceiver 4004 may be used for data interaction between the electronic device and other electronic devices, such as transmission of data and/or reception of data, etc. It should be noted that, in practical applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 is not limited to the embodiment of the present application.

The processor 4001 may be a CPU (Central Processing Unit ), general purpose processor, DSP (Digital Signal Processor, data signal processor), ASIC (Application Specific Integrated Circuit ), FPGA (Field Programmable Gate Array, field programmable gate array) or other programmable logic device, transistor logic device, hardware components, or any combination thereof. Which may implement or perform the various exemplary logic blocks, modules and circuits described in connection with this disclosure. The processor 4001 may also be a combination that implements computing functionality, e.g., comprising one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

Bus 4002 may include a path to transfer information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect, peripheral component interconnect standard) bus or an EISA (Extended Industry Standard Architecture ) bus, or the like. The bus 4002 can be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one thick line is shown in fig. 9, but not only one bus or one type of bus.

Memory 4003 may be, but is not limited to, ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, RAM (Random Access Memory ) or other type of dynamic storage device that can store information and instructions, EEPROM (Electrically Erasable Programmable Read Only Memory ), CD-ROM (Compact Disc Read Only Memory, compact disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store a computer program and that can be Read by a computer.

The memory 4003 is used for storing a computer program for executing an embodiment of the present application, and is controlled to be executed by the processor 4001. The processor 4001 is configured to execute a computer program stored in the memory 4003 to realize the steps shown in the foregoing method embodiment.

Among them, electronic devices include, but are not limited to: a server, etc.

Embodiments of the present application provide a computer readable storage medium having a computer program stored thereon, which when executed by a processor, implements the steps of the foregoing method embodiments and corresponding content.

The embodiment of the application also provides a computer program product, which comprises a computer program, wherein the computer program can realize the steps and corresponding contents of the embodiment of the method when being executed by a processor.

Based on the same principle as the method provided by the embodiments of the present application, the embodiments of the present application also provide a computer program product or a computer program, which comprises computer instructions stored in a computer-readable storage medium. The computer instructions are read from a computer-readable storage medium by a processor of a computer device, and executed by the processor, cause the computer device to perform the method provided in any of the alternative embodiments of the application described above.

It should be understood that, although various operation steps are indicated by arrows in the flowcharts of the embodiments of the present application, the order in which these steps are implemented is not limited to the order indicated by the arrows. In some implementations of embodiments of the application, the implementation steps in the flowcharts may be performed in other orders as desired, unless explicitly stated herein. Furthermore, some or all of the steps in the flowcharts may include multiple sub-steps or multiple stages based on the actual implementation scenario. Some or all of these sub-steps or phases may be performed at the same time, or each of these sub-steps or phases may be performed at different times, respectively. In the case of different execution time, the execution sequence of the sub-steps or stages can be flexibly configured according to the requirement, which is not limited by the embodiment of the present application.

The foregoing is merely an optional implementation manner of some of the implementation scenarios of the present application, and it should be noted that, for those skilled in the art, other similar implementation manners based on the technical ideas of the present application are adopted without departing from the technical ideas of the scheme of the present application, and the implementation manner is also within the protection scope of the embodiments of the present application.

Claims

1. A method of data processing, comprising:

acquiring a target key of a target block or a block chain;

writing a plurality of key value pairs in the target block into a block cache of a memory of the blockchain; or reading a target value corresponding to the target key from the block cache;

wherein the target key and the target value constitute a key-value pair; the block cache is a first-level hash table, the first-level hash table comprises a second-level hash table corresponding to the block height of each block in at least two blocks of the block chain and the block height of each block, the second-level hash table corresponding to the block height of each block in at least two blocks of the block chain and the block height of each block form a key value pair, the second-level hash table comprises a plurality of third-level hash tables, and each key value pair in all key value pairs of each block is stored in any one of the third-level hash tables.

2. The method of claim 1, wherein the writing the plurality of key-value pairs in the target block to the block cache of the blockchain memory comprises:

acquiring a new second-level hash table;

writing a plurality of key value pairs in the target block to the new second-level hash table;

and writing the new second-level hash table into a block cache of the memory of the block chain.

3. The method of claim 2, wherein the obtaining a new second level hash table comprises:

based on the object pool in the memory of the blockchain, a new second-level hash table is acquired.

4. The method of claim 2, wherein the writing the plurality of key-value pairs in the target block to the new second-level hash table comprises:

writing a plurality of key value pairs in the target block into a plurality of third-level hash tables corresponding to the new second-level hash table through a hash function; each of the plurality of key-value pairs is stored within any one of a plurality of third-level hash tables corresponding to the new second-level hash table.

5. The method of claim 2, wherein writing the new second level hash table to the blockcache of the blockchain memory comprises:

and releasing the lock on the first-stage hash table.

6. The method of claim 5, wherein writing the new second-level hash table and the block height of the target block into the first-level hash table comprises:

and deleting the second-level hash table corresponding to the minimum block height from the block heights of the at least two blocks, and writing the new second-level hash table and the block height of the target block into the first-level hash table.

7. The method of claim 1, wherein the reading the target value corresponding to the target key from the tile cache comprises:

locking the block cache, and sequencing the block heights of the at least two blocks to obtain an ordered array, wherein the ordered array comprises key value pairs of each block in the at least two blocks, and the block heights of the at least two blocks are sequenced from large to small in the ordered array;

Reading a target value corresponding to the target key from the block cache based on the ordered array;

and releasing the lock of the block cache.

8. The method of claim 7, wherein the reading the target value corresponding to the target key from the tile cache based on the ordered array comprises:

determining a first block height corresponding to the target key based on the ordered array;

and reading a target value corresponding to the target key from a second-level hash table corresponding to the first block height.

9. The method of claim 8, wherein determining the first tile height corresponding to the target key based on the ordered array comprises:

determining at least two keys from the ordered array that are identical to the target key;

10. A data processing apparatus, comprising:

The second processing module is used for writing a plurality of key value pairs in the target block into a block cache of the memory of the block chain; or reading a target value corresponding to the target key from the block cache;

11. An electronic device comprising a memory, a processor and a computer program stored on the memory, characterized in that the processor executes the computer program to carry out the steps of the method according to any one of claims 1-9.

12. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method according to any of claims 1-9.

13. A computer program product comprising a computer program, characterized in that the computer program, when executed by a processor, implements the steps of the method according to any one of claims 1-9.