CN116842012A

CN116842012A - Method, device, equipment and storage medium for storing Redis cluster in fragments

Info

Publication number: CN116842012A
Application number: CN202310786136.9A
Authority: CN
Inventors: 谢青
Original assignee: Ping An Property and Casualty Insurance Company of China Ltd
Current assignee: Ping An Property and Casualty Insurance Company of China Ltd
Priority date: 2023-06-29
Filing date: 2023-06-29
Publication date: 2023-10-03

Abstract

The application discloses a fragment storage method, device and equipment of a Redis cluster and a storage medium, belonging to the technical field of big data and the field of financial risk production. The method comprises the steps of obtaining data to be stored, carrying out hash operation on the data to be stored, obtaining hash values, dividing the hash values to obtain a first hash segment and a second hash segment, determining a data offset value of the second hash segment in a hash slot based on indexing the hash slot of the first hash segment in a Redis cluster, determining a storage bitmap position of the data to be stored in the Redis cluster according to the hash slot and the data offset value, and changing the storage state of the storage bitmap position. In addition, the application also relates to a blockchain technology, and data to be stored can be stored in the blockchain. According to the application, the hash algorithm is adopted to uniformly disperse the data to different Redis cluster nodes, so that the generation of gap values is reduced, the utilization efficiency of the memory is improved, and the expandability and the load balance of the system are improved.

Description

Method, device, equipment and storage medium for storing Redis cluster in fragments

Technical Field

The application belongs to the technical field of big data and the field of financial risk production, and particularly relates to a fragment storage method, device and equipment of a Redis cluster and a storage medium.

Background

The Redis cluster bitmap (bitmap) storage scheme refers to a bitmap storage scheme implemented by using Redis as a storage engine and using its character string data type, in which each bit (bit) is used to represent a certain state or identifier, and is suitable for processing massive integer data, for example, in a financial product system, counting whether a user logs in the data of the system, and can be implemented by using the Redis cluster bitmap storage scheme.

However, in the existing reds cluster bitmap storage scheme, data are usually stored in one reds node sequentially, when a large amount of data are stored by using the storage scheme, if the offset identification of a certain data is large, a large amount of memory is required to be occupied, a large offset value is generated, so that the memory waste is serious, all the data are stored in one node in a concentrated manner, uneven load of the node is easily caused, the fault tolerance is weak, and the performance and the expansion capacity of the system are affected.

Disclosure of Invention

The embodiment of the application aims to provide a fragment storage method, a fragment storage device, computer equipment and a storage medium of a Redis cluster, so as to solve the technical problems of memory waste, uneven load, weaker fault tolerance and influence on the performance and expansion capacity of a system in the conventional Redis cluster bitmap storage scheme.

In order to solve the above technical problems, the embodiment of the present application provides a fragment storage method of a dis cluster, which adopts the following technical scheme:

a fragment storage method of a Redis cluster comprises the following steps:

receiving a data storage instruction, and acquiring data to be stored corresponding to the data storage instruction;

carrying out hash operation on data to be stored, and obtaining a hash value corresponding to the data to be stored to obtain a first hash value;

dividing the first hash value to obtain a first hash segment and a second hash segment;

indexing a hash slot corresponding to data to be stored in a Redis cluster based on the first hash segment to obtain a first hash slot;

determining a data offset value of the second hash segment in the hash slot to obtain a first data offset value;

and determining a storage bitmap position of the data to be stored in the Redis cluster according to the first hash slot and the first data offset value, obtaining a first storage bitmap position, and changing the storage state of the first storage bitmap position.

Further, the storage states include a first storage state and a second storage state, and the method includes determining a storage bitmap position of data to be stored in the Redis cluster according to the first hash slot and the first data offset value, obtaining the first storage bitmap position, and changing the storage state of the first storage bitmap position, specifically including:

Generating a data offset pointer, traversing the first hash slot using the data offset pointer based on the first data offset value;

determining a storage bitmap position of data to be stored in a Redis cluster according to a traversing result of the data offset pointer to obtain a first storage bitmap position;

the storage state of the first storage bitmap position is changed from the first storage state to the second storage state.

Further, after determining the bitmap position of the data to be stored according to the first hash slot and the first data offset value, and changing the storage state of the bitmap position, the method further comprises:

receiving a data query instruction, and acquiring data to be queried corresponding to the data query instruction;

carrying out hash operation on the data to be queried to obtain a hash value corresponding to the data to be queried, and obtaining a second hash value;

dividing the second hash value to obtain a third hash segment and a fourth hash segment;

indexing a hash slot corresponding to the data to be queried in the Redis cluster based on the third hash segment to obtain a second hash slot;

determining a data offset value of the fourth hash segment in the hash slot to obtain a second data offset value;

determining a storage bitmap position of the data to be queried in the Redis cluster according to the second hash slot and the second data offset value to obtain a second storage bitmap position;

And detecting the storage state of the second storage bitmap position, and determining whether the bitmap storage of the data to be queried is completed according to the detection result of the storage state.

Further, detecting a storage state of the second storage bitmap position, and determining whether the data to be queried is bitmap storage according to the detection result of the storage state, specifically including:

if the storage state of the second storage bitmap position is the first storage state, determining that the bitmap storage of the data to be queried is not completed;

generating a first data query result based on the first storage state, and outputting the first data query result;

if the storage state of the second storage bitmap position is the second storage state, determining that bitmap storage of the data to be queried is completed;

and generating a second data query result based on the second storage state, and outputting the second data query result.

Further, the first hash value is divided to obtain a first hash segment and a second hash segment, which specifically include:

determining the index length of a hash slot in a Redis cluster;

performing bit operation on the first hash value based on the index length of the hash slot bit to obtain a first hash segment;

and carrying out mask operation on the first hash value based on the first hash fragment to obtain a second hash fragment.

Further, performing bit operation on the first hash value based on the index length of the hash slot to obtain a first hash segment, which specifically includes:

determining a start bit of the first hash value;

and starting from the initial bit of the first hash value, performing a bit-by-bit right shift operation on the first hash value until the right shift length is equal to the index length of the hash slot, and obtaining the first hash segment.

In order to solve the technical problems, the embodiment of the application also provides a fragment storage device of a Redis cluster, which adopts the following technical scheme:

a tiled memory device of a dis cluster, comprising:

the storage instruction receiving module is used for receiving the data storage instruction and acquiring the data to be stored corresponding to the data storage instruction;

the first hash operation module is used for carrying out hash operation on the data to be stored, obtaining a hash value corresponding to the data to be stored, and obtaining a first hash value;

the first hash dividing module is used for dividing the first hash value to obtain a first hash segment and a second hash segment;

the first slot index module is used for indexing hash slots corresponding to data to be stored in the Redis cluster based on the first hash fragments to obtain first hash slots;

The first data offset module is used for determining a data offset value of the second hash segment in the hash slot to obtain a first data offset value;

and the data bitmap storage module is used for determining the storage bitmap position of the data to be stored in the Redis cluster according to the first hash slot and the first data offset value, obtaining the first storage bitmap position and changing the storage state of the first storage bitmap position.

Further, the partitioned storage device of the dis cluster further includes:

the query instruction receiving module is used for receiving a data query instruction and acquiring data to be queried corresponding to the data query instruction;

the second hash operation module is used for carrying out hash operation on the data to be queried to obtain a hash value corresponding to the data to be queried, and obtaining a second hash value;

the second hash dividing module is used for dividing the second hash value to obtain a third hash segment and a fourth hash segment;

the second slot index module is used for indexing the hash slots corresponding to the data to be queried in the Redis cluster based on the third hash segment to obtain second hash slots;

the second data offset module is used for determining a data offset value of the fourth hash segment in the hash slot to obtain a second data offset value;

The bitmap data query module is used for determining the storage bitmap position of the data to be queried in the Redis cluster according to the second hash slot position and the second data offset value to obtain a second storage bitmap position;

and the storage state detection module is used for detecting the storage state of the second storage bitmap position and determining whether the bitmap storage of the data to be queried is completed according to the detection result of the storage state.

In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical schemes:

a computer device comprising a memory having stored therein computer readable instructions which when executed by a processor implement the steps of the method of shard storage of a dis cluster as claimed in any one of the preceding claims.

In order to solve the above technical problems, an embodiment of the present application further provides a computer readable storage medium, which adopts the following technical schemes:

a computer readable storage medium having stored thereon computer readable instructions which when executed by a processor implement the steps of the method of sharding a dis cluster as claimed in any of the preceding claims.

Compared with the prior art, the embodiment of the application has the following main beneficial effects:

the application discloses a fragment storage method, device and equipment of a Redis cluster and a storage medium, belonging to the technical field of big data and the field of financial risk production. According to the method, data to be stored corresponding to the data storage instruction are obtained, hash operation is carried out on the data to be stored, a hash value corresponding to the data to be stored is obtained, a first hash value is obtained, the first hash value is divided, a first hash segment and a second hash segment are obtained, a hash slot corresponding to the data to be stored is indexed in a Redis cluster based on the first hash segment, a first hash slot is obtained, a data offset value of the second hash segment in the hash slot is determined, a first data offset value is obtained, a storage bitmap position of the data to be stored in the Redis cluster is determined according to the first hash slot and the first data offset value, a first storage bitmap position is obtained, and a storage state of the first storage bitmap position is changed. According to the application, the hash value of the data is calculated, the hash slot position is indexed in the Redis cluster by the hash value, the data offset value is determined by the hash value, then the data is uniformly dispersed on different Redis cluster nodes, so that the data is stored in a slicing way, when a large amount of data is required to be stored, the data can be uniformly dispersed on a plurality of nodes by the slicing way, the memory occupation of a single node is avoided, the generation of a gap value is reduced, the memory utilization efficiency is improved, and the expandability and the load balance of the system are improved.

Drawings

In order to more clearly illustrate the solution of the present application, a brief description will be given below of the drawings required for the description of the embodiments of the present application, it being apparent that the drawings in the following description are some embodiments of the present application, and that other drawings may be obtained from these drawings without the exercise of inventive effort for a person of ordinary skill in the art.

FIG. 1 illustrates an exemplary system architecture diagram in which the present application may be applied;

FIG. 2 illustrates a flow chart of one embodiment of a sharded storage method of a Redis cluster in accordance with the present application;

FIG. 3 illustrates a schematic diagram of one embodiment of a tiled memory device of Redis clusters in accordance with the present application;

fig. 4 shows a schematic structural diagram of an embodiment of a computer device according to the application.

Detailed Description

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description of the applications herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having" and any variations thereof in the description of the application and the claims and the description of the drawings above are intended to cover a non-exclusive inclusion. The terms first, second and the like in the description and in the claims or in the above-described figures, are used for distinguishing between different objects and not necessarily for describing a sequential or chronological order.

Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments.

In order to make the person skilled in the art better understand the solution of the present application, the technical solution of the embodiment of the present application will be clearly and completely described below with reference to the accompanying drawings.

As shown in fig. 1, a system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used as a medium to provide communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

The user may interact with the server 105 via the network 104 using the terminal devices 101, 102, 103 to receive or send messages or the like. Various communication client applications, such as a web browser application, a shopping class application, a search class application, an instant messaging tool, a mailbox client, social platform software, etc., may be installed on the terminal devices 101, 102, 103.

The terminal devices 101, 102, 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smartphones, tablet computers, electronic book readers, MP3 players (Moving Picture Experts Group Audio Layer III, dynamic video expert compression standard audio plane 3), MP4 (Moving Picture Experts Group Audio Layer IV, dynamic video expert compression standard audio plane 4) players, laptop and desktop computers, and the like.

The server 105 may be a server that provides various services, such as a background server that provides support for pages displayed on the terminal devices 101, 102, 103, and may be a stand-alone server, or may be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

It should be noted that, the method for storing the fragments of the Redis cluster provided by the embodiment of the present application is generally executed by a server, and accordingly, the fragment storage device of the Redis cluster is generally disposed in the server.

It should be understood that the number of terminal devices, networks and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

With continued reference to FIG. 2, a flow chart of one embodiment of a sharded storage method of a Redis cluster according to the present application is shown. The embodiment of the application can acquire and process the related data based on the artificial intelligence technology. Among these, artificial intelligence (Artificial Intelligence, AI) is the theory, method, technique and application system that uses a digital computer or a digital computer-controlled machine to simulate, extend and extend human intelligence, sense the environment, acquire knowledge and use knowledge to obtain optimal results.

Artificial intelligence infrastructure technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a robot technology, a biological recognition technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and other directions.

Redis (Remote Dictionary Server) is an open-source memory data storage system, which may also be referred to as a key store database, cache, and message middleware. The method stores data in the form of key-value pairs (key-value) and stores the data in a memory so as to realize quick data access and read-write operation. Redis supports a variety of data structures including strings (String), hashes (Hash), lists (List), collections (Set), ordered collections (issued Set), and so forth. Each data structure has corresponding operation commands, and operations such as adding, deleting, modifying and checking of data can be performed.

In the existing Redis cluster bitmap storage scheme, data are stored in one Redis node in sequence, load balancing cannot be achieved, all data are stored in one node in a concentrated mode, uneven load of the nodes is easily caused, performance and expansion capacity of a system are affected, and data loss or unavailability is easily caused. When the storage scheme is used for storing a large amount of data, if the offset identification of certain data is larger, a large amount of memory is occupied, larger offset values are generated, so that the memory waste is serious, all the data are stored on one node in a concentrated manner, uneven load of the node is easy to cause, the fault tolerance is weaker, and the performance and the expansion capacity of the system are affected.

In order to solve the technical problems, the application discloses a fragment storage method, a device, equipment and a storage medium of a Redis cluster, which belong to the technical field of big data and the field of financial risk, and the application is used for calculating the hash value of the data, indexing the hash slot in the Redis cluster through the hash value, determining the data offset value through the hash value, uniformly dispersing the data on different Redis cluster nodes, realizing the fragment storage of the data, and uniformly dispersing the data on a plurality of nodes through fragment storage when a large amount of data is required to be stored, avoiding overlarge memory occupation of a single node, reducing the generation of a gap value, improving the utilization efficiency of the memory, and improving the expandability and the load balance of a system.

In the bit map storage scheme of the Redis cluster, the slicing storage refers to dividing and distributing data to different Redis nodes for storage according to a certain rule or algorithm, and the slicing storage mode can realize the decentralized storage and load balancing of the data and improve the expandability and concurrent processing capacity of the system.

Specifically, in a specific embodiment of the present application, the process of storing the fragments is as follows:

the data is fragmented using a Hash algorithm: for each data to be stored, a Hash value, which may be an integer or a string, is calculated by applying a Hash algorithm, which typically maps the input data to a Hash code of fixed length.

Selecting a storage node according to the hash value: the Hash value is mapped to Hash slots (Hash slots) of the Redis cluster, where the Hash slots are a basic unit for distributing data in the Redis cluster, and typically are a fixed number of slots, such as 65536 slots by default, and the data is mapped to corresponding slots according to modes of high bits or low bits of the Hash value, and the like.

Storing data to the corresponding node: and storing the data into the corresponding Redis node according to the mapping result of the hash slot. Each node is responsible for managing a portion of the hash slots and maintaining the corresponding data.

By distributing data across multiple nodes, nodes can be conveniently added or subtracted to meet the changing storage requirements. Each node only needs to manage a part of data, so that the load pressure of a single node is reduced, the performance and throughput of the system are improved, and as the data are stored on a plurality of nodes in a scattered manner, when a certain node fails or is not available, the data can be obtained from other available nodes, and the availability and fault tolerance of the system are improved.

The fragment storage method of the Redis cluster comprises the following steps:

s201, receiving a data storage instruction and acquiring data to be stored corresponding to the data storage instruction.

In this embodiment, after receiving the data storage instruction, the server parses the data storage instruction, and obtains the data to be stored from the preset database according to the data storage instruction, where the data storage instruction may be from an application program or other system components.

In this embodiment, the data offset pointer is used to indicate a specific storage location of the data to be stored in the node, and its value is calculated according to the first data offset value. The data offset value is multiplied by the size of the memory space occupied by each data in the node to obtain a final data offset pointer, and in Redis, each data is stored according to a fixed size, so that the data offset pointer can be calculated by multiplying the size of each data. For example, assuming that the size of each data is N bytes and the data offset value is M, the calculation formula of the data offset pointer is: m x N. The result of the calculation is the specific storage position of the data to be stored in the node, which can be used to traverse the hash slots and determine the storage bitmap position.

The traversal result is a specific position of the data offset pointer in the hash slot, from which the storage location of the data to be stored in the node can be determined, the storage bitmap location being a mechanism for recording the storage state of the node, the value of which represents the storage state of a certain position in the node, e.g. a first storage state is configured as "0", a second storage state is configured as "1", the first storage state represents that the data is not stored, and the second storage state represents that the data is stored. Changing the storage state of the first storage bitmap position from the first storage state to the second storage state, such as changing the storage state of the first storage bitmap position from "0" to "1", the state change refers to marking the storage position of the data to be stored as used for subsequent data reading and management.

S202, carrying out hash operation on data to be stored, and obtaining a hash value corresponding to the data to be stored to obtain a first hash value.

In this embodiment, a suitable Hash function is selected to map the data to be stored into an integer or a string, and common Hash functions include MD5, SHA1, murmur Hash, and the like. And carrying out hash operation on the data to be stored, obtaining a hash value corresponding to the data to be stored, and obtaining a first hash value, wherein the first hash value can be used for indexing the hash slot and determining data offset so as to determine the storage bitmap position of the data to be stored in the Redis cluster.

S203, dividing the first hash value to obtain a first hash segment and a second hash segment.

In this embodiment, the first hash value may be divided by a bit operation and a mask operation, and the division process may be implemented according to a specific hash algorithm, so as to divide the hash value into two parts for subsequent determination of the hash slot index and the data offset. The first hash segment is a high-order hash segment in the first hash value, and the second hash segment is a low-order hash segment in the first hash value.

Determining the index length of a hash slot in a Redis cluster;

determining a start bit of the first hash value;

When the hash slot bit corresponding to the index storage data in the Redis cluster is used, the hash segment used can be set, and the Redis allows the bit number of the hash segment to be changed by modifying the configuration. In particular, the number of bits of the hash fragment may be specified using a hash-bits configuration option. The option may be set to 16, 14, 12 or 10, indicating that a 16-bit, 14-bit, 12-bit or 10-bit hash segment is used to determine the slot index, respectively.

By default, the Redis cluster uses the upper 16 bits of the hash segment as the slot index, that is, uses the 16-bit hash segment to determine the slot to which the data belongs.

In this embodiment, taking the index length of the 10-bit hash slot as an example, if the hash is an integer of 32 bits, it can be expressed as a 32-bit binary number, the hash value is divided into the upper 10 bits and the lower 22 bits by the bit operation, from the start bit of the hash value, the hash value is shifted right by 10 bits by the right shift operation, to obtain the upper 10 bits value, the upper 22 bits are masked by using the bitmask (0 x3 FFFFF), the lower 22 bits value is reserved, and the offset value can be calculated by the following formula: offset = hash &0x3FFFFF.

It should be noted that, after modifying the hash-bits configuration option, the hash slot allocation and the mapping manner of the data of the Redis cluster are affected. Therefore, careful evaluation and planning is required before modifying the hash-fragment bit number, and ensuring that the configuration on all nodes remains consistent, to avoid data distribution inconsistencies and access anomalies.

S204, indexing a hash slot corresponding to data to be stored in the Redis cluster based on the first hash segment to obtain a first hash slot.

In this embodiment, a hash slot corresponding to data to be stored is indexed in a dis cluster based on a first hash segment, so as to obtain a first hash slot. The high-order hash fragments are taken out from the hash value and can be used as the basis of the positions of the slots, so that the data can be dispersed into different slots to realize the fragmentation storage. The slicing storage can improve the utilization rate of the memory to the greatest extent, avoid the waste of memory resources, equalize the data quantity stored by each node through a good data slicing strategy, and reduce redundant storage as much as possible, thereby reducing the waste of memory resources.

S205, determining a data offset value of the second hash segment in the hash slot to obtain a first data offset value.

In this embodiment, the server determines the data offset value of the second hash segment in the hash slot, and obtains the first data offset value. Assuming that the slot position is slot, the data N needs to be stored in a Redis cluster with the slot position of the corresponding node as slot, a low-order hash segment of a hash value corresponding to the data N can be used as an identification bit in a Bitmap, and the storage state setting of the identification in the Bitmap is changed from 0 to 1 according to the requirement, so that the storage of the data N is indicated.

S206, determining a storage bitmap position of the data to be stored in the Redis cluster according to the first hash slot and the first data offset value, obtaining a first storage bitmap position, and changing the storage state of the first storage bitmap position.

In this embodiment, a storage Bitmap position of data to be stored in the Redis cluster, that is, an identification bit in the Bitmap, is determined according to the first hash slot and the first data offset value, so as to obtain the first storage Bitmap position, and the storage state of the first storage Bitmap position is changed to characterize the storage of the data to be stored. Bitmap is a mechanism by which dis records the storage state of each node, dividing each node into a number of storage Bitmap locations, one for each data storage state, where the purpose is to record the storage location of the data to be stored in the storage Bitmap and change its storage state for subsequent data reading and management.

In the embodiment, the hash value of the data is calculated, the hash slot is indexed in the Redis cluster by the hash value, the data offset value is determined by the hash value, the data is uniformly dispersed to different Redis cluster nodes, the data is stored in a slicing way, when a large amount of data is required to be stored, the data can be uniformly dispersed to a plurality of nodes by the slicing way, the memory occupation of a single node is avoided, the generation of a gap value is reduced, the memory utilization efficiency is improved, and the expandability and the load balance of the system are improved.

When whether certain data exist or not is required to be inquired, the hash value of the data is calculated through the same hash function, a high-order hash segment is taken out to serve as a basis of a slot position, data offset is determined through a low-order hash segment, so that corresponding nodes, the slot position and identification bits in a Bitmap are found, and then the storage state of the corresponding identification bits in the Bitmap is inquired.

In this embodiment, when the server receives the data query instruction, acquires data to be queried corresponding to the data query instruction, performs hash operation on the data to be queried using the same hash function, acquires a hash value corresponding to the data to be queried, acquires a second hash value, divides the second hash value using the same division rule, acquires a third hash segment and a fourth hash segment, indexes a hash slot corresponding to the data to be queried in the dis cluster based on the third hash segment, acquires the second hash slot, determines a data offset value of the fourth hash segment in the hash slot, acquires a second data offset value, determines a storage Bitmap position of the data to be queried in the dis cluster according to the second hash slot and the second data offset value, acquires a second storage Bitmap position, detects a storage state of the second storage Bitmap position, and determines whether the data to be queried is stored according to a storage state detection result.

In this embodiment, if the storage state of the second storage bitmap position is the first storage state, it is determined that the bitmap storage is not complete for the data to be queried, a first data query result is generated based on the first storage state, and the first data query result is output, which indicates that the data to be queried is not stored. If the storage state of the second storage bitmap position is the second storage state, determining that bitmap storage of the data to be queried is completed, generating a second data query result based on the second storage state, and outputting the second data query result to indicate that the data to be queried is stored.

In the embodiment, the application calculates the hash value of the data, indexes the hash slot in the Redis cluster by the hash value, determines the data offset value by the hash value, and then quickly inquires the storage state of the data by the hash slot and the data offset value, thereby improving the data inquiry efficiency of the Redis cluster.

In the above embodiment, the application discloses a fragment storage method of Redis clusters, which belongs to the technical field of big data and the financial risk production field. According to the method, data to be stored corresponding to the data storage instruction are obtained, hash operation is carried out on the data to be stored, a hash value corresponding to the data to be stored is obtained, a first hash value is obtained, the first hash value is divided, a first hash segment and a second hash segment are obtained, a hash slot corresponding to the data to be stored is indexed in a Redis cluster based on the first hash segment, a first hash slot is obtained, a data offset value of the second hash segment in the hash slot is determined, a first data offset value is obtained, a storage bitmap position of the data to be stored in the Redis cluster is determined according to the first hash slot and the first data offset value, a first storage bitmap position is obtained, and a storage state of the first storage bitmap position is changed. According to the application, the hash value of the data is calculated, the hash slot position is indexed in the Redis cluster by the hash value, the data offset value is determined by the hash value, then the data is uniformly dispersed on different Redis cluster nodes, so that the data is stored in a slicing way, when a large amount of data is required to be stored, the data can be uniformly dispersed on a plurality of nodes by the slicing way, the memory occupation of a single node is avoided, the generation of a gap value is reduced, the memory utilization efficiency is improved, and the expandability and the load balance of the system are improved. In addition, the application also calculates the hash value of the data, indexes the hash slot position in the Redis cluster by the hash value, determines the data offset value by the hash value, then quickly inquires the storage state of the data by the hash slot position and the data offset value, and improves the data inquiry efficiency of the Redis cluster.

In this embodiment, an electronic device (for example, a server shown in fig. 1) on which the fragment storage method of the Redis cluster operates may receive an instruction or acquire data through a wired connection manner or a wireless connection manner. It should be noted that the wireless connection may include, but is not limited to, 3G/4G connections, wiFi connections, bluetooth connections, wiMAX connections, zigbee connections, UWB (ultra wideband) connections, and other now known or later developed wireless connection means.

It should be emphasized that, to further ensure the privacy and security of the data to be stored, the data to be stored may also be stored in a node of a blockchain.

The blockchain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, encryption algorithm and the like. The Blockchain (Blockchain), which is essentially a decentralised database, is a string of data blocks that are generated by cryptographic means in association, each data block containing a batch of information of network transactions for verifying the validity of the information (anti-counterfeiting) and generating the next block. The blockchain may include a blockchain underlying platform, a platform product services layer, an application services layer, and the like.

Those skilled in the art will appreciate that implementing all or part of the processes of the methods of the embodiments described above may be accomplished by way of computer readable instructions, stored on a computer readable storage medium, which when executed may comprise processes of embodiments of the methods described above. The storage medium may be a nonvolatile storage medium such as a magnetic disk, an optical disk, a Read-Only Memory (ROM), or a random access Memory (Random Access Memory, RAM).

It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, these steps are not necessarily performed in order as indicated by the arrows. The steps are not strictly limited in order and may be performed in other orders, unless explicitly stated herein. Moreover, at least some of the steps in the flowcharts of the figures may include a plurality of sub-steps or stages that are not necessarily performed at the same time, but may be performed at different times, the order of their execution not necessarily being sequential, but may be performed in turn or alternately with other steps or at least a portion of the other steps or stages.

With further reference to fig. 3, as an implementation of the method shown in fig. 2, the present application provides an embodiment of a partitioned storage device of a dis cluster, where the embodiment of the device corresponds to the embodiment of the method shown in fig. 2, and the device may be specifically applied to various electronic devices.

As shown in fig. 3, a fragment storage apparatus 300 of a dis cluster according to the present embodiment includes:

the storage instruction receiving module 301 is configured to receive a data storage instruction, and obtain data to be stored corresponding to the data storage instruction;

the first hash operation module 302 is configured to perform hash operation on data to be stored, obtain a hash value corresponding to the data to be stored, and obtain a first hash value;

a first hash dividing module 303, configured to divide the first hash value to obtain a first hash segment and a second hash segment;

the first slot index module 304 is configured to index, in the dis cluster, a hash slot corresponding to data to be stored based on the first hash segment, to obtain a first hash slot;

a first data offset module 305, configured to determine a data offset value of the second hash segment in the hash slot, and obtain a first data offset value;

the data bitmap storage module 306 is configured to determine a storage bitmap position of data to be stored in the dis cluster according to the first hash slot and the first data offset value, obtain a first storage bitmap position, and change a storage state of the first storage bitmap position.

Further, the storage states include a first storage state and a second storage state, and the data bitmap storage module 306 specifically includes:

the bitmap position determining unit is used for determining the storage bitmap position of the data to be stored in the Redis cluster according to the traversing result of the data offset pointer to obtain a first storage bitmap position;

and the storage state changing unit is used for changing the storage state of the first storage bitmap position from the first storage state to the second storage state.

Further, the partitioned storage device of the dis cluster further includes:

Further, the storage state detection module specifically includes:

the first checking result unit is used for determining that the data to be queried is not complete in bitmap storage when the storage state of the second storage bitmap position is the first storage state;

the first query result output unit is used for generating a first data query result based on the first storage state and outputting the first data query result;

the second checking result unit is used for determining that the bitmap storage of the data to be queried is completed when the storage state of the second storage bitmap position is the second storage state;

and the second query result output unit is used for generating a second data query result based on the second storage state and outputting the second data query result.

Further, the first hash partitioning module 303 specifically includes:

an index length determining unit, configured to determine an index length of a hash slot in a Redis cluster;

the hash bit operation unit is used for carrying out bit operation on the first hash value based on the index length of the hash slot bit to obtain a first hash segment;

and the hash mask operation unit is used for carrying out mask operation on the first hash value based on the first hash fragment to obtain a second hash fragment.

Further, the hash bit operation unit specifically includes:

a start bit determination subunit, configured to determine a start bit of the first hash value;

and the bitwise right shifting subunit is used for starting from the initial bit of the first hash value, performing bitwise right shifting operation on the first hash value until the right shifting length is equal to the index length of the hash slot bit, and obtaining the first hash segment.

In the above embodiment, the application discloses a fragment storage device of a Redis cluster, which belongs to the technical field of big data and the financial risk production field. According to the method, data to be stored corresponding to the data storage instruction are obtained, hash operation is carried out on the data to be stored, a hash value corresponding to the data to be stored is obtained, a first hash value is obtained, the first hash value is divided, a first hash segment and a second hash segment are obtained, a hash slot corresponding to the data to be stored is indexed in a Redis cluster based on the first hash segment, a first hash slot is obtained, a data offset value of the second hash segment in the hash slot is determined, a first data offset value is obtained, a storage bitmap position of the data to be stored in the Redis cluster is determined according to the first hash slot and the first data offset value, a first storage bitmap position is obtained, and a storage state of the first storage bitmap position is changed. According to the application, the hash value of the data is calculated, the hash slot position is indexed in the Redis cluster by the hash value, the data offset value is determined by the hash value, then the data is uniformly dispersed on different Redis cluster nodes, so that the data is stored in a slicing way, when a large amount of data is required to be stored, the data can be uniformly dispersed on a plurality of nodes by the slicing way, the memory occupation of a single node is avoided, the generation of a gap value is reduced, the memory utilization efficiency is improved, and the expandability and the load balance of the system are improved. In addition, the application also calculates the hash value of the data, indexes the hash slot position in the Redis cluster by the hash value, determines the data offset value by the hash value, then quickly inquires the storage state of the data by the hash slot position and the data offset value, and improves the data inquiry efficiency of the Redis cluster.

In order to solve the technical problems, the embodiment of the application also provides computer equipment. Referring specifically to fig. 4, fig. 4 is a basic structural block diagram of a computer device according to the present embodiment.

The computer device 4 comprises a memory 41, a processor 42, a network interface 43 communicatively connected to each other via a system bus. It should be noted that only computer device 4 having components 41-43 is shown in the figures, but it should be understood that not all of the illustrated components are required to be implemented and that more or fewer components may be implemented instead. It will be appreciated by those skilled in the art that the computer device herein is a device capable of automatically performing numerical calculations and/or information processing in accordance with predetermined or stored instructions, the hardware of which includes, but is not limited to, microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASICs), programmable gate arrays (fields-Programmable Gate Array, FPGAs), digital processors (Digital Signal Processor, DSPs), embedded devices, etc.

The computer equipment can be a desktop computer, a notebook computer, a palm computer, a cloud server and other computing equipment. The computer equipment can perform man-machine interaction with a user through a keyboard, a mouse, a remote controller, a touch pad or voice control equipment and the like.

The memory 41 includes at least one type of readable storage medium including flash memory, hard disk, multimedia card, card memory (e.g., SD or DX memory, etc.), random Access Memory (RAM), static Random Access Memory (SRAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), programmable Read Only Memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the storage 41 may be an internal storage unit of the computer device 4, such as a hard disk or a memory of the computer device 4. In other embodiments, the memory 41 may also be an external storage device of the computer device 4, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash Card (Flash Card) or the like, which are provided on the computer device 4. Of course, the memory 41 may also comprise both an internal memory unit of the computer device 4 and an external memory device. In this embodiment, the memory 41 is generally used to store an operating system and various application software installed on the computer device 4, such as computer readable instructions of a sharded storage method of a Redis cluster. Further, the memory 41 may be used to temporarily store various types of data that have been output or are to be output.

The processor 42 may be a central processing unit (Central Processing Unit, CPU), controller, microcontroller, microprocessor, or other data processing chip in some embodiments. The processor 42 is typically used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is configured to execute computer readable instructions stored in the memory 41 or process data, for example, execute computer readable instructions of a fragment storing method of the dis cluster.

The network interface 43 may comprise a wireless network interface or a wired network interface, which network interface 43 is typically used for establishing a communication connection between the computer device 4 and other electronic devices.

In the above embodiment, the application discloses a computer device, which belongs to the technical field of big data and the field of financial production. According to the method, data to be stored corresponding to the data storage instruction are obtained, hash operation is carried out on the data to be stored, a hash value corresponding to the data to be stored is obtained, a first hash value is obtained, the first hash value is divided, a first hash segment and a second hash segment are obtained, a hash slot corresponding to the data to be stored is indexed in a Redis cluster based on the first hash segment, a first hash slot is obtained, a data offset value of the second hash segment in the hash slot is determined, a first data offset value is obtained, a storage bitmap position of the data to be stored in the Redis cluster is determined according to the first hash slot and the first data offset value, a first storage bitmap position is obtained, and a storage state of the first storage bitmap position is changed. According to the application, the hash value of the data is calculated, the hash slot position is indexed in the Redis cluster by the hash value, the data offset value is determined by the hash value, then the data is uniformly dispersed on different Redis cluster nodes, so that the data is stored in a slicing way, when a large amount of data is required to be stored, the data can be uniformly dispersed on a plurality of nodes by the slicing way, the memory occupation of a single node is avoided, the generation of a gap value is reduced, the memory utilization efficiency is improved, and the expandability and the load balance of the system are improved. In addition, the application also calculates the hash value of the data, indexes the hash slot position in the Redis cluster by the hash value, determines the data offset value by the hash value, then quickly inquires the storage state of the data by the hash slot position and the data offset value, and improves the data inquiry efficiency of the Redis cluster.

The present application also provides another embodiment, namely, a computer readable storage medium storing computer readable instructions executable by at least one processor to cause the at least one processor to perform the steps of the method for storing a Redis cluster as described above.

In the above embodiments, the present application discloses a computer readable storage medium, which belongs to the technical field of big data and the field of financial risk production. According to the method, data to be stored corresponding to the data storage instruction are obtained, hash operation is carried out on the data to be stored, a hash value corresponding to the data to be stored is obtained, a first hash value is obtained, the first hash value is divided, a first hash segment and a second hash segment are obtained, a hash slot corresponding to the data to be stored is indexed in a Redis cluster based on the first hash segment, a first hash slot is obtained, a data offset value of the second hash segment in the hash slot is determined, a first data offset value is obtained, a storage bitmap position of the data to be stored in the Redis cluster is determined according to the first hash slot and the first data offset value, a first storage bitmap position is obtained, and a storage state of the first storage bitmap position is changed. According to the application, the hash value of the data is calculated, the hash slot position is indexed in the Redis cluster by the hash value, the data offset value is determined by the hash value, then the data is uniformly dispersed on different Redis cluster nodes, so that the data is stored in a slicing way, when a large amount of data is required to be stored, the data can be uniformly dispersed on a plurality of nodes by the slicing way, the memory occupation of a single node is avoided, the generation of a gap value is reduced, the memory utilization efficiency is improved, and the expandability and the load balance of the system are improved. In addition, the application also calculates the hash value of the data, indexes the hash slot position in the Redis cluster by the hash value, determines the data offset value by the hash value, then quickly inquires the storage state of the data by the hash slot position and the data offset value, and improves the data inquiry efficiency of the Redis cluster.

From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) comprising instructions for causing a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to perform the method according to the embodiments of the present application.

The application is operational with numerous general purpose or special purpose computer system environments or configurations. For example: personal computers, server computers, hand-held or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The application may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The application may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

It is apparent that the above-described embodiments are only some embodiments of the present application, but not all embodiments, and the preferred embodiments of the present application are shown in the drawings, which do not limit the scope of the patent claims. This application may be embodied in many different forms, but rather, embodiments are provided in order to provide a thorough and complete understanding of the present disclosure. Although the application has been described in detail with reference to the foregoing embodiments, it will be apparent to those skilled in the art that modifications may be made to the embodiments described in the foregoing description, or equivalents may be substituted for elements thereof. All equivalent structures made by the content of the specification and the drawings of the application are directly or indirectly applied to other related technical fields, and are also within the scope of the application.

Claims

1. The method for storing the Redis cluster in the partitioned mode is characterized by comprising the following steps of:

carrying out hash operation on the data to be stored, and obtaining a hash value corresponding to the data to be stored to obtain a first hash value;

indexing a hash slot corresponding to the data to be stored in the Redis cluster based on the first hash segment to obtain a first hash slot;

and determining a storage bitmap position of the data to be stored in the Redis cluster according to the first hash slot position and the first data offset value, obtaining a first storage bitmap position, and changing the storage state of the first storage bitmap position.

2. The method for storing the dis cluster in a fragmented manner according to claim 1, wherein the storage states include a first storage state and a second storage state, the determining the storage bitmap position of the data to be stored in the dis cluster according to the first hash slot and the first data offset value, to obtain a first storage bitmap position, and changing the storage state of the first storage bitmap position specifically includes:

Determining a storage bitmap position of the data to be stored in the Redis cluster according to the traversing result of the data offset pointer, and obtaining a first storage bitmap position;

and changing the storage state of the first storage bitmap position from the first storage state to the second storage state.

3. The method for storing the Redis cluster in a partitioned manner according to claim 2, further comprising, after said determining the bitmap position of the data to be stored according to the first hash slot and the first data offset value, and changing the storage state of the bitmap position:

performing hash operation on the data to be queried to obtain a hash value corresponding to the data to be queried, and obtaining a second hash value;

Determining a storage bitmap position of the data to be queried in the Redis cluster according to the second hash slot position and the second data offset value to obtain a second storage bitmap position;

4. The method for storing the Redis cluster in a partitioned manner according to claim 3, wherein the detecting the storage state of the second storage bitmap position and determining whether the bitmap storage of the data to be queried is completed according to the detection result of the storage state specifically includes:

5. The method for storing the dis cluster according to any one of claims 1 to 4, wherein the dividing the first hash value to obtain a first hash segment and a second hash segment specifically includes:

Determining the index length of a hash slot in the Redis cluster;

performing bit operation on the first hash value based on the index length of the hash slot to obtain the first hash segment;

and carrying out mask operation on the first hash value based on the first hash fragment to obtain the second hash fragment.

6. The method for storing the Redis cluster in a partitioned manner according to claim 5, wherein the bit operation is performed on the first hash value based on the index length of the hash slot to obtain the first hash segment, specifically including:

determining a start bit of the first hash value;

and starting from the start bit of the first hash value, performing a bit-by-bit right shift operation on the first hash value until the right shift length is equal to the index length of the hash slot, and obtaining the first hash segment.

7. A tiled memory device of a dis cluster, comprising:

the storage instruction receiving module is used for receiving a data storage instruction and acquiring data to be stored corresponding to the data storage instruction;

the first slot index module is used for indexing the hash slots corresponding to the data to be stored in the Redis cluster based on the first hash segments to obtain first hash slots;

and the data bitmap storage module is used for determining the storage bitmap position of the data to be stored in the Redis cluster according to the first hash slot bit and the first data offset value, obtaining a first storage bitmap position and changing the storage state of the first storage bitmap position.

8. The tiled memory arrangement of Redis clusters according to claim 7, wherein the tiled memory arrangement of Redis clusters further comprises:

the second hash operation module is used for carrying out hash operation on the data to be queried to obtain a hash value corresponding to the data to be queried to obtain a second hash value;

9. A computer device comprising a memory and a processor, the memory having stored therein computer readable instructions which when executed by the processor implement the steps of the method for storing fragments of a dis cluster as claimed in any one of claims 1 to 6.

10. A computer readable storage medium having stored thereon computer readable instructions which when executed by a processor implement the steps of the method for sharding a dis cluster according to any of claims 1 to 6.