CN111078132A - Data uniform caching method, device, terminal and storage medium based on Redis cluster - Google Patents
Data uniform caching method, device, terminal and storage medium based on Redis cluster Download PDFInfo
- Publication number
- CN111078132A CN111078132A CN201910977353.XA CN201910977353A CN111078132A CN 111078132 A CN111078132 A CN 111078132A CN 201910977353 A CN201910977353 A CN 201910977353A CN 111078132 A CN111078132 A CN 111078132A
- Authority
- CN
- China
- Prior art keywords
- data
- storage node
- set array
- index
- redis cluster
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
- G06F3/0607—Improving or facilitating administration, e.g. storage management by facilitating the process of upgrading existing storage systems, e.g. for improving compatibility between host and storage device
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0656—Data buffering arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data uniform caching method based on a Redis cluster, which comprises the following steps: initializing a set array; associating the index of the set array with a storage node in a Redis cluster; when a cache instruction of data is received, calculating a hash value of the data; performing logic AND operation on the hash value and the maximum int-type data; performing complementation operation on the result after the logical AND operation and the length of the set array to obtain a key value; matching a target index which is the same as the key value from the set array; and storing the data corresponding to the key value in a storage node corresponding to the target index. The invention also provides a data uniform cache device based on the Redis cluster, a terminal and a storage medium. The method and the device can uniformly cache the data in the Redis cluster, and avoid the situation that the memory of one storage node in the Redis cluster is used up and the memories of other storage nodes are in a large amount of idle states.
Description
Technical Field
The invention relates to the technical field of data caching, in particular to a data uniform caching method, device, terminal and storage medium based on Redis cluster.
Background
With the rapid development of information technology, the information amount of modern society is increased explosively, the volume of application data is increased rapidly in the big data era today, and the traditional data architecture cannot meet the requirements of high concurrency, high availability, high-quality response time and the like of large-scale internet application. A range of solutions, such as NoSQL, have thus been created, in which a remote data service (Redis), one of the top-quality options, is increasingly adopted by large enterprises as a complement to their data architecture.
The Redis cluster cache data is obtained by performing a consistent hash algorithm on each key value to obtain the node into which the data should be put, the consistent hash algorithm organizes the whole hash value space into a virtual ring, the whole space is organized in the clockwise direction, and then the hash is performed on each master node. This allows the position of each node on its hash ring to be determined. However, in this way, when a certain key has a large amount of data, the large amount of data of the same key is stored in one master node, the memory of the master node is used up, and the memories of other master nodes are still largely free. The used master node of the memory exceeds the self bearing capacity, performance is reduced, even data is unavailable, and meanwhile, a large number of idle master nodes of the memory cannot serve other data requests, so that resource waste is caused.
Therefore, it is necessary to provide a data uniform caching scheme based on the Redis cluster, so that the memory usage of each master node in the Redis cluster is uniform.
Disclosure of Invention
In view of the foregoing, it is necessary to provide a data uniform caching method, device, terminal and storage medium based on a Redis cluster, which can uniformly cache data in the Redis cluster.
The first aspect of the present invention provides a data uniform caching method based on a Redis cluster, where the method includes:
initializing a set array;
associating the index of the set array with a storage node in a Redis cluster;
when a cache instruction of data is received, calculating a hash value of the data;
performing logic AND operation on the hash value and the maximum int-type data;
performing complementation operation on the result after the logical AND operation and the length of the set array to obtain a key value;
matching a target index which is the same as the key value from the set array;
and storing the data corresponding to the key value in a storage node corresponding to the target index.
According to a preferred embodiment of the present invention, the initializing set array includes:
acquiring the number of storage nodes in the Redis cluster;
determining the length range of the set array according to the number;
acquiring the performance of a storage node in the Redis cluster;
when the performance is greater than a preset performance threshold, determining the length of the set array as the maximum value in the length range;
when the performance is smaller than a preset performance threshold, determining the length of the set array as a middle value in the length range, wherein the middle value refers to any value in the middle area of the length range.
According to a preferred embodiment of the present invention, the associating the index of the set array with a storage node in a Redis cluster includes:
calculating a first hash value for each index of the set array;
calculating a second hash value of each storage node in the Redis cluster;
matching a target first hash value identical to the second hash value;
and associating the index of the set array corresponding to the target first hash value with the storage node corresponding to the second hash value.
According to a preferred embodiment of the present invention, the calculating the second hash value of each storage node in the Redis cluster includes:
acquiring an identifier of each storage node in the Redis cluster, wherein the identifier is an IP address of the storage node or a host name of the storage node;
and calculating a second hash value according to the identification.
According to a preferred embodiment of the present invention, the calculating the hash value of the data when the cache instruction of the data is received includes:
when a cache instruction of data is received, reading the key values of each row of data in the data line by line;
and calculating the hash value of the key value of each row of data.
According to a preferred embodiment of the present invention, after the logically anding the hash value with the maximum int-type data, the method further comprises:
judging whether the result after logical AND operation is-2147483648;
when the result after the logical and operation is-2147483648, the result is determined to be 2147483647;
when the result after the logical AND operation is not-2147483648, the result is determined to be the absolute value after the logical AND operation.
According to a preferred embodiment of the present invention, the method may further include:
when a reading instruction of the data is received, acquiring all indexes of the set array;
acquiring a storage node corresponding to each index in the Redis cluster;
sequentially reading data in the storage nodes corresponding to each index according to the sequence of the indexes;
and connecting the read data and displaying.
A second aspect of the present invention provides a data uniform caching apparatus based on a Redis cluster, where the apparatus includes:
the initialization module is used for initializing a set array;
the index association module is used for associating the index of the set array with a storage node in the Redis cluster;
the hash calculation module is used for calculating the hash value of the data when a cache instruction of the data is received;
the logical operation module is used for carrying out logical AND operation on the hash value and the maximum int-type data;
the complementation operation module is used for carrying out complementation operation on the result after the logical AND operation and the length of the set array to obtain a key value;
the index matching module is used for matching a target index which is the same as the key value from the set array;
and the data storage module is used for storing the data corresponding to the key value in the storage node corresponding to the target index.
A third aspect of the present invention provides a terminal, which includes a processor, and the processor is configured to implement the method for uniform data caching based on Redis cluster when executing a computer program stored in a memory.
A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the Redis cluster-based data uniform caching method.
In summary, according to the method, the device, the terminal and the storage medium for uniformly caching data based on the Redis cluster, the set array is initialized, the index of the set array and the storage node in the Redis cluster are associated, for the data to be cached, the hash value of each data is calculated, each hash value and the maximum int-type data are subjected to logical and operation, and then subjected to remainder operation with the length of the set data to obtain the index of which the key value exactly corresponds to the set array, so that the target index of the set array, which is the same as each key value, can be matched, the target storage node corresponding to the target index is further determined, and finally the data corresponding to the key value is stored in the target storage node. On the basis of ensuring that the cache data cannot be repeated, the uniform distribution of large data volume on the Redis cluster is also ensured. In addition, the hash value is calculated by taking the key (key) of the data as an object, the data with the same key can be stored in the same storage node, and the data with different keys can be stored in different storage nodes; and the length of the initialized set array is comprehensively considered by combining the scale of the Redis cluster and the performance of the storage node, so that the data distribution of the Redis cluster is uniform, and the data writing and reading speed is not influenced.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a flowchart of a data uniform caching method based on a Redis cluster according to an embodiment of the present invention.
Fig. 2 is a structural diagram of a data uniform cache device based on a Redis cluster according to a second embodiment of the present invention.
Fig. 3 is a schematic structural diagram of a terminal according to a third embodiment of the present invention.
The following detailed description will further illustrate the invention in conjunction with the above-described figures.
Detailed Description
In order that the above objects, features and advantages of the present invention can be more clearly understood, a detailed description of the present invention will be given below with reference to the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and features of the embodiments may be combined with each other without conflict.
In the following description, numerous specific details are set forth to provide a thorough understanding of the present invention, and the described embodiments are merely a subset of the embodiments of the present invention, rather than a complete embodiment. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.
Example one
Fig. 1 is a flowchart of a data uniform caching method based on a Redis cluster according to an embodiment of the present invention.
In this embodiment, the method for uniformly caching data based on the Redis cluster may be applied to a terminal, and for a terminal that needs to perform uniform caching of data based on the Redis cluster, a function of uniformly caching data based on the Redis cluster provided by the method of the present invention may be directly integrated on the terminal, or may be operated in the terminal in a form of software development Kit (SKD).
As shown in fig. 1, the data uniform caching method based on the Redis cluster specifically includes the following steps, and according to different requirements, the order of the steps in the flowchart may be changed, and some of the steps may be omitted.
S11, initializing the set array.
Redis may initiate service instances at different physical machines or different ports on the same physical machine as storage nodes in a consistent hash ring. Each storage node is set according to the actual memory of the physical machine. Consistent hashing is the mapping of data and virtual nodes all to 0-231And meanwhile, the numerical space is connected end to be used as a hash ring. The virtual node is subjected to hash operation and is mapped to a number on the hash ring, and the corresponding relation between the number and the virtual node is stored to be the hash mapping table.
During initialization, except for initializing a set array, a Master node Master can be set for the Redis cluster to manage metadata of the Redis cluster, and a consistent hash mapping table is maintained, for example, an association relationship between the set array and a storage node is maintained, other service instances started by the Redis can be used as Slave nodes Slave of data storage to be responsible for data management and storage, maintain the state of the node instance, and perform data and state interaction on the Master node.
In a preferred embodiment, the initialization set array includes:
acquiring the number of storage nodes in the Redis cluster;
determining the length range of the set array according to the number;
acquiring the performance of a storage node in the Redis cluster;
when the performance is greater than a preset performance threshold, determining the length of the set array as the maximum value in the length range;
when the performance is smaller than a preset performance threshold, determining the length of the set array as a middle value in the length range, wherein the middle value refers to any value in the middle area of the length range.
In this alternative embodiment, the longer the length of the initialization set array, the more uniform the subsequent storage of data in the Redis cluster will be. However, if the length of the set array is too long and the performance of the storage nodes is low, the speed of writing and reading will be slower, although the Redis cluster data distribution will be made more uniform. The length of the initialization set array thus needs to be considered in combination with the size of the Redis cluster and the performance of the storage nodes.
Assuming that the number of storage nodes in the Redis cluster is N and the length of the set array is L, the number M of indexes of the set array possibly allocated to each storage node is equal to L/N. The range of M can be N/2-2N, and the range of L is N2/2-2N2. If the performance of the storage node is better (greater than the preset performance threshold), M may be a large point, the length of the corresponding set array may be longer, and the maximum value 2N in the range may be taken2. If the performance of the storage node is poor (less than the preset performance threshold), M can be a small point, the length of the corresponding set array is short, and the middle value N in the range can be taken2。
Wherein the intermediate value does not refer to the value at the middle of the length range, but refers to any value in the middle region of the length range. The intermediate region may be a predetermined region between 1/4-3/4 of the length range.
For example, assuming that the number N of storage nodes in the Redis cluster is 6, and the number M of indexes of the set array allocated to each storage node is 6, the length L of the set array is 6 × 6 — 36.
S12, associating the index of the set array with the storage node in the Redis cluster.
In this embodiment, after initializing the set array, an association relationship is established between the index of the set array and the storage node in the Redis cluster, and the association relationship is stored.
In a preferred embodiment, the associating the index of the set array with a storage node in a Redis cluster includes:
calculating a first hash value for each index of the set array;
calculating a second hash value of each storage node in the Redis cluster;
matching a target first hash value identical to the second hash value;
and associating the index of the set array corresponding to the target first hash value with the storage node corresponding to the second hash value.
In this alternative embodiment, the hash value of the index of the set array may be calculated from the name of the index of the set array. Through hash computation, each index of the set array can be associated with one storage node in the Redis cluster.
The Hash value may be computed using a Secure Hash Algorithm (SHA).
For example, assuming that the first hash value of the 1 st index S0 of the set array is 1, the first hash value of the 2 nd index S1 is 4, the second hash value of the 1 st storage node in the Redis cluster is 1, and the second hash value of the 4 th storage node is 4, the 1 st index S0 of the set array is associated with the 1 st storage node, and the 2 nd index S1 of the set array is associated with the 4 th storage node.
In a preferred embodiment, the calculating the second hash value of each storage node in the Redis cluster includes:
acquiring an identifier of each storage node in the Redis cluster, wherein the identifier is an IP address of the storage node or a host name of the storage node;
and calculating a second hash value according to the identification.
The IP addresses or host names of different storage nodes are different, and the second hash values obtained through calculation are different.
S13, when a cache instruction of data is received, calculating a hash value of the data.
In this embodiment, when a cache instruction of data is triggered, that is, the cache instruction of the data is considered to be received, the hash value of each piece of data in the data to be stored is calculated. The hash values of different data are different, and the hash values of the same data are the same.
In a preferred embodiment, the calculating the hash value of the data when the cache instruction of the data is received includes:
when a cache instruction of data is received, reading the key values of each row of data in the data line by line;
and calculating the hash value of the key value of each row of data.
Since the Redis cluster stores data in key-value, the hash value is calculated by using a key (key) of the data as an object, the data with the same key can be stored in the same storage node, and the data with different keys can be stored in different storage nodes.
And S14, performing logical AND operation on the hash value and the maximum int-type data.
Since the calculated hash value of the data may be a negative number, after the hash value of the data is calculated, the hash value and the maximum int-type data are logically and-operated, so that a non-negative int-type value can be obtained.
The logic and budget for Java is prior art and the present invention is not described in detail herein.
In a preferred embodiment, after the logically anding the hash value with the maximum int-type data, the method further includes:
judging whether the result after logical AND operation is-2147483648;
when the result after the logical and operation is-2147483648, the result is determined to be 2147483647;
when the result after the logical AND operation is not-2147483648, the result is determined to be the absolute value after the logical AND operation.
When int type data in Java occupies 32 bits, the maximum value can be assigned as: 2147483647 expressed in hexadecimal terms: 0x7fffffff, the minimum can be assigned as: -2147483648 expressed in hexadecimal terms: 0x8 fffffff. I.e., int-type data for Java, ranges from-2147483648 and 2147483647. It should be understood that: the highest bit of 7 in binary form in 0x7fffffff is 0, if pair 2147483647+1, the output is-2147483648, and the highest bit of 8 in binary form in 0x8fffffff is the sign bit.
Since the maximum value of int-type data in Java is 2147483647, and the absolute value of-2147483648 is 2147483648, which is 1 greater than the maximum value 2147483647, the data obtained by directly taking the absolute value of-2147483648 exceeds the range of the maximum value 2147483647 of the system, and the system will report an error. To avoid system errors after logical and, the result obtained after logical and of each hash value and the maximum int-type data is absolute except that the result is-2147483648 is modified to 2147483647.
And S15, performing complementation operation on the result after the logical AND operation and the length of the set array to obtain a key value.
And the length of the set array is the number of indexes in the set array. For example, if the number of indexes in the set array is 36, the length of the set array is 36.
In Java,% is used as an operator of an arithmetic operation (modulo operation), and the sign of the result after the arithmetic operation always matches the sign of the dividend. Therefore, the key value obtained by performing the logical and operation on the result and the length of the set array just corresponds to the index of the set array.
The modulo operation of Java is prior art and the present invention will not be described in detail herein.
Illustratively, assuming that the length of the set array is 36, the indexes of the set array are 0, 1, … …, 35. The key value obtained by performing the logical and operation on the result and the length of the set array and performing the complementation operation is between 0 and 35, so that the key value obtained by the operation can be certainly the same as one index of the set array, and thus, the index of which set array the data corresponding to the key value should correspond to, that is, the storage node to which the data should be stored can be determined. The same data is ensured to correspond to the same key, and repeated caching of the data is avoided.
And S16, matching out the target index which is the same as the key value from the set array.
In this embodiment, after the key value corresponding to each piece of data is obtained through calculation, since the key value may be definitely the same as one index of the set array, an index in the set array, which is the same as the key value, is used as a target index, which is convenient for matching a target storage node corresponding to the target index in the subsequent process, and the data is stored in the target storage node.
For example, if the key value corresponding to the first data is 6, the index S6 in the index of the set array is the target index of the first data, and the storage node M3 corresponding to the target index S6 is the target storage node; if the key value corresponding to the second data is 0, the index S0 in the index of the set array is the target index of the second data, and the storage node M1 corresponding to the target index S0 is the target storage node.
And S17, storing the data corresponding to the key value in the storage node corresponding to the target index.
In this embodiment, when the target index is determined, a target storage node corresponding to the target index may be determined according to the association relationship between the set array and the storage node, and the data corresponding to the key value is stored in the target storage node.
As in the above example, the first data is stored in storage node M3 and the second data is stored in storage node M1.
In a preferred embodiment, the method may further include:
when a reading instruction of the data is received, acquiring all indexes of the set array;
acquiring a storage node corresponding to each index in the Redis cluster;
sequentially reading data in the storage nodes corresponding to each index according to the sequence of the indexes;
and connecting the read data and displaying.
For example, suppose that the storage nodes of the Redis cluster are M1, M2, M3, M4, M5 and M6, respectively, the set array length is 12, and the indexes are S0, S1, S2, S3, S4, S5, S6, S7, S8, S9, S10 and S11, respectively. Wherein all indexes of the set array are acquired when a read instruction of the data is received, and then obtaining a storage node M1 corresponding to the index S0, a storage node M3 corresponding to the index S1, storage node M4 corresponding to index S2, storage node M2 corresponding to index S3, storage node M6 corresponding to index S4, storage node M5 corresponding to index S5, storage node M3 corresponding to index S6, storage node M1 corresponding to index S7, storage node M6 corresponding to index S8, storage node M4 corresponding to index S9, storage node M2 corresponding to index S10, the storage node M5 corresponding to the index S11 reads the data in the storage nodes M1, M3, M4, M2, M6, M5, M3, M1, M6, M4, M2, and M5, respectively, in the order of 0 to 11 of the index, and finally links the read data and displays the linked data as the final result.
Specifically, each time data in one storage node is read, the data is sequentially read from top to bottom. Preferably, after the data in the storage node is read, the data in the storage node is marked, and when the storage node is read next time, the pointer is directly positioned to the next line of the mark, so that the required data can be read.
In summary, according to the data uniform caching method based on the Redis cluster, the set array is initialized, the index of the set array and the storage node in the Redis cluster are associated, for the data to be cached, the hash value of each data is calculated, each hash value and the maximum int-type data are subjected to logical and operation, and then the logical and operation and the length of the set data are subjected to the complementation operation to obtain the index of the set array with the right key value, so that the target index of the set array, which is the same as each key value, can be matched, the target storage node corresponding to the target index is further determined, and finally the data corresponding to the key value is stored in the target storage node. On the basis of ensuring that the cache data cannot be repeated, the uniform distribution of large data volume on the Redis cluster is also ensured.
In addition, the hash value is calculated by taking the key (key) of the data as an object, the data with the same key can be stored in the same storage node, and the data with different keys can be stored in different storage nodes; and the length of the initialized set array is comprehensively considered by combining the scale of the Redis cluster and the performance of the storage node, so that the data distribution of the Redis cluster is uniform, and the data writing and reading speed is not influenced.
Example two
Fig. 2 is a structural diagram of a data uniform cache device based on a Redis cluster according to a second embodiment of the present invention.
In some embodiments, the data uniform cache apparatus 20 based on the Redis cluster may include a plurality of functional modules composed of program code segments. The program codes of the respective program segments in the data uniform cache apparatus 20 based on the Redis cluster may be stored in the memory of the terminal and executed by the at least one processor to perform (see fig. 1 for details) the function of data uniform cache based on the Redis cluster.
In this embodiment, the data uniform cache device 20 based on the Redis cluster may be divided into a plurality of functional modules according to the functions executed by the device. The functional module may include: the system comprises an initialization module 201, an index association module 202, a hash calculation module 203, a logical operation module 204, a result judgment module 205, a remainder operation module 206, an index matching module 207, a data storage module 208, an index acquisition module 209 and a data reading module 210. The module referred to herein is a series of computer program segments capable of being executed by at least one processor and capable of performing a fixed function and is stored in memory. In the present embodiment, the functions of the modules will be described in detail in the following embodiments.
An initialization module 201 for initializing a set array.
Redis may initiate service instances at different physical machines or different ports on the same physical machine as storage nodes in a consistent hash ring. Each storage node is set according to the actual memory of the physical machine. Consistent hashing is the mapping of data and virtual nodes all to 0-231And meanwhile, the numerical space is connected end to be used as a hash ring. The virtual node is subjected to hash operation and is mapped to a number on the hash ring, and the corresponding relation between the number and the virtual node is stored to be the hash mapping table.
During initialization, except for initializing a set array, a Master node Master can be set for the Redis cluster to manage metadata of the Redis cluster, and a consistent hash mapping table is maintained, for example, an association relationship between the set array and a storage node is maintained, other service instances started by the Redis can be used as Slave nodes Slave of data storage to be responsible for data management and storage, maintain the state of the node instance, and perform data and state interaction on the Master node.
In a preferred embodiment, the initialization module 201 initializes the set array including:
acquiring the number of storage nodes in the Redis cluster;
determining the length range of the set array according to the number;
acquiring the performance of a storage node in the Redis cluster;
when the performance is greater than a preset performance threshold, determining the length of the set array as the maximum value in the length range;
when the performance is smaller than a preset performance threshold, determining the length of the set array as a middle value in the length range, wherein the middle value refers to any value in the middle area of the length range.
In this alternative embodiment, the longer the length of the initialization set array, the more uniform the subsequent storage of data in the Redis cluster will be. However, if the length of the set array is too long and the performance of the storage nodes is low, the speed of writing and reading will be slower, although the Redis cluster data distribution will be made more uniform. The length of the initialization set array thus needs to be considered in combination with the size of the Redis cluster and the performance of the storage nodes.
Assuming that the number of storage nodes in the Redis cluster is N and the length of the set array is L, the number M of indexes of the set array possibly allocated to each storage node is equal to L/N. The range of M can be N/2-2N, and the range of L is N2/2-2N2. If the performance of the storage node is better (greater than the preset performance threshold), M may be a large point, the length of the corresponding set array may be longer, and the maximum value 2N in the range may be taken2. If the performance of the storage node is poor (less than the preset performance threshold), M can be a small point, the length of the corresponding set array is short, and the middle value N in the range can be taken2。
Wherein the intermediate value does not refer to the value at the middle of the length range, but refers to any value in the middle region of the length range. The intermediate region may be a predetermined region between 1/4-3/4 of the length range.
For example, assuming that the number N of storage nodes in the Redis cluster is 6, and the number M of indexes of the set array allocated to each storage node is 6, the length L of the set array is 6 × 6 — 36.
An index associating module 202, configured to associate the index of the set array with a storage node in the Redis cluster.
In this embodiment, after initializing the set array, an association relationship is established between the index of the set array and the storage node in the Redis cluster, and the association relationship is stored.
In a preferred embodiment, the index associating module 202 associates the index of the set array with a storage node in a Redis cluster, including:
calculating a first hash value for each index of the set array;
calculating a second hash value of each storage node in the Redis cluster;
matching a target first hash value identical to the second hash value;
and associating the index of the set array corresponding to the target first hash value with the storage node corresponding to the second hash value.
In this alternative embodiment, the hash value of the index of the set array may be calculated from the name of the index of the set array. Through hash computation, each index of the set array can be associated with one storage node in the Redis cluster.
The Hash value may be computed using a Secure Hash Algorithm (SHA).
For example, assuming that the first hash value of the 1 st index S0 of the set array is 1, the first hash value of the 2 nd index S1 is 4, the second hash value of the 1 st storage node in the Redis cluster is 1, and the second hash value of the 4 th storage node is 4, the 1 st index S0 of the set array is associated with the 1 st storage node, and the 2 nd index S1 of the set array is associated with the 4 th storage node.
In a preferred embodiment, the calculating the second hash value of each storage node in the Redis cluster includes:
acquiring an identifier of each storage node in the Redis cluster, wherein the identifier is an IP address of the storage node or a host name of the storage node;
and calculating a second hash value according to the identification.
The IP addresses or host names of different storage nodes are different, and the second hash values obtained through calculation are different.
The hash calculation module 203 is configured to calculate a hash value of the data when a cache instruction of the data is received.
In this embodiment, when a cache instruction of data is triggered, that is, the cache instruction of the data is considered to be received, the hash value of each piece of data in the data to be stored is calculated. The hash values of different data are different, and the hash values of the same data are the same.
In a preferred embodiment, the calculating the hash value of the data by the hash calculation module 203 when the cache instruction of the data is received includes:
reading the key values of each row of data line by line;
and calculating the hash value of the key value of each row of data.
Since the Redis cluster stores data in key-value, the hash value is calculated by using a key (key) of the data as an object, the data with the same key can be stored in the same storage node, and the data with different keys can be stored in different storage nodes.
The logical operation module 204 is configured to perform a logical and operation on the hash value and the maximum int-type data.
Since the calculated hash value of the data may be a negative number, after the hash value of the data is calculated, the hash value and the maximum int-type data are logically and-operated, so that a non-negative int-type value can be obtained.
The logic and budget for Java is prior art and the present invention is not described in detail herein.
In a preferred embodiment, after the logically anding the hash value with the maximum int-type data, the apparatus further includes: a result determination module 205, configured to:
judging whether the result after logical AND operation is-2147483648;
when the result after the logical and operation is-2147483648, the result is determined to be 2147483647;
when the result after the logical AND operation is not-2147483648, the result is determined to be the absolute value after the logical AND operation.
When int type data in Java occupies 32 bits, the maximum value can be assigned as: 2147483647 expressed in hexadecimal terms: 0x7fffffff, the minimum can be assigned as: -2147483648 expressed in hexadecimal terms: 0x8 fffffff. I.e., int-type data for Java, ranges from-2147483648 and 2147483647. It should be understood that: the highest bit of 7 in binary form in 0x7fffffff is 0, if pair 2147483647+1, the output is-2147483648, and the highest bit of 8 in binary form in 0x8fffffff is the sign bit.
Since the maximum value of int-type data in Java is 2147483647, and the absolute value of-2147483648 is 2147483648, which is 1 greater than the maximum value 2147483647, the data obtained by directly taking the absolute value of-2147483648 exceeds the range of the maximum value 2147483647 of the system, and the system will report an error. To avoid system errors after logical and, the result obtained after logical and of each hash value and the maximum int-type data is absolute except that the result is-2147483648 is modified to 2147483647.
And the complementation operation module 206 is used for performing complementation operation on the result after the logical AND operation and the length of the set array to obtain a key value.
And the length of the set array is the number of indexes in the set array. For example, if the number of indexes in the set array is 36, the length of the set array is 36.
In Java,% is used as an operator of an arithmetic operation (modulo operation), and the sign of the result after the arithmetic operation always matches the sign of the dividend. Therefore, the key value obtained by performing the logical and operation on the result and the length of the set array just corresponds to the index of the set array.
The modulo operation of Java is prior art and the present invention will not be described in detail herein.
Illustratively, assuming that the length of the set array is 36, the indexes of the set array are 0, 1, … …, 35. The key value obtained by performing the logical and operation on the result and the length of the set array and performing the complementation operation is between 0 and 35, so that the key value obtained by the operation can be certainly the same as one index of the set array, and thus, the index of which set array the data corresponding to the key value should correspond to, that is, the storage node to which the data should be stored can be determined. The same data is ensured to correspond to the same key, and repeated caching of the data is avoided.
And the index matching module 207 is used for matching a target index which is the same as the key value from the set array.
In this embodiment, after the key value corresponding to each piece of data is obtained through calculation, since the key value may be definitely the same as one index of the set array, an index in the set array, which is the same as the key value, is used as a target index, which is convenient for matching a target storage node corresponding to the target index in the subsequent process, and the data is stored in the target storage node.
For example, if the key value corresponding to the first data is 6, the index S6 in the index of the set array is the target index of the first data, and the storage node M3 corresponding to the target index S6 is the target storage node; if the key value corresponding to the second data is 0, the index S0 in the index of the set array is the target index of the second data, and the storage node M1 corresponding to the target index S0 is the target storage node.
And the data storage module 208 is configured to store the data corresponding to the key value in the storage node corresponding to the target index.
In this embodiment, when the target index is determined, a target storage node corresponding to the target index may be determined according to the association relationship between the set array and the storage node, and the data corresponding to the key value is stored in the target storage node.
As in the above example, the first data is stored in storage node M3 and the second data is stored in storage node M1.
In a preferred embodiment, the apparatus may further include:
an index obtaining module 209, configured to obtain all indexes of the set array when a read instruction of the data is received; acquiring a storage node corresponding to each index in the Redis cluster;
a data reading module 210, configured to sequentially read data in the storage node corresponding to each index according to the order of the indexes; and connecting the read data and displaying.
For example, suppose that the storage nodes of the Redis cluster are M1, M2, M3, M4, M5 and M6, respectively, the set array length is 12, and the indexes are S0, S1, S2, S3, S4, S5, S6, S7, S8, S9, S10 and S11, respectively. Wherein all indexes of the set array are acquired when a read instruction of the data is received, and then obtaining a storage node M1 corresponding to the index S0, a storage node M3 corresponding to the index S1, storage node M4 corresponding to index S2, storage node M2 corresponding to index S3, storage node M6 corresponding to index S4, storage node M5 corresponding to index S5, storage node M3 corresponding to index S6, storage node M1 corresponding to index S7, storage node M6 corresponding to index S8, storage node M4 corresponding to index S9, storage node M2 corresponding to index S10, the storage node M5 corresponding to the index S11 reads the data in the storage nodes M1, M3, M4, M2, M6, M5, M3, M1, M6, M4, M2, and M5, respectively, in the order of 0 to 11 of the index, and finally links the read data and displays the linked data as the final result.
Specifically, each time data in one storage node is read, the data is sequentially read from top to bottom. Preferably, after the data in the storage node is read, the data in the storage node is marked, and when the storage node is read next time, the pointer is directly positioned to the next line of the mark, so that the required data can be read.
In summary, the data uniform caching device based on the Redis cluster according to the present invention initializes the set array, establishes an association relationship between the index of the set array and the storage node in the Redis cluster, calculates the hash value of each data for the data to be cached, performs logical and operation on each hash value and the maximum int-type data, and performs remainder operation with the length of the set data to obtain the index of the set array to which the key value exactly corresponds, so that the target index of the set array which is the same as each key value can be matched, and further the target storage node corresponding to the target index is determined, and finally stores the data corresponding to the key value in the target storage node. On the basis of ensuring that the cache data cannot be repeated, the uniform distribution of large data volume on the Redis cluster is also ensured. In addition, the hash value is calculated by taking the key (key) of the data as an object, the data with the same key can be stored in the same storage node, and the data with different keys can be stored in different storage nodes; and the length of the initialized set array is comprehensively considered by combining the scale of the Redis cluster and the performance of the storage node, so that the data distribution of the Redis cluster is uniform, and the data writing and reading speed is not influenced.
EXAMPLE III
Fig. 3 is a schematic structural diagram of a terminal according to a third embodiment of the present invention. In the preferred embodiment of the present invention, the terminal 3 includes a memory 31, at least one processor 32, at least one communication bus 33, and a transceiver 34.
It will be appreciated by those skilled in the art that the configuration of the terminal shown in fig. 3 is not limiting to the embodiments of the present invention, and may be a bus-type configuration or a star-type configuration, and the terminal 3 may include more or less hardware or software than those shown, or a different arrangement of components.
In some embodiments, the terminal 3 is a device capable of automatically performing numerical calculation and/or information processing according to preset or stored instructions, and the hardware includes but is not limited to a microprocessor, an application specific integrated circuit, a programmable gate array, a digital processor, an embedded device, and the like. The terminal 3 may further include a client device, which includes, but is not limited to, any electronic product capable of performing human-computer interaction with a client through a keyboard, a mouse, a remote controller, a touch panel, or a voice control device, for example, a personal computer, a tablet computer, a smart phone, a digital camera, and the like.
It should be noted that the terminal 3 is only an example, and other existing or future electronic products, such as those that can be adapted to the present invention, should also be included in the scope of the present invention, and are included herein by reference.
In some embodiments, the memory 31 is used for storing program codes and various data, such as devices installed in the terminal 3, and realizes high-speed and automatic access to programs or data during the operation of the terminal 3. The Memory 31 includes a Read-Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), a One-time Programmable Read-Only Memory (OTPROM), an electronically Erasable Programmable Read-Only Memory (Electrically-Erasable Programmable Read-Only Memory (EEPROM)), an optical Read-Only disk (CD-ROM) or other optical disk Memory, a magnetic disk Memory, a tape Memory, or any other medium that can be used to carry or store data.
In some embodiments, the at least one processor 32 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or may be composed of a plurality of integrated circuits packaged with the same or different functions, including one or more Central Processing Units (CPUs), microprocessors, digital Processing chips, graphics processors, and combinations of various control chips. The at least one processor 32 is a Control Unit (Control Unit) of the terminal 3, connects various components of the entire terminal 3 using various interfaces and lines, and executes various functions of the terminal 3 and processes data by running or executing programs or modules stored in the memory 31 and calling data stored in the memory 31.
In some embodiments, the at least one communication bus 33 is arranged to enable connection communication between the memory 31 and the at least one processor 32 or the like.
Although not shown, the terminal 3 may further include a power supply (such as a battery) for supplying power to various components, and preferably, the power supply may be logically connected to the at least one processor 32 through a power management device, so as to implement functions of managing charging, discharging, and power consumption through the power management device. The power supply may also include any component of one or more dc or ac power sources, recharging devices, power failure detection circuitry, power converters or inverters, power status indicators, and the like. The terminal 3 may further include various sensors, a bluetooth module, a Wi-Fi module, and the like, which are not described herein again.
It is to be understood that the described embodiments are for purposes of illustration only and that the scope of the appended claims is not limited to such structures.
The integrated unit implemented in the form of a software functional module may be stored in a computer-readable storage medium. The software functional module is stored in a storage medium and includes several instructions to enable a computer device (which may be a personal computer, a terminal, or a network device) or a processor (processor) to execute parts of the methods according to the embodiments of the present invention.
In a further embodiment, in conjunction with fig. 2, the at least one processor 32 may execute operating means of the terminal 3 as well as installed various types of applications, program codes, etc., such as the various modules described above.
The memory 31 has program code stored therein, and the at least one processor 32 can call the program code stored in the memory 31 to perform related functions. For example, the respective modules illustrated in fig. 2 are program codes stored in the memory 31 and executed by the at least one processor 32, thereby implementing the functions of the respective modules.
In one embodiment of the invention, the memory 31 stores a plurality of instructions that are executed by the at least one processor 32 to implement all or a portion of the steps of the method of the invention.
Specifically, the at least one processor 32 may refer to the description of the relevant steps in the embodiment corresponding to fig. 1, and details are not repeated here.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is only one logical functional division, and other divisions may be realized in practice.
The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.
In addition, functional modules in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional module.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned. Furthermore, it is obvious that the word "comprising" does not exclude other elements or that the singular does not exclude the plural. A plurality of units or means recited in the apparatus claims may also be implemented by one unit or means in software or hardware. The terms first, second, etc. are used to denote names, but not any particular order.
Finally, it should be noted that the above embodiments are only for illustrating the technical solutions of the present invention and not for limiting, and although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions may be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.
Claims (10)
1. A data uniform caching method based on Redis cluster is characterized by comprising the following steps:
initializing a set array;
associating the index of the set array with a storage node in a Redis cluster;
when a cache instruction of data is received, calculating a hash value of the data;
performing logic AND operation on the hash value and the maximum int-type data;
performing complementation operation on the result after the logical AND operation and the length of the set array to obtain a key value;
matching a target index which is the same as the key value from the set array;
and storing the data corresponding to the key value in a storage node corresponding to the target index.
2. The method of claim 1, wherein initializing the set array comprises:
acquiring the number of storage nodes in the Redis cluster;
determining the length range of the set array according to the number;
acquiring the performance of a storage node in the Redis cluster;
when the performance is greater than a preset performance threshold, determining the length of the set array as the maximum value in the length range;
when the performance is smaller than a preset performance threshold, determining the length of the set array as a middle value in the length range, wherein the middle value refers to any value in the middle area of the length range.
3. The method as recited in claim 1, wherein the associating the index of the set array with a storage node in a Redis cluster comprises:
calculating a first hash value for each index of the set array;
calculating a second hash value of each storage node in the Redis cluster;
matching a target first hash value identical to the second hash value;
and associating the index of the set array corresponding to the target first hash value with the storage node corresponding to the second hash value.
4. The method of claim 3, wherein the calculating the second hash value for each storage node in the Redis cluster comprises:
acquiring an identifier of each storage node in the Redis cluster, wherein the identifier is an IP address of the storage node or a host name of the storage node;
and calculating a second hash value according to the identification.
5. The method of claim 1, wherein the computing the hash value of the data when the cache instruction for the data is received comprises:
when a cache instruction of data is received, reading the key values of each row of data in the data line by line;
and calculating the hash value of the key value of each row of data.
6. The method of claim 1, wherein after the logically anding the hash value with max int-type data, the method further comprises:
judging whether the result after logical AND operation is-2147483648;
when the result after the logical and operation is-2147483648, the result is determined to be 2147483647;
when the result after the logical AND operation is not-2147483648, the result is determined to be the absolute value after the logical AND operation.
7. The method of any one of claims 1 to 6, wherein the method further comprises:
when a reading instruction of the data is received, acquiring all indexes of the set array;
acquiring a storage node corresponding to each index in the Redis cluster;
sequentially reading data in the storage nodes corresponding to each index according to the sequence of the indexes;
and connecting the read data and displaying.
8. A data uniform cache device based on Redis cluster, the device comprising:
the initialization module is used for initializing a set array;
the index association module is used for associating the index of the set array with a storage node in the Redis cluster;
the hash calculation module is used for calculating the hash value of the data when a cache instruction of the data is received;
the logical operation module is used for carrying out logical AND operation on the hash value and the maximum int-type data;
the complementation operation module is used for carrying out complementation operation on the result after the logical AND operation and the length of the set array to obtain a key value;
the index matching module is used for matching a target index which is the same as the key value from the set array;
and the data storage module is used for storing the data corresponding to the key value in the storage node corresponding to the target index.
9. A terminal, characterized in that the terminal comprises a processor for implementing a Redis cluster based data uniform caching method as claimed in any one of claims 1 to 7 when executing a computer program stored in a memory.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements a Redis cluster-based data uniform caching method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910977353.XA CN111078132A (en) | 2019-10-15 | 2019-10-15 | Data uniform caching method, device, terminal and storage medium based on Redis cluster |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910977353.XA CN111078132A (en) | 2019-10-15 | 2019-10-15 | Data uniform caching method, device, terminal and storage medium based on Redis cluster |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111078132A true CN111078132A (en) | 2020-04-28 |
Family
ID=70310380
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910977353.XA Pending CN111078132A (en) | 2019-10-15 | 2019-10-15 | Data uniform caching method, device, terminal and storage medium based on Redis cluster |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111078132A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113806384A (en) * | 2021-08-19 | 2021-12-17 | 紫光云(南京)数字技术有限公司 | Method for allocating incremental integer data based on redis |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1940922A (en) * | 2005-09-30 | 2007-04-04 | 腾讯科技（深圳）有限公司 | Method and system for improving information search speed |
| CN102402394A (en) * | 2010-09-13 | 2012-04-04 | 腾讯科技（深圳）有限公司 | Hash algorithm-based data storage method and device |
| US8996535B1 (en) * | 2013-10-02 | 2015-03-31 | Netapp, Inc. | Extent hashing technique for distributed storage architecture |
| CN109769028A (en) * | 2019-01-25 | 2019-05-17 | 深圳前海微众银行股份有限公司 | Redis cluster management method, device, equipment and readable storage medium storing program for executing |
-
2019
- 2019-10-15 CN CN201910977353.XA patent/CN111078132A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1940922A (en) * | 2005-09-30 | 2007-04-04 | 腾讯科技（深圳）有限公司 | Method and system for improving information search speed |
| CN102402394A (en) * | 2010-09-13 | 2012-04-04 | 腾讯科技（深圳）有限公司 | Hash algorithm-based data storage method and device |
| US8996535B1 (en) * | 2013-10-02 | 2015-03-31 | Netapp, Inc. | Extent hashing technique for distributed storage architecture |
| CN109769028A (en) * | 2019-01-25 | 2019-05-17 | 深圳前海微众银行股份有限公司 | Redis cluster management method, device, equipment and readable storage medium storing program for executing |
Non-Patent Citations (1)
| Title |
|---|
| 吴恩慈等: "基于优化的 CAS 算法实现线程安全的 HashMap", 《软件》, pages 185 - 190 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113806384A (en) * | 2021-08-19 | 2021-12-17 | 紫光云(南京)数字技术有限公司 | Method for allocating incremental integer data based on redis |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR101441188B1 (en) | Techniques for memory de-duplication in a virtual system | |
| JP5852677B2 (en) | Register mapping method | |
| CN103782282B (en) | There is the computer system of the treater locally coherence for virtualization I/O | |
| US7996570B1 (en) | Usage of keyboard driver in extensible firmware interface for adding new hot keys at firmware level | |
| US10691601B2 (en) | Cache coherence management method and node controller | |
| CN108701026A (en) | Method and apparatus for the managing process under memory constraints | |
| US20220121383A1 (en) | Computing system and method for sharing device memories of different computing devices | |
| CN116662038B (en) | Industrial information detection method, device, equipment and medium based on shared memory | |
| CN113890712A (en) | Data transmission method and device, electronic equipment and readable storage medium | |
| US11954419B2 (en) | Dynamic allocation of computing resources for electronic design automation operations | |
| CN111078132A (en) | Data uniform caching method, device, terminal and storage medium based on Redis cluster | |
| US9792042B2 (en) | Systems and methods for set membership matching | |
| CN112052144B (en) | Information management method, device, electronic equipment and storage medium | |
| CN109983443B (en) | Techniques to implement bifurcated non-volatile memory flash drives | |
| CN112598135A (en) | Model training processing method and device, computer equipment and medium | |
| US20060069900A1 (en) | Method and system for supporting virtual mappings for shared firmware | |
| CN110730228A (en) | Data storage method, electronic device, system and medium | |
| US10754792B2 (en) | Persistent virtual address spaces | |
| CN114817288A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN115687384A (en) | UUID (user identifier) identification generation method, device, equipment and storage medium | |
| US10776021B2 (en) | Exit-less host page table switching and virtual machine function detection with memory pages storing an identification value that are mapped at the same guest physical addresses | |
| CN114417069A (en) | Page data interaction method and device and electronic equipment | |
| CN114116673A (en) | Data migration method based on artificial intelligence and related equipment | |
| CN105659216B (en) | The CACHE DIRECTORY processing method and contents controller of multi-core processor system | |
| US11016666B2 (en) | Memory system and operating method thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |