CN115277850A - Data processing method, device, equipment and storage medium - Google Patents

Data processing method, device, equipment and storage medium Download PDF

Info

Publication number
CN115277850A
CN115277850A CN202211161353.0A CN202211161353A CN115277850A CN 115277850 A CN115277850 A CN 115277850A CN 202211161353 A CN202211161353 A CN 202211161353A CN 115277850 A CN115277850 A CN 115277850A
Authority
CN
China
Prior art keywords
node
data processing
nodes
identifier
request
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202211161353.0A
Other languages
Chinese (zh)
Other versions
CN115277850B (en
Inventor
明磊
向家勇
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China Unicom Smart Connection Technology Ltd
Original Assignee
China Unicom Smart Connection Technology Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China Unicom Smart Connection Technology Ltd filed Critical China Unicom Smart Connection Technology Ltd
Priority to CN202211161353.0A priority Critical patent/CN115277850B/en
Publication of CN115277850A publication Critical patent/CN115277850A/en
Application granted granted Critical
Publication of CN115277850B publication Critical patent/CN115277850B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23Updating
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/245Query processing
    • G06F16/2455Query execution
    • G06F16/24552Database cache management
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/10Protocols in which an application is distributed across nodes in the network

Abstract

The application discloses a data processing method, a data processing device, data processing equipment and a storage medium, relates to the technical field of computers, and can solve the problem that the cluster service function pause time is too long due to the fact that nodes need to be updated. The method comprises the following steps: receiving a data processing request sent by a user terminal, and determining a first node from candidate nodes based on a preset algorithm and an identifier carried by the data processing request; sending a first request carrying an identifier to a first node; and if the response information fed back by the first node is not received or the processing failure information is received within the preset time length, sending a second request carrying the identifier to the candidate node in a multicast mode, wherein the second request is used for indicating each node in the candidate node to search the target data corresponding to the identifier.

Description

Data processing method, device, equipment and storage medium
Technical Field
The present application relates to the field of computer technologies, and in particular, to a data processing method, apparatus, device, and storage medium.
Background
A Remote Dictionary service (Redis) cluster is a set of programs that can provide a multi-node read-write service to a user terminal, and data stored in each node is different. In the prior art, read-write requests initiated by a user terminal can be dispersed on different nodes in a bucket-dividing mode, and when the read-write requests are dispersed to a certain node, the node responds to the read-write requests to read or write data.
Currently, in the process of processing the read-write request by using the above manner, when the number of nodes is changed, in order to avoid data reading failure, a service function of a cluster generally needs to be suspended, and bucket information in all nodes is updated during the suspension period. However, it takes a long time to update the bucket information in all the nodes, which causes the service function of the cluster to be suspended for a long time, and affects the user experience.
Disclosure of Invention
The application provides a data processing method, a data processing device, a data processing apparatus and a storage medium, which can solve the problem that the cluster service function pause time is too long due to the need of updating nodes, thereby improving the user experience.
In order to achieve the purpose, the technical scheme is as follows:
in a first aspect, the present application provides a data processing method, including: under the condition of receiving a data processing request sent by a user terminal, determining a first node from candidate nodes based on a preset algorithm and an identifier carried by the data processing request; sending a first request carrying an identifier to a first node; and if the response information fed back by the first node is not received or the processing failure information is received within the preset time length, sending a second request carrying the identifier to the candidate node in a multicast mode, wherein the second request is used for indicating each node in the candidate node to search the target data corresponding to the identifier.
In the technical scheme provided by the application, when a data processing request sent by a user terminal is received, a first node can be determined from candidate nodes based on a preset algorithm and an identifier carried by the data processing request, and the first node is a node for logically storing target data, namely a node for storing the target data when no node change occurs. The first request may be sent to the first node after the first node is determined. When the data processing request is a non-write request such as a read request or an update request, the first request is used for acquiring target data corresponding to the identifier or updating the target data. In this case, after the first request is sent, if no node change occurs or the node change does not affect the internal data of the first node, the target data fed back by the first node can be successfully acquired from the first node or the target data can be successfully updated; if the node changes occur and the node changes affect the internal data of the first node, the response information fed back by the first node will not be received or the processing failure information will be received within a preset time, at this time, the application may send a second request to all candidate nodes in the cluster in a multicast manner, each node will perform self-checking after receiving the second request, and the node that successfully inquires the target data (i.e., the second node in the application) will feed back the target data. It can be seen that, in the technical solution provided by the present application, when a node change occurs, the target data can be successfully acquired without suspending the cluster service function (that is, without updating the internal data of each node), so that the problem of too long suspension time of the cluster service function due to the need to update the node can be solved by the present application, and thus, the user experience can be improved.
Optionally, in a possible design manner, the "determining a first node from candidate nodes based on a preset algorithm and an identifier carried by the data processing request" may include: determining a first numerical value based on a preset algorithm, the identifier and the total node number of the candidate nodes; determining a first node based on the first value and a preset corresponding relation under the condition that the first value and the number of enabled nodes in the candidate nodes meet a preset condition; the preset corresponding relation comprises a numerical value corresponding to each starting node in the candidate nodes; and under the condition that the first numerical value and the number of the enabled nodes do not meet the preset condition, determining a second numerical value based on the first numerical value and the number of the enabled nodes, and determining the first node according to the second numerical value and the preset corresponding relation.
Optionally, in another possible design manner, the preset algorithm is a hash algorithm, and the "determining the first value based on the preset algorithm, the identifier, and the total node number of the candidate nodes" may include: generating a hash value of the identifier according to a hash algorithm; performing complementation operation on the hash value based on the total node number to obtain a first numerical value; determining a second value based on the first value and the number of enabled nodes, comprising: and performing complementation operation on the first numerical value based on the number of the starting nodes to obtain a second numerical value.
Optionally, in another possible design manner, the data processing method provided by the present application may further include: receiving a logoff request sent by a third node; the third node is any one starting node in the candidate nodes; determining whether the third node is a last node or not based on a preset corresponding relation; the preset corresponding relation comprises a numerical value corresponding to each starting node in the candidate nodes; the last node is a node with the maximum corresponding numerical value in the enabled nodes; and under the condition that the third node is not the last node, adjusting the value corresponding to the last node in the preset corresponding relation to the value corresponding to the third node, and deleting the mapping relation between the third node and the value corresponding to the third node.
Optionally, in another possible design, the data processing method provided by the present application may further include: receiving an offline request sent by a third node; the third node is any one starting node in the candidate nodes; adjusting the read-write state of the third node to be a readable but unwritable state; and if the third node becomes a null node, adjusting the read-write state of the third node to be an unreadable and unwritable state.
Optionally, in another possible design manner, after the sending the second request carrying the identifier to the candidate node in the multicast manner is further performed: and if the target data corresponding to the identifier is found in a second node except the first node, transferring the target data to the first node.
Optionally, in a possible design, the candidate nodes include an enabled node and a node in a readable but unwritable state, and the node in an unreadable and unwritable state is not included, and the method may further include: if the starting nodes are increased or decreased, the preset corresponding relation is adjusted based on the number of the starting nodes, so that each second value can correspond to the starting nodes through the preset corresponding relation.
In a second aspect, the present application provides a data processing apparatus comprising: a determining module and a sending module;
the determining module is used for determining a first node from the candidate nodes based on a preset algorithm and an identifier carried by the data processing request under the condition of receiving the data processing request sent by the user terminal;
a sending module, configured to send a first request carrying an identifier to a first node;
and the sending module is further configured to send a second request carrying the identifier to the candidate node in a multicast manner if the response information fed back by the first node is not received or the processing failure information is received within the preset time length, and the second request is used for indicating each node in the candidate node to search for the target data corresponding to the identifier.
Optionally, in a possible design manner, the determining module is specifically configured to: determining a first numerical value based on a preset algorithm, the identifier and the total node number of the candidate nodes; determining a first node based on the first value and a preset corresponding relation under the condition that the first value and the number of enabled nodes in the candidate nodes meet a preset condition; the preset corresponding relation comprises a numerical value corresponding to each starting node in the candidate nodes; and under the condition that the first numerical value and the number of the enabled nodes do not meet the preset condition, determining a second numerical value based on the first numerical value and the number of the enabled nodes, and determining the first node according to the second numerical value and a preset corresponding relation.
Optionally, in another possible design manner, the preset algorithm is a hash algorithm; the determining module is specifically further configured to: generating a hash value of the identifier according to a hash algorithm; performing complementation operation on the hash value based on the total node number to obtain a first numerical value; and under the condition that the first numerical value is larger than the number of the starting nodes, performing complementation operation on the first numerical value based on the number of the starting nodes to obtain a second numerical value.
Optionally, in another possible design, the data processing apparatus provided in this application may further include: a receiving module; the receiving module is used for receiving an offline request sent by a third node; the third node is any one starting node in the candidate nodes; the determining module is further used for determining whether the third node is a last node or not based on the preset corresponding relation; the preset corresponding relation comprises a numerical value corresponding to each starting node in the candidate nodes; the last node is a node with the maximum corresponding numerical value in the enabled nodes; and the determining module is further used for adjusting the value corresponding to the last node in the preset corresponding relationship to the value corresponding to the third node and deleting the mapping relationship between the third node and the value corresponding to the third node under the condition that the third node is not the last node.
Optionally, in another possible design, the data processing apparatus provided in this application may further include: a receiving module; the receiving module is used for receiving an offline request sent by a third node; the third node is any one starting node in the candidate nodes; the determining module is further configured to adjust the read-write state of the third node to a readable but unwritable state when the receiving module receives the offline request sent by the third node; the determining module is further configured to adjust the read-write state of the third node to be an unreadable and unwritable state if the third node becomes an empty node.
Optionally, in another possible design manner, the data processing apparatus provided in the present application may further include a transferring module, where the transferring module is configured to, after the sending module sends the second request carrying the identifier to the candidate node in a multicast manner, transfer the target data to the first node if the target data corresponding to the identifier is found in a second node other than the first node.
Optionally, in a possible design manner, the candidate nodes include enabled nodes and nodes in a readable but unwritable state, and do not include nodes in an unreadable and unwritable state, and the data processing apparatus provided in the application may further include an adjustment module, where the adjustment module is configured to adjust the preset corresponding relationship based on the number of enabled nodes if the enabled nodes are increased or decreased, so that each second value can correspond to the enabled nodes through the preset corresponding relationship.
In a third aspect, the present application provides a data processing apparatus comprising a memory, a processor, a bus, and a communication interface; the memory is used for storing computer execution instructions, and the processor is connected with the memory through a bus; when the data processing apparatus is operating, the processor executes computer executable instructions stored by the memory to cause the data processing apparatus to perform the data processing method as provided above in the first aspect.
In a fourth aspect, the present application provides a computer-readable storage medium having stored therein instructions, which, when executed by a computer, cause the computer to perform the data processing method as provided in the first aspect.
In a fifth aspect, the present application provides a computer program product comprising computer instructions which, when run on a computer, cause the computer to perform the data processing method as provided in the first aspect.
It should be noted that the computer instructions may be stored in whole or in part on a computer-readable storage medium. The computer-readable storage medium may be packaged with a processor of a data processing device, or may be packaged separately from the processor of the data processing device, which is not limited in this application.
For the description of the second, third, fourth and fifth aspects in this application, reference may be made to the detailed description of the first aspect; in addition, for the beneficial effects described in the second aspect, the third aspect, the fourth aspect and the fifth aspect, reference may be made to the beneficial effect analysis of the first aspect, and details are not repeated here.
In the present application, the names of the above-mentioned devices or functional modules are not limited, and in actual implementation, the devices or functional modules may be represented by other names. Insofar as the functions of the respective devices or functional modules are similar to those of the present application, they are within the scope of the claims of the present application and their equivalents.
These and other aspects of the present application will be more readily apparent from the following description.
Drawings
FIG. 1 is a block diagram of a data processing system according to an embodiment of the present application;
fig. 2 is a schematic flowchart of a data processing method according to an embodiment of the present application;
FIG. 3 is a block diagram of an alternative data processing system according to an embodiment of the present application;
fig. 4 is a schematic flowchart of another data processing method according to an embodiment of the present application;
fig. 5 is a schematic flowchart of another data processing method according to an embodiment of the present application;
fig. 6 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application;
fig. 7 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application.
Detailed Description
The following describes in detail a data processing method, an apparatus, a device, and a storage medium provided in an embodiment of the present application with reference to the accompanying drawings.
The term "and/or" herein is merely an association describing an associated object, meaning that three relationships may exist, e.g., a and/or B, may mean: a exists alone, A and B exist simultaneously, and B exists alone.
The terms "first" and "second" and the like in the specification and drawings of the present application are used for distinguishing different objects or for distinguishing different processes for the same object, and are not used for describing a specific order of the objects.
Furthermore, the terms "including" and "having," and any variations thereof, as referred to in the description of the present application, are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements recited, but may alternatively include other steps or elements not recited, or may alternatively include other steps or elements inherent to such process, method, article, or apparatus.
It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described herein as "exemplary" or "e.g.," is not necessarily to be construed as preferred or advantageous over other embodiments or designs. Rather, use of the word "exemplary" or "such as" is intended to present relevant concepts in a concrete fashion.
In the description of the present application, the meaning of "a plurality" means two or more unless otherwise specified.
In addition, the data acquisition, storage, use, processing and the like in the technical scheme of the application all conform to relevant regulations of national laws and regulations.
In the existing Redis cluster architecture, in the process of processing read-write requests in a bucket-splitting manner, when the number of cluster nodes changes, in order to avoid data reading failure, a service function of a cluster generally needs to be suspended, and bucket information in all nodes is updated during the suspension period. However, it takes a long time to update the bucket information in all the nodes, which causes the service function of the cluster to be suspended for a long time, and affects the user experience.
In view of the problems in the prior art, an embodiment of the present application provides a data processing method, which may search for target data corresponding to an identifier in a multicast manner when a node is changed, so that the target data may be successfully acquired even when a cluster service function is not suspended, and therefore, a problem of an excessively long suspension time of the cluster service function due to a need to update the node may be solved, so as to improve user experience.
The data processing method provided by the embodiment of the present application may be executed by the data processing apparatus provided by the embodiment of the present application, and the apparatus may be implemented by software and/or hardware and integrated in a data processing device executing the method.
The data processing method provided by the embodiment of the application can be applied to a data processing system. Fig. 1 shows a possible configuration of the data processing system. As shown in fig. 1, a data processing system may include a data processing apparatus, a plurality of candidate nodes (such as node a, node B, node C, and node D in fig. 1), and a plurality of user terminals (such as terminal a, terminal B, and terminal C in fig. 1).
In one possible implementation, the data processing device may be a server. For example, the data processing method provided in the embodiment of the present application may be applied to a Redis cluster architecture, the candidate node in the embodiment of the present application may be a master node in the Redis cluster, and the data processing device may be a background server of the Redis cluster.
The user terminal may be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a notebook computer, an ultra-mobile personal computer (UMPC), a handheld computer, a netbook, a Personal Digital Assistant (PDA), a wearable electronic device, a virtual reality device, and other different types of terminals connected to the data processing device.
It can be understood that fig. 1 shows four candidate nodes and three user terminals as an example, and in practical application, the number of the candidate nodes and the number of the user terminals are not limited in the embodiment of the present application, and both the candidate nodes and the user terminals may be any number.
In addition, in the Redis cluster architecture, each master node may correspond to one slave node, and is configured to back up data stored in the master node, so that when the data processing method provided in the embodiment of the present application is applied to the Redis cluster architecture, the data processing system may further include a slave node corresponding to each candidate node.
It should be noted that, the processing procedure of the data processing device for the data processing request initiated by each user terminal is the same, and in the embodiment of the present application, the data processing method provided in the embodiment of the present application will be described in detail by taking the processing procedure of the data processing device for the data processing request initiated by any user terminal as an example.
The data processing method provided by the present application is described below with reference to the data processing system shown in fig. 1. Referring to fig. 2, a data processing method provided in an embodiment of the present application may include S201 to S204:
s201, under the condition that a data processing request sent by a user terminal is received, a first node is determined from candidate nodes based on a preset algorithm and an identifier carried by the data processing request.
The data processing request sent by the user terminal may be a read request, or may be a write request, or may be an update request, etc. When the data processing request is a reading request, the data processing request carries an identifier, and the request initiated by the user terminal is used for acquiring target data corresponding to the identifier; when the data processing request is an updating request, the data processing request carries an identifier, and the request initiated by the user terminal is used for updating target data corresponding to the identifier; when the data processing request is a write request, the data processing request carries an identifier and corresponding target data, and the data processing device may determine a node from the candidate nodes based on the request initiated by the user terminal, and write the identifier and the corresponding target data by the node.
The preset algorithm may be a predetermined algorithm. For example, the preset algorithm may be an algorithm determined in advance for generating a value according to a certain rule based on the identifier.
The identifier may be an identification for uniquely representing a set of data. For example, when the data processing method provided by the embodiment of the present application is applied to a Redis cluster architecture, data may be read or written based on a key-Value pair (key-Value), the identifier may be a key, and the destination data may be a Value. Each data may have a corresponding key.
Optionally, in this embodiment of the application, the first numerical value may be determined based on a preset algorithm, the identifier, and the total node number of the candidate nodes; determining a first node based on the first value and a preset corresponding relation under the condition that the first value and the number of enabled nodes in the candidate nodes meet a preset condition; and under the condition that the first numerical value and the number of the enabled nodes do not meet the preset condition, determining a second numerical value based on the first numerical value and the number of the enabled nodes, and determining the first node according to the second numerical value and the preset corresponding relation.
The preset corresponding relation comprises a numerical value corresponding to each starting node in the candidate nodes. The preset condition may be a predetermined condition, for example, the preset condition may be that the first value is smaller than the number of enabled nodes in the candidate nodes.
Because the particularity of the Redis cluster on data storage is that each time a node is increased or decreased, the data storage structure of the whole cluster is affected, in the data processing method provided by the embodiment of the application, when the Redis cluster is created, the number of nodes in the cluster can be limited, that is, the total number of nodes of candidate nodes is limited, so as to reduce the influence of node change on the data storage structure of the cluster. For example, when the total number of nodes is 20, the user may select the number of nodes (i.e., the number of enabled nodes) to be used from the candidate nodes according to a requirement, for example, the number of enabled nodes is 19, and when the number of nodes is increased, the number of the increased nodes cannot exceed 20, that is, only 1 node can be increased at most.
In the embodiment of the application, different values can be assigned to different nodes by determining the preset corresponding relationship in advance, then the preset operation is performed on the identifier and the total node number according to the preset algorithm to obtain the first value, and then the node corresponding to the first value can be determined according to the preset corresponding relationship. Since the first value is obtained by performing the predetermined operation on the identifier and the total number of nodes, the node corresponding to the first value may not be the enabled node, that is, the node corresponding to the first value may not be able to write or read data. Therefore, in the embodiment of the present application, after the first value is obtained, the first value may be compared with the number of enabled nodes, and if the first value is greater than or equal to the number of enabled nodes, it indicates that the node corresponding to the first value is not an enabled node, and at this time, the preset operation may be performed on the first value and the number of enabled nodes again to obtain the second value. Since the second value is obtained based on the number of enabled nodes, the node corresponding to the second value is an enabled node. By the method for determining the first node, the determined first node is not the enabled node set by the user, so that the occurrence of data reading or writing failure can be reduced.
The preset operation may be a predetermined operation manner, and for example, the preset operation may be a remainder operation.
Optionally, in a possible implementation manner, the preset algorithm may be a hash algorithm, in the embodiment of the present application, a hash value of the identifier may be generated according to the hash algorithm, and then a remainder operation is performed on the hash value based on the total number of nodes to obtain a first value; determining a first node based on the first value and a preset corresponding relation under the condition that the first value is smaller than the number of enabled nodes in the candidate nodes; when the first value is greater than or equal to the number of the enabled nodes, the first value may be subjected to a remainder operation based on the number of the enabled nodes to obtain a second value, and the first node may be determined according to the second value and a preset corresponding relationship.
Illustratively, the total number of the candidate nodes is 20, and the number of the enabled nodes is 5. In the preset correspondence, the 5 enabled nodes may respectively correspond to 5 values from 0 to 4. If the Hash value of the identifier generated by the Hash algorithm is represented by Hash (key), the first value = Hash (key)% 20, i.e., the remainder operation is performed with 20 as the divisor and the Hash value of the identifier as the dividend. For example, when the first value is 10, since 10 is greater than 5, 10 may be used to perform a modulo operation on 5 again to obtain a second value 0, and since 0 is less than 5, a node corresponding to 0, that is, a first node of the 5 enabled nodes may be determined as the first node. For another example, when the first value is 3, since 3 is less than 5, the fourth node of the 5 enabled nodes may be directly determined as the first node.
S202, sending a first request carrying the identifier to the first node.
Illustratively, in the case that the data processing request is a read request, the first request is used to instruct the first node to search for and feed back the target data corresponding to the identifier. In the case that the data processing request is an update request, the first request is used to instruct the first node to search for and update the target data corresponding to the identifier. And under the condition that the data processing request is a write request, the first request also carries target data, and the first request is used for indicating the corresponding relation between the write identifier of the first node and the target data.
In the embodiment of the application, since the determined first node is the enabled node set by the user, after the first node receives the data processing request without node change or without influence of the node change on internal data of the first node, if the data processing request is a read request, the corresponding target data Value can be searched according to the identifier key, and then the searched target data Value can be fed back to the data processing device, and then the target data Value is returned to the user terminal initiating the data processing request by the data processing device.
In addition, in the existing method of splitting the read-write request initiated by the user terminal based on the bucket splitting method, part of nodes may be overheated due to uneven data on each node. In the embodiment of the application, when the data processing device receives the write request, the data can be randomly written into each node by adopting the hash algorithm according to the identifier key, so that the problem that the data are not uniformly distributed in each node can be solved. In addition, the processing requirement of the write request is to find a node capable of storing the target data, which does not involve data acquisition, and the change of the node does not cause data write failure.
And S203, if the response information fed back by the first node is not received or the processing failure information is received within the preset time length, sending a second request carrying the identifier to the candidate node in a multicast mode, wherein the second request is used for indicating each node in the candidate node to search the target data corresponding to the identifier.
The preset duration may be a duration determined in advance, which is not limited in the embodiment of the present application.
When a node change occurs and the node change affects internal data of the first node, the first node may not be able to find the destination data Value corresponding to the identifier key, and may not respond to the first request initiated by the data processing device or may send processing failure information to the data processing device. At this time, if the data processing device does not receive the target data fed back by the first node for a long time or receives the processing failure information, the data processing device may send a second request carrying an identifier to the candidate nodes in the cluster in a multicast manner, after each candidate node receives the second request, the candidate nodes may search for the target data Value corresponding to the identifier key, and if a certain node (i.e., the second node in the embodiment of the present application) is successfully searched, the node may respond to the second request and feed back the searched target data Value to the data processing device. Here, multicasting may mean sending information for all candidate nodes (redis nodes), i.e. may send information to enabled nodes as well as to downline candidate nodes (i.e. nodes in a readable but unwritable state).
And S204, receiving the target data fed back by the second node, and returning the target data to the user terminal.
And the second node is a node which successfully finds the target data in the candidate nodes. Wherein, step S204 may be an optional step. In the case where the data processing request is a write request, feedback of target data may not be required.
According to the data processing method provided by the embodiment of the application, the node which actually stores the target data can be searched in a multicast mode when the node is changed, and the situation that the target data cannot be acquired is avoided. Optionally, in order to reduce the number of multicast and improve the data processing efficiency, in this embodiment of the application, when a node change occurs, an idle service period of a cluster (for example, 12 am to 5 am) may be selected to update the internal data of each node (that is, to perform offline update). Therefore, the nodes are searched in a multicast mode in a short time after the nodes are changed, and the data are updated uniformly in an idle service period, so that the cluster service suspension function is ensured not to influence users, and the data processing efficiency is also ensured.
Optionally, after sending the second request carrying the identifier to the candidate node in a multicast manner, the data processing method provided in the embodiment of the present application may further include: and if the target data corresponding to the identifier is found in a second node except the first node, transferring the target data to the first node.
For example, after receiving the target data fed back by the second node, a third request may be sent to the first node, for instructing the first node to write the correspondence between the identifier and the target data; and sending a fourth request to the second node, wherein the fourth request is used for indicating the second node to delete the corresponding relation between the identifier and the target data. I.e. the target data is transferred from the second node to the first node.
In the embodiment of the application, under the condition that the node is changed, the node which actually stores the target data can be searched in a multicast mode, and after the node is searched, the target data can be synchronously updated to the node which is logically positioned by the target data. In this way, updating of node data may be achieved while not suspending cluster service functionality. Under the condition that the data volume of the node is small, the data migration during node change can be realized in the mode, and the data migration is not required to be realized in an off-line updating mode.
Optionally, the data processing method provided in the embodiment of the present application may further include: receiving an offline request sent by a third node; adjusting the read-write state of the third node to be a readable but unwritable state; and if the third node becomes a null node, adjusting the read-write state of the third node to be an unreadable and unwritable state. Wherein the state of the enabling node may be a readable and writable state. The candidate nodes include nodes in an enabled state (readable and writable state) and a readable but unwritable state, and nodes in an unreadable and unwritable state are not included.
When receiving the offline request of the third node, the data processing device may first adjust the read-write state of the third node to a readable but unwritable state, and then actually offline the third node after the data in the third node is synchronously migrated in a multicast or offline updating manner. Therefore, after the node which actually stores the target data is found in a multicast mode, the target data can be successfully acquired.
Optionally, the data processing method provided in the embodiment of the present application may further include receiving an offline request sent by a third node; determining whether the third node is a last node or not based on a preset corresponding relation; and under the condition that the third node is not the last node, adjusting the value corresponding to the last node in the preset corresponding relation to the value corresponding to the third node, and deleting the mapping relation between the third node and the value corresponding to the third node.
The third node is any one starting node in the candidate nodes, and the last node is a node with the maximum corresponding numerical value in the starting nodes.
In the embodiment of the application, when a user increases a node, the mapping relationship between the node and a corresponding numerical value can be directly added in a preset corresponding relationship, at this time, the mapping relationship between an original enabled node and the numerical value cannot be influenced, and when the node is reduced, if the node is the node with the maximum corresponding numerical value in the enabled node, the node is directly deleted, and the mapping relationship between the original enabled node and the numerical value cannot be influenced. However, if the node is not the node with the largest corresponding numerical value in the enabled nodes, directly deleting the node will affect the mapping relationship between the original enabled node and the numerical value. Therefore, in the embodiment of the present application, when the offline request of the third node is received, the position of the third node in the enabled node needs to be determined, and if the third node is not the last node, the value corresponding to the last node in the preset corresponding relationship needs to be adjusted again.
Optionally, the data processing method provided in the embodiment of the present application may further include, if the number of enabled nodes is increased or decreased, adjusting the preset corresponding relationship based on the number of enabled nodes, so that each second value may correspond to an enabled node through the preset corresponding relationship.
Referring to fig. 3, an embodiment of the present application further provides another data processing system. As shown in fig. 3, the data processing system may include a data processing apparatus, a plurality of servers (such as server a and server B in fig. 3), a plurality of candidate nodes (such as node a, node B, node C, and node D in fig. 3), and a plurality of user terminals (such as terminal a, terminal B, and terminal C in fig. 3).
In a possible implementation manner, when the data processing method is applied to a Redis cluster architecture, a background Server of the Redis cluster may be a Linux Virtual Server (LVS), the LVS is a Virtual Server cluster system, and the data processing device may be a load balancing scheduler in the Server cluster system, and is configured to schedule a data processing request initiated by a user terminal to a different Server, determine a first node from candidate nodes by the scheduled Server, and read or write data from or to the first node.
Optionally, the data processing method provided in this embodiment of the present application may also be applied to the data processing system shown in fig. 3, and specifically, when receiving a data processing request sent by a user terminal, the data processing device may further determine, based on a load balancing algorithm, a target server from candidate servers (such as server a and server B in fig. 3), and then determine, by the target server, a first node based on a preset algorithm and an identifier.
When the data processing device is a server, if the server is abnormal and cannot work, the service function of the cluster is also suspended. Therefore, in the embodiment of the present application, an LVS may be added to the data processing system, and a load balancing scheduler in the LVS is used as the data processing device, so as to split the data processing request based on a load balancing algorithm. Therefore, when a certain server is abnormal, the data processing request can be processed by other servers, and the normal operation of the service function of the cluster is ensured.
For example, the load balancing algorithm may be a round-robin scheduling algorithm, a weighted round-robin scheduling algorithm, or a weighted minimum connection scheduling algorithm, which is not limited in this embodiment of the present application.
Optionally, as shown in fig. 4, an embodiment of the present application further provides a data processing method, which may be applied to the data processing system shown in fig. 3, where an execution subject is a data processing device, and the data processing method includes S401 to S405:
s401, under the condition that a data processing request sent by a user terminal is received, a target server is determined from candidate servers based on a load balancing algorithm.
S402, the data processing device determines a first node through the target server based on a preset algorithm and the identifier.
S403, the data processing device sends a first request carrying the identifier to the first node through the target server.
The request content of the first request may refer to the corresponding description, and is not described herein again.
S404, if the response information fed back by the first node is not received or the processing failure information is received within the preset time length, a second request carrying the identifier is sent to the candidate node through the target server in a multicast mode and used for indicating each node in the candidate node to search the target data corresponding to the identifier.
S405, receiving target data fed back by the second node through the target server, and returning the target data to the user terminal.
Wherein, step S405 may be an optional step.
In summary, in the data processing method provided in this embodiment of the present application, when a data processing request sent by a user terminal is received, a first node may be determined from candidate nodes based on a preset algorithm and an identifier carried in the data processing request, where the first node is a node that logically stores target data, that is, a node that stores the target data when no node change occurs. The first request may be sent to the first node after the first node is determined. When the data processing request is a non-write request such as a read request or an update request, the first request is used for acquiring target data corresponding to the identifier or updating the target data. In this case, after the first request is sent, if no node change occurs or the node change does not affect the internal data of the first node, the target data fed back by the first node can be successfully acquired from the first node or the target data can be successfully updated; if the node change occurs and the node change affects the internal data of the first node, the response information fed back by the first node will not be received or the processing failure information will be received within a preset time, at this time, in the embodiment of the present application, the second request may be all sent to the candidate nodes in the cluster in a multicast manner, each node may perform self-checking after receiving the second request, and the node that successfully inquires the target data (i.e., the second node in the present application) may feed back the target data. It can be seen that, in the embodiment of the present application, when a node change occurs, the target data may also be successfully acquired without suspending the cluster service function (that is, without updating the internal data of each node), so that the embodiment of the present application may solve the problem of an excessively long suspension time of the cluster service function due to the need to update the node, and may improve user experience.
In order to verify that the data processing method provided by the embodiment of the application can successfully acquire the target data even when the internal data of each node is not updated when the node is changed. The following description is made with two specific examples.
Example one: take the total number of candidate nodes as 20 and the number of enabled nodes as 5 (nodes a, B, C, D and E) as an example. These 5 enabled nodes may correspond to 5 values of 0 to 4, respectively. For example, when the data processing apparatus receives a write request, and the first Value of the identifier key in the write request is 5, since the first Value 5 is not less than the number 5 of enabled nodes, the second Value 0 needs to be calculated again, at this time, the a node corresponding to the second Value 0 may be determined as the first node, and the destination data Value corresponding to the identifier key may be written in the a node. After writing, when a user adds an enabled node at a certain time, that is, when the F node that has not been enabled before can now read or write data, the data that should be stored in the F node in the a, B, C, D, and E nodes may not be migrated to the F node. If the F node is enabled, the data processing device receives a read request of the identifier key, the first Value of the identifier key is 5, since the number of the current enabled nodes is changed to 6, and the first Value 5 is smaller than the number 6 of the enabled nodes, the F node corresponding to the first Value 5 can be determined as the first node, but before the node is changed, the destination data Value corresponding to the identifier key is stored in the node a according to the logic of the previous writing, so that the destination data Value cannot be searched by the F node at this time, the data processing device does not receive the feedback of the F node for a long time, and the data processing device searches the node a through a multicast mode and obtains the destination data Value from the node a. Before the F node is enabled, the preset correspondence may be that nodes a, B, C, D, and E correspond to 0, 1, 2, 3, and 4, respectively, and after the F node is enabled, the F node may correspond to 5. That is, if the enabled nodes are increased or decreased, the preset corresponding relationship is changed based on the number of the enabled nodes, so that each second value can correspond to the enabled nodes through the preset corresponding relationship.
In example two, the total number of candidate nodes is 20, and the number of enabled nodes is 6 (nodes a, B, C, D, E, and F). These 6 enabled nodes may correspond to 6 values of 0 to 5, respectively.
In a possible implementation manner, when the data processing apparatus receives a write request, and the first Value of the identifier key in the write request is 5, since the first Value 5 is smaller than the number 6 of enabled nodes, at this time, an F node corresponding to the first Value 5 may be determined as a first node, and then the destination data Value corresponding to the identifier key may be written in the F node. Then, when an offline request of the F node is received, since the F node is the last node and does not affect the mapping relationship between other nodes and the numerical value, the mapping relationship between the F node and the numerical value 5 in the preset corresponding relationship can be directly deleted, and the read-write state of the F node can be adjusted to be a readable but unwritable state. After adjustment, if a read request of the identifier key is received, the first Value of the identifier key is 5, since the number of the current enabled nodes is changed to 5, and the first Value 5 is not less than the number of the enabled nodes 5, a second Value 0 needs to be recalculated, the a node corresponding to the second Value 0 may be determined as the first node, but before the node is changed, the destination data Value corresponding to the identifier key is stored in the F node according to the logic in the previous writing, so the a node cannot search the destination data Value at this time, the data processing device does not receive the feedback of the a node for a long time, the F node may be searched in a multicast manner, and at this time, although the F node cannot write data, the data can be normally read, and the destination data Value can be successfully obtained from the F node.
In another possible implementation manner, the data processing device receives a write request, and the first Value of the identifier key1 in the write request is 5, since the first Value 5 is smaller than the number 6 of enabled nodes, at this time, the F node corresponding to the first Value 5 may be determined as the first node, and then the destination data Value1 corresponding to the identifier key1 may be written in the F node. In addition, the data processing apparatus further receives a write request, and the first Value of the identifier key2 in the write request is 3, and since the first Value 3 is smaller than the number 6 of enabled nodes, at this time, the D node corresponding to the first Value 3 may be determined as the first node, and then the destination data Value2 corresponding to the identifier key2 may be written into the D node. Then, when a logout request of the D node is received, since the D node is not the last node and the logout will affect the mapping relationship between other nodes and the value, the mapping relationship between the F node and the value 5 may be adjusted to the mapping relationship between the F node and the value 3, the mapping relationship between the D node and the value 3 may be deleted, and the read-write state of the D node may be adjusted to a readable but non-writable state. After adjustment, if a read request of the identifier key1 is received, the first Value of the identifier key1 is 5, since the number of the current enabled nodes is changed to 5, and the first Value 5 is not less than the number 5 of the enabled nodes, recalculation is needed to obtain the second Value 0, and the a node corresponding to the second Value 0 may be determined as the first node, but before the nodes are changed, the destination data Value1 corresponding to the identifier key1 is stored in the F node according to the logic in the previous writing, so that the a node cannot find the destination data Value1 at this time, the data processing device does not receive the feedback of the a node for a long time, and will find the F node by a multicast manner, and at this time, although the F node cannot write data, but can normally read data, the destination data Value1 can be successfully obtained from the F node. If a read request of the identifier key2 is received after adjustment, the first Value of the identifier key2 is 3, the number of the current enabled nodes is changed to 5, the first Value 3 is less than the number of the enabled nodes 5, and the F node corresponding to the first Value 3 can be determined as the first node, but before the node change, the destination data Value2 corresponding to the identifier key2 is stored in the D node according to the logic written before, so that the destination data Value2 cannot be found by the F node at this time, the data processing device does not receive the feedback of the F node for a long time, and can find the D node by a multicast mode, and at this time, although the D node cannot write data, the data can be read normally, and the destination data Value2 can be successfully obtained from the D node.
As can be seen from the two examples, no matter nodes are added or nodes are reduced, the target data can be successfully acquired under the condition that internal data of each node is not updated, so that the problem that the cluster service function is suspended for too long time due to the fact that the nodes need to be updated can be solved, and user experience is improved.
Optionally, as shown in fig. 5, an embodiment of the present application further provides a data processing method, where the method may be applied to the data processing system shown in fig. 1 or fig. 3, and the execution subject may be a data processing device, and the method may include S501 to S5012:
s501, receiving a data processing request carrying an identifier sent by a user terminal.
S502, generating a hash value of the identifier according to a hash algorithm, and performing complementation operation on the hash value based on the total node number to obtain a first numerical value.
It can be understood that, when the data processing method provided in the embodiment of the present application is applied to the data processing system shown in fig. 1, the data processing device may directly process the data processing request, and when the data processing method provided in the embodiment of the present application is applied to the data processing system shown in fig. 3, the data processing device may process the data processing request through the target server after determining the target server.
S503, judging whether the first numerical value is smaller than the number of the enabled nodes.
Executing step S504 under the condition that the first numerical value is smaller than the number of the enabled nodes; in the case where it is determined that the first value is greater than or equal to the number of enabled nodes, step S505 is performed.
S504, determining a first node based on the first numerical value and the preset corresponding relation.
After step S504, step S506 is executed.
And S505, performing complementation operation on the first numerical value based on the number of the enabled nodes to obtain a second numerical value, and determining the first node according to the second numerical value and a preset corresponding relation.
S506, judging whether the data processing request is a writing request.
In a case where it is determined that the data processing request is not a write request, step S507 is executed; in a case where it is determined that the data processing request is a write request, step S5012 is executed.
S507, sending a first request to a first node in the candidate nodes.
And S508, judging whether the target data fed back by the first node is received within a preset time.
If the target data fed back by the first node is determined to be received within the preset time, executing the step S5011; if it is determined that the target data fed back by the first node is not received within the preset time, step S509 is executed.
And S509, sending the second request to the candidate node by adopting a multicast mode.
And S5010, receiving target data fed back by a second node in the candidate nodes.
And S5011, returning the target data to the user terminal or updating the target data.
S5012, sending a first request carrying the identifier and the target data to the first node, where the first request is used to indicate that the first node writes a corresponding relationship between the identifier and the target data.
As shown in fig. 6, an embodiment of the present application further provides a data processing apparatus, where the apparatus may include: a determination module 11 and a sending module 21.
The determining module 11 executes S201 in the above method embodiment, and the sending module 21 executes S202, S203, and S204 in the above method embodiment.
Specifically, the determining module 11 is configured to determine, based on a preset algorithm and an identifier carried in a data processing request, a first node from candidate nodes when the data processing request sent by the user terminal is received;
a sending module 21, configured to send a first request carrying an identifier to a first node;
the sending module 21 is further configured to send a second request carrying an identifier to the candidate node in a multicast manner if no response information fed back by the first node is received or no processing failure information is received within the preset time length, and is used to instruct each node in the candidate node to search for target data corresponding to the identifier.
Optionally, the sending module 21 is further configured to receive target data fed back by the second node, and return the target data to the user terminal.
Optionally, in a possible design manner, the determining module 11 is specifically configured to: determining a first numerical value based on a preset algorithm, the identifier and the total node number of the candidate nodes; determining a first node based on the first value and a preset corresponding relation under the condition that the first value and the number of enabled nodes in the candidate nodes meet a preset condition; the preset corresponding relation comprises numerical values corresponding to all the starting nodes in the candidate nodes; and under the condition that the first numerical value and the number of the enabled nodes do not meet the preset condition, determining a second numerical value based on the first numerical value and the number of the enabled nodes, and determining the first node according to the second numerical value and a preset corresponding relation.
Optionally, in another possible design manner, the preset algorithm is a hash algorithm; the determining module 11 is further specifically configured to: generating a hash value of the identifier according to a hash algorithm; performing remainder operation on the hash value based on the total node number to obtain a first numerical value; and under the condition that the first numerical value is larger than the number of the starting nodes, performing complementation operation on the first numerical value based on the number of the starting nodes to obtain a second numerical value.
Optionally, in another possible design, the data processing apparatus provided in this application may further include: a receiving module; the receiving module is used for receiving an offline request sent by a third node; the third node is any one starting node in the candidate nodes; the determining module 11 is further configured to determine whether the third node is a last node based on a preset correspondence; the preset corresponding relation comprises numerical values corresponding to all the starting nodes in the candidate nodes; the last node is a node with the maximum corresponding numerical value in the enabled nodes; the determining module 11 is further configured to, when it is determined that the third node is not the last node, adjust a value corresponding to the last node in the preset corresponding relationship to a value corresponding to the third node, and delete the mapping relationship between the third node and the value corresponding to the third node.
Optionally, in another possible design manner, the data processing apparatus provided in the embodiment of the present application may further include: a receiving module; the receiving module is used for receiving an offline request sent by a third node; the third node is any one starting node in the candidate nodes; the determining module 11 is further configured to, when the receiving module receives the offline request sent by the third node, adjust the read-write state of the third node to a readable but unwritable state; the determining module 11 is further configured to adjust the read-write state of the third node to an unreadable and unwritable state if the third node becomes an empty node.
Optionally, in another possible design manner, the data processing apparatus provided in this embodiment of the present application may further include a transferring module, where the transferring module is configured to, after the sending module 21 sends the second request carrying the identifier to the candidate node in a multicast manner, transfer the target data to the first node if the target data corresponding to the identifier is found in a second node other than the first node.
Optionally, in a possible design manner, the candidate nodes include an enabling node and a node in a readable but unwritable state, and do not include a node in an unreadable and unwritable state, and the data processing apparatus provided in this embodiment of the application may further include an adjusting module, where the adjusting module is configured to, if the enabling nodes are increased or decreased, adjust the preset corresponding relationship based on the number of the enabling nodes, so that each second value can correspond to the enabling node through the preset corresponding relationship.
Optionally, the data processing apparatus may further include a storage module, and the storage module is configured to store the program code of the data processing apparatus, and the like.
As shown in fig. 7, the embodiment of the present application further provides a data processing device, which includes a memory 41, a processor (e.g., 42-1 and 42-2 in fig. 7), a bus 43, and a communication interface 44; the memory 41 is used for storing computer execution instructions, and the processor is connected with the memory 41 through a bus 43; when the data processing apparatus is operating, the processor executes computer-executable instructions stored by the memory 41 to cause the data processing apparatus to perform the data processing method provided by the above-described embodiments.
In a particular implementation, as one embodiment, the processor may include one or more Central Processing Units (CPUs), such as CPU0 and CPU1 shown in FIG. 7. And as an example, the data processing device may include a plurality of processors, such as processor 42-1 and processor 42-2 shown in fig. 7. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor herein may refer to one or more devices, circuits, and/or processing cores for processing data (e.g., computer program instructions).
The memory 41 may be, but is not limited to, a read-only memory 41 (ROM) or other type of static storage device that can store static information and instructions, a Random Access Memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact disc, laser disc, optical disc, digital versatile disc, blu-ray disc, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. The memory 41 may be self-contained and coupled to the processor via a bus 43. The memory 41 may also be integrated with the processor.
In a specific implementation, the memory 41 is used for storing data in the present application and computer-executable instructions corresponding to a software program for executing the present application. The processor may perform various functions of the data processing apparatus by running or executing software programs stored in the memory 41, and calling data stored in the memory 41.
The communication interface 44 may be any device, such as a transceiver, for communicating with other devices or communication networks, such as a control system, a Radio Access Network (RAN), a Wireless Local Area Network (WLAN), etc. The communication interface 44 may include a receiving unit implementing a receiving function and a transmitting unit implementing a transmitting function.
The bus 43 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an extended ISA (enhanced industry standard architecture) bus, or the like. The bus 43 may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown in FIG. 7, but this is not intended to represent only one bus or type of bus.
As an example, in connection with fig. 6, the function implemented by the sending module in the data processing apparatus is the same as the function implemented by the sending unit in fig. 7, and the function implemented by the determining module in the data processing apparatus is the same as the function implemented by the processor in fig. 7. When the data processing apparatus includes the storage module, the function realized by the storage module is the same as that realized by the memory in fig. 7; when the data processing apparatus includes a receiving module therein, the function realized by the receiving module is the same as that realized by the receiving unit in fig. 7.
For the explanation of the related content in this embodiment, reference may be made to the above method embodiment, which is not described herein again.
Through the description of the foregoing embodiments, it may be clearly understood by those skilled in the art that, for convenience and simplicity of description, only the division of each functional module is used for illustration, and in practical applications, the above function distribution may be completed by different functional modules according to needs, that is, the internal structure of the device may be divided into different functional modules to complete all or part of the above described functions. For the specific working processes of the system, the device and the unit described above, reference may be made to the corresponding processes in the foregoing method embodiments, and details are not described here again.
The embodiment of the present application further provides a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, and when the instructions are executed by a computer, the computer is enabled to execute the data processing method provided by the foregoing embodiment.
The computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM), a register, a hard disk, an optical fiber, a CD-ROM, an optical storage device, a magnetic storage device, any suitable combination of the foregoing, or any other form of computer-readable storage medium known in the art. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. Of course, the storage medium may also be integral to the processor. The processor and the storage medium may reside in an Application Specific Integrated Circuit (ASIC). In embodiments of the present application, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The above description is only an embodiment of the present application, but the scope of the present application is not limited thereto, and any changes or substitutions within the technical scope of the present disclosure should be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims (10)

1. A data processing method, comprising:
under the condition of receiving a data processing request sent by a user terminal, determining a first node from candidate nodes based on a preset algorithm and an identifier carried by the data processing request;
sending a first request carrying the identifier to the first node;
and if the response information fed back by the first node or the processing failure information is not received within a preset time length, sending a second request carrying the identifier to the candidate nodes in a multicast mode, wherein the second request is used for indicating each node in the candidate nodes to search for target data corresponding to the identifier.
2. The data processing method according to claim 1, wherein the determining the first node from the candidate nodes based on a preset algorithm and an identifier carried by the data processing request comprises:
determining a first numerical value based on the preset algorithm, the identifier and the total node number of the candidate nodes;
determining the first node based on the first numerical value and a preset corresponding relation under the condition that the first numerical value and the number of enabled nodes in the candidate nodes meet a preset condition; the preset corresponding relation comprises numerical values corresponding to all the enabled nodes in the candidate nodes;
and under the condition that the first numerical value and the number of the enabled nodes do not meet the preset condition, determining a second numerical value based on the first numerical value and the number of the enabled nodes, and determining the first node according to the second numerical value and the preset corresponding relation.
3. The data processing method of claim 2, wherein the predetermined algorithm is a hash algorithm, and the determining a first value based on the predetermined algorithm, the identifier and the total node number of the candidate nodes comprises:
generating a hash value of the identifier according to the hash algorithm; performing a complementation operation on the hash value based on the total node number to obtain the first numerical value;
said determining a second value based on said first value and said number of enabled nodes comprises: and carrying out complementation operation on the first numerical value based on the number of the starting nodes to obtain the second numerical value.
4. The data processing method of claim 1, wherein the method further comprises:
receiving an offline request sent by a third node; the third node is any one enabling node in the candidate nodes;
determining whether the third node is a last node or not based on a preset corresponding relation; the preset corresponding relation comprises a numerical value corresponding to each starting node in the candidate nodes; the last node is a node with the maximum corresponding numerical value in the enabled nodes;
and under the condition that the third node is not the last node, adjusting the numerical value corresponding to the last node in the preset corresponding relationship to the numerical value corresponding to the third node, and deleting the mapping relationship between the third node and the numerical value corresponding to the third node.
5. The data processing method of claim 1, wherein the method further comprises:
receiving an offline request sent by a third node; the third node is any one enabling node in the candidate nodes;
adjusting the read-write state of the third node to be a readable but unwritable state;
and if the third node becomes a null node, adjusting the read-write state of the third node to be an unreadable and unwritable state.
6. The data processing method according to claim 1, wherein after sending the second request carrying the identifier to the candidate node in a multicast manner, the method further comprises:
and if the target data corresponding to the identifier is found in a second node except the first node, transferring the target data to the first node.
7. The data processing method of claim 2, wherein the candidate nodes include the enabled node and a node in a readable but unwritable state, excluding a node in an unreadable and unwritable state, the method further comprising:
if the starting nodes are increased or decreased, the preset corresponding relation is adjusted based on the number of the starting nodes, so that each second numerical value can correspond to the starting nodes through the preset corresponding relation.
8. A data processing apparatus, comprising:
the determining module is used for determining a first node from candidate nodes based on a preset algorithm and an identifier carried by a data processing request under the condition of receiving the data processing request sent by a user terminal;
a sending module, configured to send a first request carrying the identifier to the first node;
the sending module is further configured to send a second request carrying the identifier to the candidate node in a multicast manner if no response information fed back by the first node is received or no processing failure information is received within a preset time length, so as to instruct each node in the candidate node to search for target data corresponding to the identifier.
9. A data processing apparatus comprising a memory, a processor, a bus and a communication interface; the memory is used for storing computer execution instructions, and the processor is connected with the memory through the bus;
when the data processing apparatus is running, a processor executes the computer executable instructions stored by the memory to cause the data processing apparatus to perform the data processing method of any one of claims 1 to 7.
10. A computer-readable storage medium having stored therein instructions, which when executed by a computer, cause the computer to perform a data processing method according to any one of claims 1 to 7.
CN202211161353.0A 2022-09-23 2022-09-23 Data processing method, device, equipment and storage medium Active CN115277850B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211161353.0A CN115277850B (en) 2022-09-23 2022-09-23 Data processing method, device, equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211161353.0A CN115277850B (en) 2022-09-23 2022-09-23 Data processing method, device, equipment and storage medium

Publications (2)

Publication Number Publication Date
CN115277850A true CN115277850A (en) 2022-11-01
CN115277850B CN115277850B (en) 2023-01-03

Family

ID=83757045

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211161353.0A Active CN115277850B (en) 2022-09-23 2022-09-23 Data processing method, device, equipment and storage medium

Country Status (1)

Country Link
CN (1) CN115277850B (en)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106598488A (en) * 2016-11-24 2017-04-26 北京小米移动软件有限公司 Distributed data reading method and device
US20180302404A1 (en) * 2016-09-13 2018-10-18 Tencent Technology (Shenzhen) Company Limited Method for processing data request and system therefor, access device, and storage device
CN109375872A (en) * 2018-09-27 2019-02-22 腾讯科技(深圳)有限公司 Processing method, device and the equipment and storage medium of data access request

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180302404A1 (en) * 2016-09-13 2018-10-18 Tencent Technology (Shenzhen) Company Limited Method for processing data request and system therefor, access device, and storage device
CN106598488A (en) * 2016-11-24 2017-04-26 北京小米移动软件有限公司 Distributed data reading method and device
CN109375872A (en) * 2018-09-27 2019-02-22 腾讯科技(深圳)有限公司 Processing method, device and the equipment and storage medium of data access request

Also Published As

Publication number Publication date
CN115277850B (en) 2023-01-03

Similar Documents

Publication Publication Date Title
EP3410288B1 (en) Online upgrade method, device and system
US8332367B2 (en) Parallel data redundancy removal
CN103019960A (en) Distributed cache method and system
CN108737176B (en) Data gateway control method, electronic equipment, storage medium and architecture
CN111580963B (en) Load balancing method and device for storage cluster, computer equipment and storage medium
CN111639061A (en) Data management method, device, medium and electronic equipment in Redis cluster
WO2021114848A1 (en) Data reading and writing method and device for database
US20180123791A1 (en) Highly available and reliable secret distribution infrastructure
CN108228842B (en) Docker mirror image library file storage method, terminal, device and storage medium
US11134121B2 (en) Method and system for recovering data in distributed computing system
US10061725B2 (en) Scanning memory for de-duplication using RDMA
CN115277850B (en) Data processing method, device, equipment and storage medium
CN111885184A (en) Method and device for processing hot spot access keywords in high concurrency scene
EP3964996A1 (en) Database access method and device
CN112631994A (en) Data migration method and system
US20230273801A1 (en) Method for configuring compute mode, apparatus, and computing device
US9405470B2 (en) Data processing system and data processing method
CN115686746A (en) Access method, task processing method, computing device, and computer storage medium
CN113900815A (en) High-bandwidth memory access method and device of heterogeneous many-core processor
CN109088913B (en) Method for requesting data and load balancing server
CN115220640A (en) Method, electronic device and computer program product for processing data
CN114968505A (en) Task processing system, method, device, apparatus, storage medium, and program product
CN111857548A (en) Data reading method, device and system
CN116915510B (en) Distributed storage system based on high-speed encryption algorithm
US8806504B2 (en) Leveraging performance of resource aggressive applications

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant