CN114201117A - Cache data processing method and device, computer equipment and storage medium - Google Patents

Cache data processing method and device, computer equipment and storage medium Download PDF

Info

Publication number
CN114201117A
CN114201117A CN202111582022.XA CN202111582022A CN114201117A CN 114201117 A CN114201117 A CN 114201117A CN 202111582022 A CN202111582022 A CN 202111582022A CN 114201117 A CN114201117 A CN 114201117A
Authority
CN
China
Prior art keywords
node
data
cache data
cache
request
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202111582022.XA
Other languages
Chinese (zh)
Other versions
CN114201117B (en
Inventor
魏传程
胡玉鹏
李红卫
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Suzhou Inspur Intelligent Technology Co Ltd
Original Assignee
Suzhou Inspur Intelligent Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Suzhou Inspur Intelligent Technology Co Ltd filed Critical Suzhou Inspur Intelligent Technology Co Ltd
Priority to CN202111582022.XA priority Critical patent/CN114201117B/en
Publication of CN114201117A publication Critical patent/CN114201117A/en
Application granted granted Critical
Publication of CN114201117B publication Critical patent/CN114201117B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0614—Improving the reliability of storage systems
    • G06F3/0617—Improving the reliability of storage systems in relation to availability
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
    • G06F3/0656—Data buffering arrangements
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The application provides a processing method and device of cache data, computer equipment and a storage medium, and relates to the technical field of networks. The method includes the steps that a cache client sends a reading request of cache data to a main node of a node list where the cache data are located, the main node and each replica node in the node list return different data to the cache client according to the reading request, the cache client determines that the main node fails according to first data information returned by any one replica node, a standby node is determined from the replica nodes in the node list, the reading request corresponding to the cache data is sent to the standby node again, and the standby node returns corresponding cache data to the cache client according to the reading request. The technical scheme can avoid the problems of cache data loss and incapability of acquiring cache data caused by the failure of the main node, and improves the reliability and high availability of the system.

Description

Cache data processing method and device, computer equipment and storage medium
Technical Field
The present application relates to the field of network technologies, and in particular, to a method and an apparatus for processing cache data, a computer device, and a storage medium.
Background
In openstack platform design, all components can be deployed independently, but as a unified cloud platform, it still needs a unified authentication system for user management. Along with the continuous expansion of the scale of the cloud platform, more and more components interact with the keystone, the database is generally accessed during the keystone authentication, memcached is generally adopted as an authentication data cache to avoid overlarge pressure of the database, and when the authentication is performed, cache data is obtained from the memcached, so that the interaction speed between the cloud platform components is accelerated.
The basic principle of the openstack platform for user authentication is as follows: when a certain component accesses the keystone for the first time, the memcached is accessed first, if cache data corresponding to the authentication information is not stored, a preset database is accessed, the authentication information is cached into the memcached from the database, and when the component accesses the keystone again, the authentication information can be directly obtained from the memcached. However, in the memcached design, the design is only a simple multi-node design, no communication exists between the service ends, the service ends are pseudo-distributed, the realization of the distribution is realized by the client, and the client realizes that a distributed algorithm stores data to different memcached service ends. When a problem occurs in a certain node, the client cannot access the information cache, so that the performance is reduced, and a large amount of time is consumed for data synchronization and caching again when the node is recovered.
Disclosure of Invention
In view of the foregoing, it is necessary to provide a method, an apparatus, a computer device, and a storage medium for processing cache data, which can improve the high availability of a memached cluster.
In a first aspect, the present application provides a method for processing cache data, where the method includes:
sending a reading request of cache data to a main node of the node list; the node list comprises a main node and at least one replica node;
receiving first data information returned by any copy node according to the reading request;
and determining a standby node from the replica nodes and resending a reading request to the standby node to acquire cache data read by the standby node according to a request returned by the reading request if the main node is determined to have a fault according to the first data information.
In one embodiment, after the step of retransmitting the read request to the standby node, the method further includes:
and if the main node is determined to be normal according to the first data information, sending a cache data reading request to the main node so as to receive second data information returned by the main node according to the reading request, wherein the second data information comprises cache data requested to be read.
In one embodiment, before the step of sending a read request for cached data to the master node of the node list, the method further includes:
acquiring the number of copies corresponding to preset cache data, and determining a node list corresponding to the cache data through a Hash algorithm according to the number of copies; the length of the node list is the same as the number of copies;
writing the cache data into each node of the node list; the first node in the node list is a master node, and the other nodes are replica nodes.
In one embodiment, before the step of receiving the first data information returned by any replica node according to the read request, the method further includes:
establishing communication connection between a main node and a plurality of replica nodes;
the method comprises the steps that a main node is connected through communication to announce the service state of the main node to each replica node at a preset time interval; the service state includes a fault state or a normal state.
In one embodiment, before the step of receiving the second data information returned by the master node according to the read request, the method further includes:
determining whether the host node stores cache data;
and if the main node does not store the cache data, sending a write-in request of the cache data to each node in the node list, so that the node asynchronously writes the cache data in the preset database according to the write-in request.
In one embodiment, before the step of determining that the master node recovers to normal according to the first data information, the method further includes:
in the process of data synchronization of the main node, the main node acquires cache data from the replica node to perform data synchronization;
and when the data synchronization of the main node is completed, the service state of the main node is announced to each replica node.
In one embodiment, before the step of sending a read request for cached data to the master node of the node list, the method further includes:
receiving a cache data acquisition request sent by an application; the obtaining request comprises a target identifier corresponding to the cache data;
and determining a node list corresponding to the cache data according to the target identification.
In a second aspect, the present application further provides a processing apparatus for caching data, where the apparatus includes:
a read request sending module, configured to send a read request for cached data to a master node of the node list; the node list comprises a main node and a plurality of replica nodes;
the data information receiving module is used for receiving first data information returned by any copy node according to the reading request;
and the first cache data acquisition module is used for determining a standby node from the replica nodes if the main node is determined to have a fault according to the first data information, and resending the reading request to the standby node so as to acquire cache data which is returned by the standby node according to the reading request and is requested to be read.
In a third aspect, the present application further provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the steps of the processing method for caching data mentioned in the first aspect when executing the computer program.
In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, realizes the steps of the processing method for caching data mentioned in the first aspect.
According to the processing method and device for the cache data, the computer device and the storage medium, when the cache client needs to perform cache data reading operation, a cache client sends a cache data reading request to a main node of a node list where the cache data is located, wherein each node in the node list, namely the cache data corresponding to the main node and at least one copy node, should be consistent. The main node and each copy node in the node list can return different data to the cache client according to the reading request, the cache client determines that the main node fails according to first data information returned by any copy node, a standby node is determined from the copy nodes in the node list, the reading request corresponding to the cache data is sent to the standby node again, and the standby node returns corresponding cache data to the cache client according to the reading request, so that the problems that the cache data is lost and the cache data cannot be obtained due to the failure of the main node are solved, and the reliability and the high availability of the system are improved.
In addition, in the technical scheme, when the cache client sends a cache data reading request to the server, only one node, such as a main node or a standby node, returns cache data, so that the network pressure in the system cluster is reduced, and unnecessary network bandwidth waste is avoided. In addition, the technical scheme also supports the number of copies of the user-defined cache data, a plurality of nodes are selected through the Hash algorithm in a balanced mode to obtain a node list with the length being the same as that of the number of copies, the cache data are stored in each node of the corresponding node list, and the memory utilization efficiency of the memcached cluster is improved.
Drawings
Fig. 1 is a schematic view of an application scenario of a processing method for caching data in an embodiment;
FIG. 2 is a flow diagram of a method for processing cached data in one embodiment;
FIG. 3 is another flowchart of a method for processing cached data according to an embodiment;
FIG. 4 is a block diagram of a processing device for caching data according to an embodiment;
FIG. 5 is a diagram illustrating an internal structure of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
The names to which this application relates will be explained first below.
OpenStack: the cloud computing management platform is an open-source cloud computing management platform project, is a combination of a series of software open-source projects, and aims to provide the cloud computing management platform which is simple to implement, can be expanded in a large scale, is rich and has a unified standard.
Keystone: the components used for unified authentication in the openstack platform manage authority authentication and authorization between the user and each component in the cloud platform.
Memcached: a high-performance distributed memory object cache system can be used for storing data in various formats, including images, videos, files, database retrieval results and the like, by maintaining a uniform and huge hash table in a memory. The method is simple in that data are called into a memory and then are read from the memory, so that the reading speed is greatly improved, the method is used for caching authentication information between a keystone and other components in OpenStack, the database is prevented from being accessed during authentication every time, the performance of a cloud platform is improved, and the method plays an important role in large-scale cloud platform design and construction.
Distributed Cache (Distributed Cache), that is, Cache data cached in a memory of a Distributed system, disperses contents to be cached into physically isolated Cache service nodes for storage, and can provide high availability, high performance and scalability for an event-driven application program. Obtaining data from the cache may increase access speed relative to obtaining data from a persistent database, thereby increasing performance of third party applications. Distributed caching mainly comprises two types, namely server-side distributed caching and client-side distributed caching.
Memcached is a client distributed cache, a client distributed cache structure is composed of a client and a server, the server comprises a plurality of service nodes, the server is not responsible for distributing data and synchronizing data, but distributes data of different keys to different servers through algorithm setting of the client to form an expandable client distributed cache system, however, the data cache mode lacks a fault-tolerant mechanism, and when a certain data cache service node is unavailable, the system loses part of data, thereby affecting normal use of a service system.
The cache data processing method, the cache data processing device, the computer equipment and the storage medium can improve the high availability of memcached and solve the technical problems in the prior art.
In order to facilitate understanding of the application scheme, an application environment to which the embodiments of the present application are applicable is first described.
Fig. 1 is a schematic view of an Application scenario of a processing method for caching data in an embodiment, and as shown in fig. 1, the Application environment aims at data caching processing of a memcached cluster and includes an Application (APP), a cache client, and a plurality of service nodes;
different applications need to acquire corresponding cache data when executing specific functions so as to quickly complete a certain operation, and send cache data acquisition requests to corresponding cache clients so as to request to acquire a certain cache data.
The cache client is used for determining a plurality of target nodes for storing cache data in a plurality of service nodes of the memcached cluster according to the preset copy number of the cache data, and obtaining a node list obtained by the plurality of target nodes, wherein the node list comprises a main node and at least one copy node, and under a normal condition, corresponding cache data on the main node and the copy node are kept consistent. As shown in fig. 1, a first node list corresponding to cache data requested to be read by application a includes a service node 1 and a service node 3, where the service node 1 is a master node and the service node 3 is a replica node; the second node list corresponding to the cache data requested to be read by the application B includes the service node 2 and the service node 5, where the service node 5 is a master node and the service node 2 is a replica node. In addition, a certain service node may be a master node or a replica node, for example, the service node 3 is a replica node of the first node list and is also a master node of a third node list corresponding to the cache data requested to be read by the application C. In the technical scheme, communication connection can be established among all service nodes in the same node list so as to perform self service state notification and buffer data synchronization.
The cache client is further configured to receive an obtaining request of the cache data of the application, and determine a node list where the cache data requested to be obtained by the application is located according to the obtaining request. Each service node (hereinafter, may be simply referred to as a "node") on the node list corresponds to a logical cache server for caching data written from the persistent database. Specifically, each node on the node list asynchronously writes corresponding cache data from a preset persistent database according to a write request of the cache client. For example, referring to fig. 1, the cache client B sends write requests to the service node 2 and the service node 5 in the second node list to write the cache data corresponding to the application B from the preset database N into the service node 2 and the service node 5.
When the main node fails, the cache client determines a standby node from the replica nodes in the node list to acquire cache data from the standby node, and when the main node returns to normal, the main node acquires newly added cache data from the replica nodes to perform data synchronization.
The following describes the technical solutions of the present application and how to solve the above technical problems with specific embodiments. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments.
Embodiments of the present application will be described below with reference to the accompanying drawings.
Fig. 2 is a flowchart of a processing method for caching data in an embodiment, where the processing method for caching data may be executed in a computer device, and optionally, in this embodiment, may be executed by a caching client.
Specifically, as shown in fig. 2, the method for processing the cache data may include the following steps:
s210, sending a reading request of cache data to a main node of a node list; the node list comprises a main node and at least one replica node;
in this embodiment, the first node in the node list is taken as a master node, and the remaining nodes are replica nodes, where the replica nodes may be one or multiple replica nodes. And the cache data corresponding to the main node and each corresponding copy node are consistent.
Optionally, the node lists corresponding to different cache data are different, and in the same node list, there are a master node and at least one replica node, where the master node is often located at the first node position in the node list. A node may be a master node of a node list corresponding to a certain cached data, and may also be a replica node of a node list corresponding to another cached data.
S220, receiving first data information returned by any copy node according to the reading request.
In this embodiment, the master node returns cache data to the cache client, and each replica node returns the service state of the replica node and the service state of the master node to the cache client. The first data information includes a service state of the master node, which is notified to each replica node by the master node.
And S230, determining a standby node from the replica node if the main node is determined to have a fault according to the first data information, and repeatedly sending a reading request to the standby node to obtain cache data which is returned by the standby node according to the reading request and is requested to be read.
When the main node fails, the cache client cannot receive the cache data returned by the main node, but can receive the first data information returned by the replica node. The cache client acquires the service state of the main node according to the first data information returned by any one replica node and the first data information, and then determines that the main node fails.
When the cache client determines that the master node fails, optionally, one of the replica nodes in the node list is selected as a standby node. The standby node stores complete cache data. And the cache client sends a reading request of the cache data to the standby node again, and the standby node receives the reading request, inquires corresponding cache data according to the reading request and returns the cache data to the cache client.
Optionally, the first duplicate node in the node list is determined as a standby node. Specifically, when the cache data is written into each node of the node list in an asynchronous writing mode, because the primary node firstly writes the cache data and secondly writes the cache data into the first replica node, when the primary node fails, the first replica node is determined as a standby node, and the reliability and the high availability of the cache system can be improved.
The processing method for the cache data provided by this embodiment may be executed by a cache client, where the cache client sends a cache data read request to a master node in a node list; the node list comprises a main node and at least one replica node; the cache client receives first data information returned by any copy node according to the reading request; and determining a standby node from the replica node according to the first data information when the main node is determined to have a fault, and retransmitting the reading request to the standby node to obtain the cache data which is returned by the standby node according to the reading request and is requested to be read, so that the problem of cache data loss or unavailability caused by the node fault is avoided, and the high availability of the memcached cluster is improved.
Fig. 3 is another flowchart of a processing method of cached data according to an embodiment, and as shown in fig. 3, the processing method of cached data may include the following steps:
s310, sending a reading request of cache data to a main node of the node list; wherein the node list includes a master node and at least one replica node.
And S320, receiving first data information returned by any copy node according to the reading request.
S330, judging whether the main node fails according to the first data information, if so, executing a step S340; otherwise, step S380 is executed.
S340, determining a standby node from the replica nodes, and resending the reading request to the standby node.
The implementation process of steps S310 to S340 can refer to steps S210 to S230, which are not described in detail herein.
And S350, obtaining the cache data which is returned by the standby node according to the reading request and is requested to be read, and receiving the first data information returned by any copy node according to the reading request.
When the cache client sends a reading request of the cache data to the standby node, any copy node returns first data information to the cache client according to the reading request, wherein the first data information comprises the service state of the main node and can also comprise the service state of the copy node.
And S360, judging whether the main node is recovered to be normal or not according to the first data information, if so, executing the step S370, otherwise, returning to execute the step S340.
S370, send a read request for the cached data to the master node.
And S380, receiving second data information returned by the main node according to the reading request, wherein the second data information comprises cache data requested to be read.
And if the main node is determined to be normal according to the first data information, sending a reading request of the cache data to the main node so as to obtain second data information returned by the main node according to the reading request, wherein the second data information comprises the cache data requested to be read.
Specifically, the cache client determines that the master node has recovered to be normal according to the first data information returned by each replica node, where the recovery of the master node to be normal means that the master node can work normally and the cache data is synchronized. The cache client sends a cache data reading request to the host node, and returns second data information to the cache client, optionally, the host node queries cache data requested to be read after receiving the reading request, and packages the cache data and returns the cache data to the cache client, where the second data information includes the cache data requested to be read, and may also include information such as a service state of the host node itself.
For ease of understanding, the above-described scheme will be exemplified.
Firstly, a cache client sends a cache data reading request to a master node in a node list, the master node and at least one replica node in the node list return different data information according to the reading request, under normal conditions, the master node returns cache data requested to be read to the cache client, and the replica node returns a service state of the master node to the cache client.
When the cache client cannot receive the cache data returned by the main node or the main node is determined to have a fault according to the service state of the main node returned by any one replica node, at this time, the cache client determines a standby node from each replica node in the node list and retransmits a cache data reading request to the standby node, after receiving the reading request, the standby node inquires the cache data according to the reading request and returns the cache data to the cache client, and the rest replica nodes return the service state of the main node to the cache client according to the reading request. Optionally, the standby node may return the service state of the host node to the cache client, in addition to returning the cache data to the cache client.
Before the main node does not recover to normal, when the cache client reads the cache data for the next time, sending a read request of the cache data to the standby node so as to obtain the cache data from the standby node. When the main node returns to normal, the cache client sends a reading request of cache data to the standby node, and the main node is determined to return to normal according to the received service state of the main node returned by the replica node. In an embodiment, the cache client receives cache data returned by the current standby node, and when the next cache data reading is performed, the cache client sends a cache data reading request to the host node. In another embodiment, after determining that the primary node is normal, the cache client may resend the read request of the cache data to the primary node to read the cache data from the primary node, and the standby node is restored to the use of the replica node.
In the technical scheme, the cache data in the main node and the cache data in the replica nodes in the node list are consistent, when the main node fails and cannot return the cache data requested to be read, the backup node is determined from the replica nodes in the node list, and the read request is retransmitted to the backup node to obtain the cache data from the backup node, so that cache data loss caused by node failure is avoided, and the reliability and the high availability of the system are improved.
In order to more clearly illustrate the technical solution of the present application, the following further describes an implementation manner of multiple steps of the processing method for caching data.
In an embodiment, before the step S210 of sending the read request for caching data to the master node of the node list, the method may further include the following steps:
s110, acquiring the number of copies corresponding to preset cache data, and determining a node list corresponding to the cache data through a Hash algorithm according to the number of copies.
The length of the node list is the same as the number of copies.
The user may preset the number of copies of the corresponding cache data according to actual needs, and if the number of copies of the cache data a is 3 and the number of copies of the cache data B is 4, the cache client determines, according to the preset number of copies corresponding to the cache data, a plurality of nodes storing the corresponding cache data by using a hash algorithm, and sorts the nodes to obtain a corresponding node list. The length of the node list, i.e. the number of nodes in the node list, is the same as the number of copies.
In the embodiment, each node to be written in the cache data is determined according to the preset copy number of the cache data by using the hash algorithm, so that the load balance of each node in the cluster is ensured.
And S120, writing the cache data into each node of the node list.
The first node in the node list is a master node, and the other nodes are replica nodes.
Optionally, in an embodiment, the cache data may be asynchronously written into each node in the node list from a preset persistent database, and the cache data may be sequentially written into each node in the node list according to the sequence of each node in the node list and stored in each node. Since the master node is located at the first position of the node list, the cache data is preferentially written into the master node, and then the cache data is written into the second node, namely the first replica node, until the cache data is written into all the nodes in the node list.
In other embodiments, the cached data may also be synchronously written into each node of the node list from a preset persistent database.
It should be noted that, in the related art, for example, when data is stored in memcached by memcached ha, the memcached ha is automatically synchronized to other memcached nodes, so that the nodes store the full amount of cached data to ensure the consistency of the cached data, the implementation manner provided by the present solution may select a plurality of nodes from the memcached cluster as the master node and the replica node to form a node list according to the difference of the cached data, a certain node may be a master node of a certain cached data or a replica node of another cached data, and a certain node is only a master node or a replica node of a certain cached data, so that a certain node only stores part of cached data, and the problem of large resource consumption caused by that each node stores the full amount of cached data is avoided; in addition, according to the scheme of the application, the number of copies of the cache data is set at the cache client independently, so that multi-copy caching of the cache data can be realized, and a node list corresponding to each cache data is determined by adopting a Hash algorithm, so that the load balance of each node in a cluster is ensured, and the load imbalance caused by the concentrated writing of the cache data into partial nodes is avoided.
In an embodiment, before receiving the first data information returned by any replica node according to the read request in step S220, the method may further include the following steps:
s130, establishing communication connection between the main node and the plurality of replica nodes.
And establishing communication connection between each node in the node list so as to realize the establishment of communication connection between the main node and each replica node, and also establishing communication connection between each replica node.
And S140, informing the service state of the main node to the replica node at a preset time interval through the communication connection main node.
The service state includes a fault state or a normal state.
After the communication connection is established, the main node notifies the service state of each replica node in the node list at a preset time interval so as to inform that each replica node is in a normal state or a fault state currently. When the host node is in a normal state, the host node may return the cache data to the cache client, and when the host node is in a failure state, the host node may not return the cache data to the cache client.
In an embodiment, before the step of receiving the second data information returned by the master node according to the read request, the method may further include the following steps:
s301, determining whether the cache data is stored in the main node.
Optionally, whether the host node stores the corresponding cache data may be determined by whether the host node queries the identifier corresponding to the cache data requested to be read, or by information such as a write log of the cache data requested to be read by the host node. And if the identifier corresponding to the cache data cannot be inquired on the main node, or the main node is determined not to write the cache data according to the write log of the main node to the cache data, determining that the cache data is not stored in the main node.
And S302, if the main node does not store the cache data, sending a write-in request of the cache data to each node in the node list, so that the nodes asynchronously write the cache data into the preset database according to the write-in request.
When determining that the host node does not store the cache data requested to be read, the cache client sends a write-in request of the cache data to each node in the node list, where the write-in request may include an identifier of the cache data to be written in.
And each node of the node list determines the position of the cache data to be written according to the write request, acquires the cache data to be written from a preset persistent database, and asynchronously writes the cache data to each node of the node list. Since the master node is located at the first node position in the node list, the cache data is first written into the master node, and then sequentially written into each replica node in the node list.
In an embodiment, before determining that the master node returns to normal according to the first data information, the method may further include the following steps:
s401, in the process of data synchronization of the main node, the main node obtains cache data from the replica node to perform data synchronization.
Because the main node is in a fault state, the newly added cache data cannot be obtained in time, and the replica node can still obtain the newly added cache data from the preset persistent database due to normal work, the corresponding cache data of the main node and other replica nodes are inconsistent. When the master node returns to normal, data synchronization is firstly carried out, and at the moment, the master node acquires newly added cache data from other replica nodes which are in communication connection with the master node so as to carry out data synchronization, so that the cache data corresponding to the master node is consistent with the cache data corresponding to the other replica nodes.
Further, when the data synchronization of the master node is completed, the service state of the master node is notified to the replica node, that is, the state that the master node is recovered to be normal and the data synchronization is completed is notified.
Specifically, when the data synchronization of the master node is completed, that is, the data of the master node and the data of the other replica nodes are restored to be consistent, the service state of the master node is notified to the replica nodes, that is, the current master node of the replica nodes is notified to be in a normal state. When the data synchronization of the master node completes the announcement of the service state of the master node to the replica node, it can be understood that: when the data synchronization of the main node is completed, the service state of the main node is immediately announced to the replica node, or after the data synchronization of the main node is completed, the service state of the main node is announced to the replica node at a time point corresponding to a preset time interval.
In an embodiment, before the step of sending the cache data read request to the master node of the node list in step S210, the method further includes:
s2001, receiving a cache data acquisition request sent by an application; the obtaining request comprises a target identifier corresponding to the cache data.
In an embodiment, when an application needs a certain cache data when executing a certain function, the application sends an obtaining request for obtaining the cache data to a cache client, where the obtaining request may include a target identifier corresponding to the cache data, where the target identifier is used to indicate a location of the cache data, and the target identifier may be represented by a key value or a hash value, or may be identified by other identification methods.
And S2002, determining a node list corresponding to the cache data according to the target identifier.
When the cache client receives an obtaining request of cache data sent by an application, the target identification is analyzed to determine a node list corresponding to the cache data, and then a reading request of the cache data is sent to a main node in the node list so as to request to obtain the cache data obtained by the application request.
The above examples are merely used to assist in explaining the technical solutions of the present disclosure, and the drawings and specific flows related thereto do not constitute a limitation on the usage scenarios of the technical solutions of the present disclosure.
A related embodiment of the processing means for caching data is explained in detail below.
Fig. 4 is a schematic structural diagram of a processing apparatus for caching data in an embodiment, where the processing apparatus for caching data may be executed on a computer device, and in the embodiment, the processing apparatus may be executed by a caching client.
As shown in fig. 4, the processing apparatus 200 for caching data may include: a read request sending module 210, a data information receiving module 220 and a cache data obtaining module 230;
the read request sending module 210 is configured to send a read request for caching data to a master node in a node list; the node list comprises a main node and a plurality of replica nodes;
a data information receiving module 220, configured to receive first data information returned by any replica node according to the read request;
the first cache data obtaining module 230 is configured to determine, according to the first data information, that the primary node fails, determine a standby node from the replica nodes, and resend the read request to the standby node, so as to obtain cache data, which is returned by the standby node according to the read request and is requested to be read.
The processing device for the cache data solves the problems that the cache data are lost and cannot be obtained when the main node is in failure, and improves the reliability and high availability of the system.
In one embodiment, the processing apparatus 200 for caching data further includes: and the second cache data acquisition module is used for determining that the main node is normal according to the first data information, and then sending a cache data reading request to the main node so as to receive second data information returned by the main node according to the reading request, wherein the second data information comprises cache data requested to be read.
In one embodiment, the processing apparatus 200 for caching data further includes: the cache data writing module comprises a copy number acquisition unit and a cache data writing unit; the system comprises a copy number acquisition unit, a node storage unit and a cache data processing unit, wherein the copy number acquisition unit is used for acquiring the copy number corresponding to preset cache data and determining a node list corresponding to the cache data through a Hash algorithm according to the copy number; the length of the node list is the same as the number of copies; the cache data writing unit is used for writing the cache data into each node of the node list; the first node in the node list is a master node, and the other nodes are replica nodes.
In one embodiment, the processing apparatus 200 for caching data further includes: the service state notification module comprises a communication connection establishing unit and a service state first notification unit; the communication connection establishing unit is used for establishing communication connection between the main node and the plurality of replica nodes; the service state first notification unit is used for notifying the service state of each replica node to each replica node at a preset time interval through the communication connection main node; the service state includes a fault state or a normal state.
In one embodiment, the processing apparatus 200 for caching data further includes: the cache data writing module comprises a cache data determining unit and a cache data writing unit; the cache data determining unit is used for determining whether the host node stores cache data; and the cache data writing unit is used for sending a writing request of the cache data to each node in the node list if the host node does not store the cache data, so that the nodes asynchronously write the cache data into the preset database according to the writing request.
In one embodiment, the processing apparatus 200 for caching data further includes: the data synchronization module comprises a data synchronization unit and a service state second notification unit; the data synchronization unit is used for acquiring cache data from the replica node by the main node for data synchronization in the data synchronization process of the main node; and the service state second informing unit is used for informing the service state of each replica node when the data synchronization of the main node is completed.
In one embodiment, the processing apparatus 200 for caching data further includes: a node list determination module including an acquisition request receiving unit and a node list determination unit; the device comprises an acquisition request receiving unit, a cache data processing unit and a cache data processing unit, wherein the acquisition request receiving unit is used for receiving an acquisition request of cache data sent by an application; the obtaining request comprises a target identifier corresponding to the cache data; and the node list determining unit is used for determining a node list corresponding to the cache data according to the target identifier.
The processing apparatus for cache data of this embodiment can execute the processing method for cache data shown in the foregoing embodiments of this application, and the implementation principles thereof are similar, and are not described herein again.
The modules in the processing device for caching data may be implemented in whole or in part by software, hardware, and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a server, the internal structure of which may be as shown in fig. 5. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a processing method of caching data.
Those skilled in the art will appreciate that the architecture shown in fig. 5 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is provided, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the following steps when executing the computer program:
sending a reading request of cache data to a main node of the node list; the node list comprises a main node and at least one replica node;
receiving first data information returned by any copy node according to the reading request;
and determining a standby node from the replica nodes and resending a reading request to the standby node to acquire cache data read by the standby node according to a request returned by the reading request if the main node is determined to have a fault according to the first data information.
In one embodiment, the processor, when executing the computer program, further performs the steps of:
and if the main node is determined to be normal according to the first data information, sending a reading request of the cache data to the main node so as to receive second data information returned by the main node according to the reading request, wherein the second data information comprises the cache data requested to be read.
In one embodiment, the processor, when executing the computer program, further performs the steps of: acquiring the number of copies corresponding to preset cache data, and determining a node list corresponding to the cache data through a Hash algorithm according to the number of copies; the length of the node list is the same as the number of copies; writing the cache data into each node of the node list; the first node in the node list is a master node, and the other nodes are replica nodes.
In one embodiment, the processor, when executing the computer program, further performs the steps of: establishing communication connection between a main node and a plurality of replica nodes; the method comprises the steps that a main node is connected through communication to announce the service state of the main node to each replica node at a preset time interval; the service state includes a fault state or a normal state.
In one embodiment, the processor, when executing the computer program, further performs the steps of: determining whether the host node stores cache data; and if the main node does not store the cache data, sending a write-in request of the cache data to each node in the node list, so that the node asynchronously writes the cache data in the preset database according to the write-in request.
In one embodiment, the processor, when executing the computer program, further performs the steps of: in the process of data synchronization of the main node, the main node acquires cache data from the replica node to perform data synchronization; and when the data synchronization of the main node is completed, the service state of the main node is announced to each replica node.
In one embodiment, the processor, when executing the computer program, further performs the steps of: receiving a cache data acquisition request sent by an application; the obtaining request comprises a target identifier corresponding to the cache data; and determining a node list corresponding to the cache data according to the target identification.
In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of:
sending a reading request of cache data to a main node of the node list; the node list comprises a main node and at least one replica node;
receiving first data information returned by any copy node according to the reading request;
and determining a standby node from the replica nodes and resending a reading request to the standby node to acquire cache data read by the standby node according to a request returned by the reading request if the main node is determined to have a fault according to the first data information.
In one embodiment, the computer program when executed by the processor further performs the steps of: and if the main node is determined to be normal according to the first data information, sending a reading request of the cache data to the main node so as to receive second data information returned by the main node according to the reading request, wherein the second data information comprises the cache data requested to be read.
In one embodiment, the computer program when executed by the processor further performs the steps of: acquiring the number of copies corresponding to preset cache data, and determining a node list corresponding to the cache data through a Hash algorithm according to the number of copies; the length of the node list is the same as the number of copies; writing the cache data into each node of the node list; the first node in the node list is a master node, and the other nodes are replica nodes.
In one embodiment, the computer program when executed by the processor further performs the steps of: establishing communication connection between a main node and a plurality of replica nodes; the method comprises the steps that a main node is connected through communication to announce the service state of the main node to each replica node at a preset time interval; the service state includes a fault state or a normal state.
In one embodiment, the computer program when executed by the processor further performs the steps of: determining whether the host node stores cache data; and if the main node does not store the cache data, sending a write-in request of the cache data to each node in the node list, so that the node asynchronously writes the cache data in the preset database according to the write-in request.
In one embodiment, the computer program when executed by the processor further performs the steps of: in the process of data synchronization of the main node, the main node acquires cache data from the replica node to perform data synchronization; and when the data synchronization of the main node is completed, the service state of the main node is announced to each replica node.
In one embodiment, the computer program when executed by the processor further performs the steps of: receiving a cache data acquisition request sent by an application; the obtaining request comprises a target identifier corresponding to the cache data; and determining a node list corresponding to the cache data according to the target identification.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware related to instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, the computer program can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
It should be understood that, although the steps in the above-described flowcharts are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in the above figures may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of performing the sub-steps or stages is not necessarily sequential, but may be performed alternately or alternately with other steps or at least some of the sub-steps or stages of other steps.
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above examples only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims (10)

1. A processing method for caching data, the method comprising:
sending a reading request of cache data to a main node of the node list; wherein the node list comprises a master node and at least one replica node;
receiving first data information returned by any one copy node according to the reading request;
and if the main node is determined to have a fault according to the first data information, determining a standby node from the replica nodes, and retransmitting the reading request to the standby node to obtain cache data which is returned by the standby node according to the reading request and is requested to be read.
2. The method of claim 1, wherein after the step of resending the read request to the standby node, further comprising:
and if the main node is determined to be normal according to the first data information, sending a reading request of cache data to the main node so as to receive second data information returned by the main node according to the reading request, wherein the second data information comprises the cache data requested to be read.
3. The method of claim 1, wherein the step of sending a read request to cache data to a home node of the node list is preceded by the step of:
acquiring the number of copies corresponding to preset cache data, and determining a node list corresponding to the cache data through a Hash algorithm according to the number of copies; wherein the length of the node list is the same as the number of copies;
writing cache data into each node of the node list; and the first node in the node list is a main node, and the rest nodes are replica nodes.
4. The method according to claim 3, before the step of receiving the first data information returned by any one of the replica nodes according to the read request, further comprising:
establishing communication connection between the main node and a plurality of replica nodes;
the communication connection is used for informing the service state of each replica node to the main node at a preset time interval; the service state includes a fault state or a normal state.
5. The method according to claim 2, wherein the step of receiving the second data information returned by the master node according to the read request is preceded by the step of:
determining whether the host node stores the cache data;
and if the host node does not store the cache data, sending a write-in request of the cache data to each node in the node list, so that the node asynchronously writes the cache data in a preset database according to the write-in request.
6. The method of claim 2, wherein before the step of determining that the master node is normal according to the first data information, the method further comprises:
in the process of data synchronization of the main node, the main node acquires newly added cache data from the replica node to perform data synchronization;
and when the data synchronization of the main node is completed, notifying the service state of each replica node.
7. The method of any of claims 1 to 6, wherein the step of sending a read request to cache data to a home node of the node list is preceded by the step of:
receiving a cache data acquisition request sent by an application; the obtaining request comprises a target identifier corresponding to the cache data;
and determining a node list corresponding to the cache data according to the target identification.
8. A processing apparatus for caching data, the apparatus comprising:
a read request sending module, configured to send a read request for cached data to a master node of the node list; the node list comprises a main node and a plurality of replica nodes;
a data information receiving module, configured to receive first data information returned by any one of the replica nodes according to the read request;
and the first cache data acquisition module is used for determining a standby node from the replica nodes if the main node is determined to have a fault according to the first data information, and resending the reading request to the standby node to acquire cache data requested to be read, which is returned by the standby node according to the reading request.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the method of any of claims 1 to 7 are implemented when the computer program is executed by the processor.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.
CN202111582022.XA 2021-12-22 2021-12-22 Processing method and device for cache data, computer equipment and storage medium Active CN114201117B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111582022.XA CN114201117B (en) 2021-12-22 2021-12-22 Processing method and device for cache data, computer equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111582022.XA CN114201117B (en) 2021-12-22 2021-12-22 Processing method and device for cache data, computer equipment and storage medium

Publications (2)

Publication Number Publication Date
CN114201117A true CN114201117A (en) 2022-03-18
CN114201117B CN114201117B (en) 2023-09-01

Family

ID=80655954

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111582022.XA Active CN114201117B (en) 2021-12-22 2021-12-22 Processing method and device for cache data, computer equipment and storage medium

Country Status (1)

Country Link
CN (1) CN114201117B (en)

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102609454A (en) * 2012-01-12 2012-07-25 浪潮(北京)电子信息产业有限公司 Replica management method for distributed file system
CN106331166A (en) * 2016-10-11 2017-01-11 杭州宏杉科技有限公司 Access method and device of storage resource
US20190095293A1 (en) * 2016-07-27 2019-03-28 Tencent Technology (Shenzhen) Company Limited Data disaster recovery method, device and system
CN110351313A (en) * 2018-04-02 2019-10-18 武汉斗鱼网络科技有限公司 Data cache method, device, equipment and storage medium
CN112148798A (en) * 2020-10-10 2020-12-29 腾讯科技（深圳）有限公司 Data processing method and device applied to distributed system
CN112671554A (en) * 2020-11-27 2021-04-16 苏州浪潮智能科技有限公司 Node fault processing method and related device
WO2021226905A1 (en) * 2020-05-14 2021-11-18 深圳市欢太科技有限公司 Data storage method and system, and storage medium

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102609454A (en) * 2012-01-12 2012-07-25 浪潮(北京)电子信息产业有限公司 Replica management method for distributed file system
US20190095293A1 (en) * 2016-07-27 2019-03-28 Tencent Technology (Shenzhen) Company Limited Data disaster recovery method, device and system
CN106331166A (en) * 2016-10-11 2017-01-11 杭州宏杉科技有限公司 Access method and device of storage resource
CN110351313A (en) * 2018-04-02 2019-10-18 武汉斗鱼网络科技有限公司 Data cache method, device, equipment and storage medium
WO2021226905A1 (en) * 2020-05-14 2021-11-18 深圳市欢太科技有限公司 Data storage method and system, and storage medium
CN112148798A (en) * 2020-10-10 2020-12-29 腾讯科技（深圳）有限公司 Data processing method and device applied to distributed system
CN112671554A (en) * 2020-11-27 2021-04-16 苏州浪潮智能科技有限公司 Node fault processing method and related device

Also Published As

Publication number Publication date
CN114201117B (en) 2023-09-01

Similar Documents

Publication Publication Date Title
CN111131451A (en) Service processing system and service processing method
US10114848B2 (en) Ensuring the same completion status for transactions after recovery in a synchronous replication environment
CN110581782B (en) Disaster tolerance data processing method, device and system
CN104935654A (en) Caching method, write point client and read client in server cluster system
CN110768873B (en) Distributed heartbeat detection method, system and device and computer equipment
CN111049928B (en) Data synchronization method, system, electronic device and computer readable storage medium
CN104468651B (en) Distributed more copy data storage methods and device
CN107623703B (en) Synchronization method, device and system for Global Transaction Identifier (GTID)
EP4213038A1 (en) Data processing method and apparatus based on distributed storage, device, and medium
CN113010496B (en) Data migration method, device, equipment and storage medium
CN112052230B (en) Multi-machine room data synchronization method, computing device and storage medium
CN111159233A (en) Distributed caching method, system, computer device and storage medium
CN114900449B (en) Resource information management method, system and device
CN115277727A (en) Data disaster recovery method, system, device and storage medium
CN106951443B (en) Method, equipment and system for synchronizing copies based on distributed system
CN110351122B (en) Disaster recovery method, device, system and electronic equipment
CN114201117B (en) Processing method and device for cache data, computer equipment and storage medium
CN113076187A (en) Distributed lock management method and device
CN109992447B (en) Data copying method, device and storage medium
CN114363350B (en) Service management system and method
CN112783982B (en) Data processing method, device, system, computer equipment and storage medium
CN107145302B (en) Method and equipment for executing file writing in distributed storage system
CN113518131B (en) Fault-tolerant processing method, device and system for transmission data of network abnormality
CN111338848B (en) Failure application copy processing method and device, computer equipment and storage medium
CN113391759B (en) Communication method and equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant