CN115981915A

CN115981915A - Node capacity expansion system and method

Info

Publication number: CN115981915A
Application number: CN202111202206.9A
Authority: CN
Inventors: 彭志成
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2021-10-15
Filing date: 2021-10-15
Publication date: 2023-04-18

Abstract

The application provides a node capacity expansion system and a method, and the system comprises: the management node is used for responding to the node capacity expansion instruction and switching the node state of the standby nodes with the target number into a preparation state; the standby node is used for acquiring a plurality of first messages written by the main node when in a standby state, and caching a second message corresponding to the message writing request when receiving the message writing request in the process of acquiring the plurality of first messages; when the first messages consistent with the second messages exist in the plurality of first messages, releasing the second messages, and when the number of the remaining messages cached by the standby node is lower than a first number threshold value, sending first notification information to the management node; the management node is also used for switching the standby node which sends the first notification information from a preparation state to an online state to be used as a main node for message writing; by the method and the device, the node capacity expansion process can be unaware, and the stability of the node capacity expansion is improved.

Description

Node capacity expansion system and method

Technical Field

The present application relates to the field of information processing technologies, and in particular, to a node capacity expansion system and method.

Background

In the related art, when capacity expansion is performed on a master node in a distributed system (for example, an original master node is N and needs to be expanded to 2N), new node devices need to be prepared first, then data backup is performed, and the data after the backup is completed are migrated to the prepared node devices one by one, so as to implement the capacity expansion.

However, in the main node capacity expansion mode, if network blocking is encountered, data backup and migration failure is likely to occur, which leads to capacity expansion failure, and in the capacity expansion process, since data backup and migration and other work are required between nodes (including the original main node and the newly prepared node), a message write request at a service side cannot be completed normally, which affects normal use.

Disclosure of Invention

Embodiments of the present application provide a node capacity expansion system, a node capacity expansion method, a node capacity expansion device, a node capacity expansion apparatus, a storage medium, and a computer program product, which can implement imperceptibility of a node capacity expansion flow and improve stability of node capacity expansion.

The technical scheme of the embodiment of the application is realized as follows:

an embodiment of the present application further provides a node capacity expansion system, including: a management node, a primary node, and at least one backup node, wherein,

the management node is used for responding to the node capacity expansion instruction and switching the node states of the standby nodes with the target quantity into a preparation state;

the master node is used for writing the message when receiving the message writing request;

the standby node is used for acquiring a plurality of first messages written by the main node when the standby node is in a standby state, and caching a second message corresponding to a message writing request when the message writing request is received in the process of acquiring the plurality of first messages;

when a plurality of first messages consistent with the second message exist in the first messages, releasing the second message, and when the number of the remaining messages cached by the standby node is lower than a first number threshold value, sending first notification information to the management node;

and the management node is further configured to switch the standby node that sends the first notification information from the standby state to an online state when receiving the first notification information, so as to serve as a master node to write in a message.

The embodiment of the application also provides a node capacity expansion method, which is applied to a node capacity expansion system comprising a management node, a main node and at least one standby node; the method comprises the following steps:

the standby node in the standby state acquires a plurality of first messages written in by the main node, and the standby node in the standby state responds to a node capacity expansion instruction for the management node and performs state switching on standby nodes in a target number to obtain the standby nodes in the standby state;

in the process of acquiring a plurality of first messages, when a message write request is received, caching a second message corresponding to the message write request;

when a plurality of first messages consistent with the second message exist in the first messages, the second message is released, and when the number of the remaining messages cached by the standby node is lower than a first number threshold value, first notification information is sent to the management node so that the management node can enable the first notification information to be used for notifying the standby node that the remaining messages are consistent with the second message

And the management node switches the standby node sending the first notification information from the preparation state to an online state based on the first notification information, and the standby node is used as a main node to write in messages.

The embodiment of the present application further provides a node capacity expansion device, which is applied to a node capacity expansion system including a management node, a master node, and at least one standby node; the method comprises the following steps:

an obtaining module, configured to obtain, by a standby node in a standby state, a plurality of first messages written by the host node, where the standby node in the standby state is obtained by performing state switching on a target number of standby nodes in response to a node capacity expansion instruction for the management node;

the cache module is used for caching a second message corresponding to a message write request when the message write request is received in the process of acquiring a plurality of first messages;

a sending module, configured to release the second message when a first message consistent with the second message exists in the plurality of first messages, and send first notification information to the management node when the number of remaining messages cached by the standby node is lower than a first number threshold, so that the management node switches, based on the first notification information, the standby node that sends the first notification information from the standby state to an online state, so as to perform message writing as a master node.

An embodiment of the present application further provides an electronic device, including:

a memory for storing executable instructions;

and the processor is used for implementing the node capacity expansion method provided by the embodiment of the application when the executable instructions stored in the memory are executed.

The embodiment of the present application further provides a computer-readable storage medium, where executable instructions are stored, and when the executable instructions are executed by a processor, the method for expanding the capacity of the node provided in the embodiment of the present application is implemented.

The embodiments of the present application further provide a computer program product, which includes a computer program or a computer readable medium, where the computer program or the computer readable medium is executed by a processor to implement the node capacity expansion method provided in the embodiments of the present application.

The embodiment of the application has the following beneficial effects:

by applying the embodiment of the application, when the management node receives a node capacity expansion instruction, the standby nodes of the control target number are in a standby state, the main node writes messages based on the message writing request, and the standby nodes in the standby state acquire a plurality of first messages written by the main node and cache second messages corresponding to the received message writing request in the process of acquiring the plurality of first messages; when the standby node determines that the first message consistent with the second message exists in the acquired first message, releasing the second message, and when the number of the remaining messages cached by the standby node is lower than a first number threshold, sending first notification information to the management node; and the management node switches the standby node sending the first notification information from a standby state to an online state based on the received first notification information, so that the standby node is used as a main node to write in the information, and the capacity expansion of the main node is realized.

Here, the message write-in request of the master node in the whole master node capacity expansion flow is uninterrupted and normal write-in is kept, and the node capacity expansion flow is not sensed by the service side; and the standby node acquires the first message after the first message corresponding to the message write request is written into the main node, so that the message synchronization of the standby node relative to the main node is realized, and when the acquired first message contains a message which is consistent with a second message cached aiming at the message write request, the second message is released, so that whether the standby node can be used as the main node or not is determined based on whether the remaining number of the second message is lower than a number threshold (if the remaining number of the second message is lower than the number threshold, the message synchronization of the standby node relative to the main node is completed, and the standby node can be used as the main node), so that the problem of capacity expansion failure caused by data backup and migration failure is solved, and the stability and the availability of node capacity expansion are improved.

Drawings

Fig. 1 is a schematic architecture diagram of a node capacity expansion system 100 according to an embodiment of the present application;

fig. 2 is a schematic structural diagram of an electronic device 500 for implementing a node capacity expansion method according to an embodiment of the present application;

fig. 3 is a schematic view illustrating an interaction flow between nodes in the node capacity expansion system according to an embodiment of the present application;

fig. 4 is a schematic flowchart of a node capacity expansion method according to an embodiment of the present application;

FIG. 5 is a schematic diagram of an architecture of a distributed computing storage system provided by an embodiment of the present application;

fig. 6 is a schematic diagram of state 1 in a host node capacity expansion process according to an embodiment of the present application;

fig. 7 is a schematic diagram of state 2 in a host node capacity expansion flow provided in an embodiment of the present application;

fig. 8 is a schematic diagram of state 3 in a host node capacity expansion process according to an embodiment of the present application;

fig. 9 is a schematic diagram of state 4 in a host node capacity expansion process according to an embodiment of the present application;

fig. 10 is a schematic diagram of state 5 in a host node capacity expansion process according to an embodiment of the present application;

fig. 11 is a schematic diagram of state 6 in a master node capacity expansion flow provided in the embodiment of the present application.

Detailed Description

In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be described in further detail with reference to the accompanying drawings, the described embodiments should not be considered as limiting the present application, and all other embodiments obtained by a person of ordinary skill in the art without making creative efforts fall within the protection scope of the present application.

In the following description, reference is made to "some embodiments" which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict.

In the following description, references to the terms "first \ second \ third" are only to distinguish similar objects and do not denote a particular order or importance, but rather "first \ second \ third" may, where permissible, be interchanged in a particular order or sequence so that embodiments of the present application described herein can be practiced in other than the order shown or described herein.

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of the present application only and is not intended to be limiting of the application.

Before further detailed description of the embodiments of the present application, terms and expressions referred to in the embodiments of the present application will be described, and the terms and expressions referred to in the embodiments of the present application will be used for the following explanation.

1) In response to the condition or state on which the performed operation depends, one or more of the performed operations may be in real-time or may have a set delay when the dependent condition or state is satisfied; there is no restriction on the order of execution of the operations performed unless otherwise specified.

Based on the above explanations of terms and terms involved in the embodiments of the present application, the node capacity expansion system provided by the embodiments of the present application is explained below. Referring to fig. 1, fig. 1 is a schematic architecture diagram of a node capacity expansion system 10 according to an embodiment of the present application, where the node capacity expansion system includes: a management node, a primary node, and at least one backup node. To support an exemplary application, the terminal 100 is connected to the management node 300 through the network 200, and the node devices are also connected through the network, for example, the management node 300, the main node 400, and the standby node (the standby node 500-1 and the standby node 500-2 are exemplarily shown) are also connected through the network, and the network 200 may be a wide area network or a local area network, or a combination thereof, and uses wireless or wired links to implement data transmission.

A terminal 100, configured to send a node expansion instruction to a management node 300 in response to a node expansion operation triggered by a user;

the management node 300 is configured to switch the node states of the standby nodes of the target number to the preparation state in response to the node capacity expansion instruction;

a master node 400 configured to perform message writing when receiving a message writing request;

the standby node (such as the standby node 500-1) is configured to, when in a standby state, acquire a plurality of first messages written by the master node, and when receiving a message write request in a process of acquiring the plurality of first messages, cache a second message corresponding to the message write request; when the first messages consistent with the second messages exist in the plurality of first messages, releasing the second messages, and when the number of the remaining messages cached by the standby node is lower than a first number threshold value, sending first notification information to the management node; the first notification information is used for indicating that the standby node has the capability of being switched to an online state;

the management node 303 is further configured to, when receiving the first notification information, switch the standby node that sends the first notification information from the standby state to an online state, so as to serve as a master node to write a message.

In practical applications, each node device (such as the management node 300, the standby node or the master node 400) may be an independent physical server, may also be a server cluster or a distributed system formed by a plurality of physical servers, and may also be a cloud server that provides basic cloud computing services such as a cloud service, a cloud database, cloud computing, a cloud function, cloud storage, a network service, cloud communication, a middleware service, a domain name service, a security service, a CDN, and a big data and artificial intelligence platform. The terminal 100 may be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart television, a smart watch, and the like, but is not limited thereto. The terminal and the node device may be directly or indirectly connected through wired or wireless communication, and the application is not limited herein.

Referring to fig. 2, fig. 2 is a schematic structural diagram of an electronic device 600 implementing a node capacity expansion method according to an embodiment of the present application. In practical applications, the electronic device 600 may be each node device shown in fig. 1, and the electronic device 600 is taken as a standby node shown in fig. 1 as an example to describe an electronic device implementing the node capacity expansion method according to the embodiment of the present application, where the electronic device 600 provided in the embodiment of the present application includes: at least one processor 610, memory 650, at least one network interface 620, and a user interface 630. The various components in electronic device 600 are coupled together by bus system 640. It is understood that bus system 640 is used to enable communications among these components. The bus system 640 includes a power bus, a control bus, and a status signal bus in addition to a data bus. For clarity of illustration, however, the various buses are labeled as bus system 640 in fig. 2.

The Processor 610 may be an integrated circuit chip having Signal processing capabilities, such as a general purpose Processor, a Digital Signal Processor (DSP), or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like, wherein the general purpose Processor may be a microprocessor or any conventional Processor, or the like.

The user interface 630 includes one or more output devices 631 that enable presentation of media content, including one or more speakers and/or one or more visual displays. The user interface 630 also includes one or more input devices 632, including user interface components to facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons, and controls.

The memory 650 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid state memory, hard disk drives, optical disk drives, and the like. Memory 650 optionally includes one or more storage devices physically located remote from processor 610.

The memory 650 includes volatile memory or nonvolatile memory, and can also include both volatile and nonvolatile memory. The nonvolatile Memory may be a Read Only Memory (ROM), and the volatile Memory may be a Random Access Memory (RAM). The memory 650 described in embodiments herein is intended to comprise any suitable type of memory.

In some embodiments, memory 650 can store data to support various operations, examples of which include programs, modules, and data structures, or subsets or supersets thereof, as exemplified below.

An operating system 651 including system programs for handling various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and for handling hardware-based tasks;

a network communication module 652 for reaching other computing devices via one or more (wired or wireless) network interfaces 620, exemplary network interfaces 620 including: bluetooth, wireless compatibility authentication (WiFi), and Universal Serial Bus (USB), etc.;

a presentation module 653 for enabling presentation of information (e.g., a user interface for operating peripherals and displaying content and information) via one or more output devices 631 (e.g., display screens, speakers, etc.) associated with the user interface 630;

an input processing module 654 for detecting one or more user inputs or interactions from one of the one or more input devices 632 and translating the detected inputs or interactions.

In some embodiments, the node capacity expansion apparatus provided in this embodiment may be implemented in a software manner, and fig. 2 illustrates the node capacity expansion apparatus 655 stored in the memory 650, which may be software in the form of programs and plug-ins, and includes the following software modules: an obtaining module 6551, a buffering module 6552 and a sending module 6553, which are logical modules and therefore may be arbitrarily combined or further split according to the implemented functions, and the functions of the respective modules will be described below.

In other embodiments, the node capacity expansion Device provided in this embodiment may be implemented by a combination of hardware and software, and for example, the node capacity expansion Device provided in this embodiment may be a processor in the form of a hardware decoding processor, which is programmed to execute the node capacity expansion method provided in this embodiment, for example, the processor in the form of the hardware decoding processor may employ one or more Application Specific Integrated Circuits (ASICs), DSPs, programmable Logic Devices (PLDs), complex Programmable Logic Devices (CPLDs), field Programmable Gate Arrays (FPGAs), or other electronic components.

In some embodiments, the standby node may implement the node capacity expansion method provided in the embodiments of the present application by running a computer program. For example, the computer program may be a native program or a software module in an operating system; may be a local (Native) Application (APP). In general, the computer programs described above may be any form of application, module, or plug-in.

Next, a node capacity expansion system provided in an embodiment of the present application is described in detail. The node capacity expansion system provided by the embodiment of the application comprises: a management node, a primary node, and at least one backup node. Referring to fig. 3, fig. 3 is a schematic view of an interaction flow between nodes in the node capacity expansion system provided in the embodiment of the present application, where the interaction flow between nodes in the node capacity expansion system provided in the embodiment of the present application includes:

step 101: and the management node responds to the node capacity expansion instruction and switches the node states of the standby nodes with the target number into the standby state.

Here, in practical application, the node capacity expansion system may be a storage computing type service system, and is configured to carry a read-write request of a service side (for example, a message request, a service for storing information, and the like), for example, perform message writing in response to the write request, return corresponding data information in response to the read request, and the like. In practical implementation, the load of the read-write request is mainly completed by node devices in the system, and the node devices are based on a main node + standby node architecture. In the embodiment of the present application, each primary node has a corresponding backup node, and the backup node is equivalent to a mirror node of the corresponding primary node, so as to implement data synchronization between the primary node and the backup node. Therefore, the read-write requests in the embodiment of the application are separated, that is, the main node writes the message to respond to the write request, and the standby node reads the message to respond to the read request, so that the main node does not need to bear the read request, when the main node has a capacity expansion requirement, the capacity expansion of the main node is realized, meanwhile, the processing of the read request is avoided being influenced, the read request is uninterrupted, and for a service side and a user, the effect that the read request service is not sensed in the capacity expansion process of the main node is realized.

It should be noted that the standby nodes in mirroring relationship with the master node are other standby nodes in the system except for the target number of standby nodes, where the target number of standby nodes refers to a specific standby node, for example, the target number of standby nodes A, B, C and D is 4, and then the subsequent target number of standby nodes refers specifically to the standby nodes A, B, C and D.

Here, the node capacity expansion system provided in the embodiment of the present application is applied to capacity expansion of a host node in a system, and then the node capacity expansion system provided in the embodiment of the present application is described in detail based on a capacity expansion flow of the host node.

In some embodiments, the node capacity expansion system further includes a management node, and the management node is mainly responsible for managing node states of the node devices, selection of the master node, heartbeat maintenance, and the like. In the embodiment of the present application, when a user has a capacity expansion demand for a host node, for example, when the host node needs to perform capacity expansion processing due to conditions such as service overload of the host node, the user may trigger a node capacity expansion instruction for the host node. The node capacity expansion instruction acts on the management node, and the capacity expansion of the main node is realized through the management node. After the management node receives the node capacity expansion instruction, a standby node capable of being used for main node capacity expansion is searched in the system, so that the capacity expansion of the main node is realized by converting the standby node into the main node. In practical application, the node capacity expansion instruction may further include the number of nodes of the standby node required by the capacity expansion main node, or the number of nodes to be reached by the main node after capacity expansion, so that the management node may control the standby nodes of the corresponding number to implement the main node capacity expansion task corresponding to the node capacity expansion instruction.

When the management node determines that the target number of standby nodes are needed to realize the capacity expansion of the main node, the management node switches the node states of the target number of standby nodes into a standby state, namely, the standby state is used as the standby state of the main node. In actual implementation, the management node may send a control instruction for instructing the corresponding standby node to switch the node state to the standby state, and when the standby node receives the control instruction, the standby node controls the node state of the management node to be switched from the current state (for example, an offline state) to the standby state, so as to wait for a subsequent operation to implement conversion from the standby node to the master node, and achieve the purpose of capacity expansion of the master node. For example, the node capacity expansion instruction indicates that the number of nodes after the capacity expansion of the master node is 2, the number of the original master nodes is 1, at this time, if the original master nodes need to be offline after the capacity expansion, the target number of the standby nodes required for the capacity expansion is 2, and if the original master nodes do not need to be offline after the capacity expansion, the target number of the standby nodes required for the capacity expansion is 1.

It should be noted that the target number of standby nodes refers to a specific standby node, such as the target number of 4 standby nodes A, B, C and D, and then the subsequent target number of standby nodes refers to the standby nodes A, B, C and D.

In some embodiments, in the process of capacity expansion of the standby node as the master node, the standby node may also be in a trial state before being in the preparation state, so as to switch to the preparation state after completing the target operation in the trial state. Based on the above, the management node is further configured to switch the node states of the standby nodes of the target number into trial states in response to the node capacity expansion instruction; the standby node is further used for acquiring a plurality of first messages written by the main node when the standby node is in a trial state, and sending second notification information to the management node when the number of the first messages which are not acquired is lower than a second number threshold value in the process of acquiring the plurality of first messages; and the management node is also used for receiving the second notification information and switching the standby node which sends the second notification information from the trial state to the standby state. The second notification information is used for indicating that the standby node in the trial state has the capability of being switched to the standby state;

here, before the standby nodes controlling the target number are in the standby state, the management node may further switch the node states of the standby nodes of the target number to the trial state in response to the node capacity expansion instruction. Specifically, the management node may send a control instruction for instructing the corresponding standby node to switch the node state to the trial state, and when the standby node receives the control instruction, the management node controls the node state of the management node to be switched from the current state (for example, the offline state) to the trial state. It should be noted that the target number of standby nodes refers to a specific standby node, such as the target number of 4 standby nodes A, B, C and D, and then the subsequent target number of standby nodes refers to the standby nodes A, B, C and D.

Correspondingly, when the standby nodes in the target number are in the trial state, a plurality of first messages written by the main node (namely, the original main node) need to be acquired, the plurality of first messages include historical messages written before the node capacity expansion instruction is triggered and messages written in real time after the node capacity expansion instruction is triggered (namely, during the node capacity expansion), and in the acquisition process, the messages are specifically acquired according to the writing sequence of the messages, namely, according to the writing time points of the messages, the messages are sequentially acquired from the first to the last. Therefore, the write request can be uninterrupted in the host node capacity expansion process, and the effect that the write request service is not perceived in the host node capacity expansion process is achieved for the service side and the user. Here, the reason why the standby node in the trial state acquires the plurality of first messages written by the master node (i.e., the original master node) is to chase the messages, and the standby node can synchronize the messages written by the original master node to itself, so that the loss of information during capacity expansion of the master node is avoided. In actual implementation, the plurality of first messages written by the master node (i.e., the original master node) may be stored in the master node itself, or may be stored in other node devices dedicated to data storage.

The standby node determines the number of the first messages which remain unacquired in real time or periodically in the process of acquiring a plurality of first messages. When the number of the first messages which are not acquired is determined to be lower than the second number threshold, the standby node is characterized to be qualified for switching from the current trial state to the standby state, and at this time, the standby node sends second notification information to the management node, wherein the second notification information is used for indicating that the standby node in the trial state has the capability of being switched to the standby state.

And after receiving the second notification information sent by the standby node, the management node switches the standby node sending the second notification information from the trial state to the standby state based on the second notification information. Here, the standby node can acquire the first message written by the main node and write the first message into the standby node in a trial state, so that the synchronization of the message written in the original main node is realized; meanwhile, the standby node in the trial state does not need to receive the message writing request and execute corresponding message caching operation, so that data synchronization of the main node can be performed quickly and efficiently, and the capacity expansion of the standby node is ensured to be performed stably for the main node.

In some embodiments, the node capacity expansion system further comprises a proxy node. Correspondingly, the management node is also used for sending state synchronization information to the agent node when the standby node is switched over; and the proxy node is used for receiving the state synchronization information and updating the node state of the corresponding standby node based on the state synchronization information.

Here, a proxy node is also included in the system. The proxy node is mainly used for sending read-write services (such as message write requests and message read requests) on the service side to a master node or a standby node carrying the read services on the service side. In practical application, the proxy node may be a node device, or a node cluster composed of a plurality of node devices, where each node bears a request for a service-side read-write service through a consistent hash algorithm, and transmits the read-write request of the service side to a master node or a standby node responsible for completing read-write through the consistent hash algorithm.

Correspondingly, the management node is further configured to synchronize node states of the nodes (including the master node and the standby node) to the proxy node, so that the proxy node is also convenient to transmit a read-write request of a service side according to the node states of the nodes (including the master node and the standby node), and the specific management node may perform state synchronization in a heartbeat manner. Here, when the standby node is switched, the management node sends state synchronization information to the proxy node, where the state synchronization information is used to indicate the node state after the state switching of the corresponding standby node, and the node state may include an offline state, a trial state, a ready state, an online state, and the like.

And after the agent node receives the state synchronization information sent by the management node, updating the node state of the corresponding standby node according to the state synchronization information. For example, if the node status of the standby node a stored in the agent node is down, and the status synchronization information sent by the management node is received and indicates that the standby node a is currently in the standby status, the stored node status of the standby node a is updated from the down status to the standby status.

In some embodiments, the proxy node is further configured to reduce a sending rate of the message write request when the state synchronization information indicates that the standby node is in the standby state.

Here, in order to better and faster implement the switching of the node state of the standby node (for example, switching from the trial state to the standby state and then switching to the online state) so as to shorten the distribution flow of the entire master node expansion, a slow write strategy may be added for the proxy node. Specifically, when the state synchronization information received by the proxy node indicates that the standby node is in a prepared state, that is, in the host node capacity expansion flow, the proxy node reduces the sending rate of the message write request sent to the host node or the standby node. In practical implementation, when the node is in a non-capacity expansion flow, the sending mode of the message write request of the proxy node is in a normal sending mode, and when the state synchronization information indicates that the standby node is in a prepared state, the sending mode of the message write request is controlled to be switched from the normal sending mode to a target sending mode, and the sending rate of the message write request in the target sending mode is lower than that of the message write request in the normal sending mode.

Therefore, the write-in request of the external service side can be still served in the main node capacity expansion process, and by adding the slow write-in strategy, the switching of the node state of the standby node (for example, switching from the trial state to the preparation state and then switching to the online state) is better and faster realized, and the whole main node capacity expansion issuing process is shortened.

In practical application, after the host node completes capacity expansion, the sending rate of the message write request of the proxy node can still be controlled to be restored to the original sending rate, that is, the sending mode of the message write request is controlled to be switched from the target sending mode to the conventional sending mode.

Step 102: and when receiving the message writing request, the master node writes the message.

Here, in the process of host node capacity expansion, in the embodiment of the present application, a message write request from an external service side may still be received to continue to carry write service from the service side, so as to implement uninterrupted write request.

At this time, if receiving a message write request from the service side, the master node performs a message write operation in response to the message write request, and performs message write corresponding to the message write request. In practical application, when the master node writes a message, the master node firstly writes the message corresponding to the message write request into the kafka message sequence, and after the completion of the writing of the kafka message sequence, writes the message corresponding to the message write request into the local storage.

In the embodiment of the application, because the read-write requests are separated, that is, the master nodes perform message writing to respond to the write requests, and the standby nodes perform message reading to respond to the read requests, each master node has a corresponding standby node correspondingly, and the standby node is equivalent to a mirror node of the corresponding master node, so that data synchronization between the master node and the standby node is realized. Specifically, after the master node writes the message corresponding to the message write request into the kafka message sequence, the backup node corresponding to the master node (i.e., the mirror node of the master node) reads the message corresponding to the message write request from the kafka message sequence, and writes the read message corresponding to the message write request into the local storage, so as to implement data synchronization of the master node and its corresponding backup node. Therefore, when receiving a message reading request of a service side, the standby node (namely the mirror image node of the main node) can realize the service of the reading request, and return the corresponding message to be read, thereby realizing the separation of the reading and writing requests, avoiding the influence on the reading request in the main node capacity expansion process to the greatest extent, ensuring that the reading request is uninterrupted, and realizing the effect that the reading request service is not sensed in the main node capacity expansion process for the service side and users. It should be noted that the standby nodes in mirroring relationship with the primary node are other standby nodes in the system besides the target number of standby nodes.

Step 103: the standby node acquires a plurality of first messages written by the main node when in a standby state, and caches a second message corresponding to a message write request when receiving the message write request in the process of acquiring the plurality of first messages.

Here, when the standby node is in the standby state, it needs to acquire a plurality of first messages written by the host node (i.e., the original host node), where the plurality of first messages include a history message written before a node expansion instruction is triggered and a message written in real time after the node expansion instruction is triggered (i.e., during the node expansion), and in the acquisition process, the plurality of first messages are specifically acquired according to the writing sequence of the messages, that is, according to the writing time point of the messages, and are sequentially acquired from first to last. Therefore, the write request can be ensured to be uninterrupted in the host node capacity expansion process, and the effect that the write request service is not perceived in the host node capacity expansion process is realized for the service side and the user. Here, the reason that the standby node in the standby state acquires the plurality of first messages written by the main node (i.e., the original main node) is to catch up the messages, and the standby node can synchronize the messages written by the original main node to itself, so that the loss of information during the capacity expansion of the main node is avoided. In actual implementation, the plurality of first messages written by the master node (i.e., the original master node) may be stored in the master node itself, or may be stored in other node devices dedicated to data storage.

The standby node in the standby state also receives the message write-in request in the process of acquiring a plurality of first messages, and at the moment, the system is in a double write mode, namely, the original main node and the standby node in the standby state both receive the message write-in request. Here, the original master node needs to perform message writing (i.e. execute the procedure of writing the kafka message sequence first and then writing the local storage) in response to the message writing request, that is, step 102 above; the standby node in the standby state needs to respond to the message write request, and does not perform the message write (i.e., does not need to perform the procedure of writing the kafka message sequence first and then writing the message sequence to the local storage), but caches the second message corresponding to the message write request in the local memory (e.g., a set container in the memory).

In some embodiments, the proxy node is further configured to send a message write request carrying the target identifier to the standby node when the message write instruction is received and the state synchronization information indicates that the standby node is in the standby state; and the standby node is further used for caching a second message corresponding to the message write request when the message write request carrying the target identifier is received. The target identifier is used for indicating the corresponding standby node to cache the message corresponding to the message write request carrying the target identifier.

Here, the corresponding standby node is instructed by the target identifier to cache the message corresponding to the first message write request. And the proxy node is further used for sending a message write request carrying the target identifier to the standby node when the message write instruction is received and the state synchronization information indicates that the standby node is in the standby state. When the standby node receives the message write request carrying the target identifier, the message write request is responded, and the message write is not executed (that is, the procedure of writing the kafka message sequence first and then writing the message sequence to the local storage does not need to be executed), but the second message corresponding to the message write request is cached to the local memory (such as a set container in the memory).

Step 104: and when the first messages consistent with the second messages exist in the plurality of first messages, the standby node releases the second messages, and when the number of the remaining messages cached by the standby node is lower than a first number threshold value, first notification information is sent to the management node.

The first notification information is used for indicating that the standby node in the standby state has the capability of switching to the on-line state.

Here, the standby node further determines whether there is a first message identical to the second message among the acquired plurality of first messages in the process of acquiring the plurality of first messages. The step is performed in order to determine the progress of the backup node in the ready state relative to the additional message of the original master node by judging whether the acquired first message is consistent with the second message or not, because the first message is acquired in sequence according to the writing sequence of the messages when the first message is acquired, and the master node (namely the original master node) and the standby node in the ready state both receive the message writing request and perform corresponding operations in the capacity expansion process of the master node, and the messages written in the same message writing request are consistent.

If the first message consistent with the second message exists in the plurality of first messages, the standby node in the preparation state is indicated to read the history message written by the main node before the node capacity expansion instruction is triggered, and the message written after the node capacity expansion instruction is triggered starts to be read at present. And based thereon it can be determined which second message the read first message corresponds to. The standby node in the standby state already synchronously acquires the message corresponding to the current message writing request by acquiring the first message written by the main node, namely, the message synchronization of the standby node corresponding to the original main node is completed, at this time, the second message cached by the standby node is released, and specifically, the message sequence number item stored in the set container by the second message is removed.

Meanwhile, in the process of releasing the cached second message, the standby node in the preparation state needs to determine the remaining message quantity of the cached second message after the release. When the number of the remaining messages cached by the standby node is lower than a first number threshold, the standby node is characterized to have the qualification of being switched from the standby state to the online state, and at this time, the standby node sends first notification information to the management node, wherein the first notification information is used for indicating that the standby node in the standby state can be switched to the online state.

Step 105: and the management node receives the first notification information and controls the corresponding standby node to be switched from the standby state to the online state so as to be used as the main node for writing the message.

Here, after receiving the first notification information sent by the standby node in the standby state, the management node controls the corresponding standby node to switch from the standby state to the online state based on the first notification information. The standby node in the online state can be used as a main node to write messages, the messages corresponding to the message write requests are written into the kafka message sequence, and after the writing of the kafka message sequence is completed, the messages corresponding to the message write requests are written into a local storage.

In some embodiments, the proxy node is further configured to send a first message write request that does not carry the target identifier to the standby node when the message write instruction is received and the number of the standby nodes in the online state reaches a third number threshold; the standby node is further used for determining whether a target message corresponding to the first message write request exists in the standby node or not when the first message write request is received; when determining that the target message corresponding to the first message writing request exists, not writing the message corresponding to the first message writing request; and when determining that the target message corresponding to the first message writing request does not exist, performing message writing corresponding to the first message writing request.

Here, when the number of the standby nodes in the online state reaches the third number threshold, if a message write instruction is received, the proxy node sends a first message write request not carrying the target identifier, so that the standby node performs a normal message write operation, that is, writes the kafka message queue first, and then writes the message queue to the local storage.

Because the standby node also acquires a first message from the original main node, the first message writing request is simultaneously sent to the original main node and the standby node and the message writing is carried out according to the sequence of writing the original main node first and then writing the standby node, the standby node may acquire a message corresponding to the first message writing request from the original main node, at the moment, when the standby node writes the message, whether the message corresponding to the first message writing request (such as the message mail key) exists locally is judged first, if so, the message corresponding to the first message writing request is not written, and a failure notice is returned to the proxy node; and if the message does not exist, performing message writing corresponding to the first message writing request.

In some embodiments, the standby node is further configured to stop acquiring the first plurality of messages written by the primary node when the number of remaining messages cached is zero.

Here, the standby node in the online state still needs to acquire the first message written by the master node, and meanwhile, in the process of acquiring the plurality of first messages, it is further determined whether there is a first message in the plurality of acquired first messages, which is consistent with the cached second message. If the first message consistent with the second message exists in the plurality of first messages, the second message cached by the standby node is released, and meanwhile, in the process of releasing the cached second message, the standby node in the preparation state needs to determine the residual message quantity of the second message after the cached second message is released. When the number of the cached remaining messages is zero, the messages written in the standby node are considered to have traced the messages written in the original main node, namely the data synchronization is completed, and the standby node stops acquiring the plurality of first messages written in by the main node.

At this time, the token indicates that the capacity expansion of the master node is completed, and when the target number of the standby nodes is greater than the original number of the master nodes, since the data and the traffic of the original master nodes are synchronized to the standby nodes in an online state, the master nodes can be offline in order to reduce the waste of node resources, and the master nodes can be used as the standby nodes to be online when receiving the capacity expansion instruction of the standby nodes. In practical application, after the main node is offline, the main node is in an offline state; at this time, when the machine needs to be expanded, that is, when the capacity of the standby node in the system needs to be expanded, the node state of the master node in the offline state may be switched to the online state, so that the master node in the online state again serves as the standby node.

In some embodiments, the target number is greater than the original number of master nodes; correspondingly, the management node is also used for the offline main node when the standby nodes with the target number are in the online state, so that the offline main node can be used as the standby node to be online when receiving the standby node capacity expansion instruction.

Here, when the target number of the standby nodes is greater than the original number of the master nodes, and if the management node determines that the standby nodes of the target number are all in the on-line state, since the data and the traffic of the original master node are synchronized to the standby nodes in the on-line state, in order to reduce the waste of node resources, the master node may be taken off-line, so as to be used as the standby nodes on-line when receiving the capacity expansion instruction of the standby nodes.

In some embodiments, the number of the standby nodes is greater than the target number, and when the standby nodes with the target number are used as the primary node to perform message writing, the other standby nodes except the standby nodes with the target number are mirror nodes of the primary node;

correspondingly, the standby node is used for responding to the message writing request when serving as the main node, writing a third message corresponding to the message writing request into the message queue, and writing the third message into the local storage; the other standby nodes are used for acquiring the third message from the message queue and writing the third message into the local storage; and when receiving the message reading request, returning a third message corresponding to the message reading request.

Here, the number of the standby nodes in the system is larger than the target number, and when the target number of the standby nodes perform message writing as the master node, the other standby nodes except the target number of the standby nodes are mirror nodes of the master node. For example, the number of the standby nodes is 4, and includes the standby nodes A, B, C and D, where the target number (2) of the standby nodes refers to the standby nodes a and B specifically, and the standby nodes C and D are other standby nodes except the target number of the standby nodes, and the other standby nodes may be mirror nodes of the standby nodes a and B that have become the master node.

And after the expansion of the main node is finished, the standby nodes with the target number are used as the main node to write the message. When the standby node is used as the main node, the third message corresponding to the message write request is firstly written into the kafka message sequence, and after the writing of the kafka message sequence is completed, the third message corresponding to the message write request is written into the local storage. At this time, a mirror node (i.e., other standby nodes) corresponding to the target number of standby nodes also exists in the system, and the mirror node can be used for providing a message reading service. Specifically, after the primary node (i.e., the target number of standby nodes) writes the third message corresponding to the message write request into the kafka message sequence, the other standby nodes corresponding to the primary node read the written third message corresponding to the message write request from the kafka message sequence, and write the read third message corresponding to the message write request into the local storage, so as to implement data synchronization between the primary node and the corresponding other standby nodes. Therefore, when receiving a message reading request of a service side, the standby node (namely the mirror image node of the main node) can realize the service of the reading request, and returns the corresponding message to be read, thereby realizing the separation of the reading and writing requests, avoiding the influence on the reading request in the expansion process of the main node to the greatest extent, ensuring that the reading request is uninterrupted, and realizing the effect that the reading request service is not perceived in the expansion process of the main node for the service side and a user.

The embodiment of the present application can be applied to various scenes such as cloud technology, artificial intelligence, smart traffic, and driving assistance.

By applying the embodiment of the application, when the management node receives a node capacity expansion instruction, the standby nodes of the control target number are in the standby state, the main node writes messages based on the message writing request, the standby nodes in the standby state acquire a plurality of first messages written by the main node, and cache second messages corresponding to the received message writing request in the process of acquiring the plurality of first messages; when the standby node determines that the first message consistent with the second message exists in the acquired first message, releasing the second message, and when the number of the remaining messages cached by the standby node is lower than a first number threshold, sending first notification information to the management node; the management node switches the standby node sending the first notification information from a standby state to an online state based on the received first notification information, so that the standby node is used as a main node to write information, and the capacity expansion of the main node is realized.

Next, a node capacity expansion method provided in the embodiment of the present application is described in detail. The node capacity expansion method provided by the embodiment of the application is applied to a node capacity expansion system comprising a management node, a main node and at least one standby node. Referring to fig. 4, fig. 4 is a schematic flowchart of a node capacity expansion method provided in an embodiment of the present application, where the node capacity expansion method provided in the embodiment of the present application includes:

step 201: the standby node in the standby state acquires a plurality of first messages written by the main node.

The standby nodes in the standby state are obtained by performing state switching on the standby nodes in the target number for the management node to respond to the node capacity expansion instruction.

Here, in practical application, the node capacity expansion system may be a storage computing type service system, and is configured to carry a read-write request of a service side (for example, a message request, a service for storing information, and the like), for example, perform message writing in response to the write request, return corresponding data information in response to the read request, and the like. In practical implementation, the load bearing of the read-write request is mainly completed by node equipment in the system, and the node equipment exists in an architecture mode that the node equipment is mainly based on a main node and a standby node. In the embodiment of the application, each main node correspondingly has a corresponding standby node, and the standby node is equivalent to a mirror node of the corresponding main node, so that data synchronization between the main node and the standby node is realized. Therefore, the read-write requests in the embodiment of the application are separated, that is, the main node writes the messages to respond to the write requests, and the standby node reads the messages to respond to the read requests, so that the main node does not need to bear the read requests, when the main node has the capacity expansion requirement, the capacity expansion of the main node is realized, meanwhile, the processing of the read requests is avoided being influenced, the read requests are uninterrupted, and the effect that the read request service is not perceived in the capacity expansion process of the main node is realized for a service side and a user.

Here, the node capacity expansion method provided in this embodiment of the present application is applied to capacity expansion of a host node in a system, and then a detailed description is given to the node capacity expansion system provided in this embodiment of the present application based on a capacity expansion flow of the host node.

In some embodiments, the node capacity expansion system further includes a management node, and the management node is mainly responsible for managing node states of the node devices, selection of the master node, heartbeat maintenance, and the like. In the embodiment of the present application, when a user has a capacity expansion demand for a host node, for example, when the host node needs to perform capacity expansion processing due to conditions such as service overload of the host node, the user may trigger a node capacity expansion instruction for the host node. The node capacity expansion instruction acts on the management node, and the capacity expansion of the main node is achieved through the management node. After the management node receives the node capacity expansion instruction, a standby node capable of being used for main node capacity expansion is searched in the system, so that the capacity expansion of the main node is realized by converting the standby node into the main node. In practical application, the node capacity expansion instruction may further include the number of nodes of the standby node required by the capacity expansion main node, or the number of nodes to be reached by the main node after capacity expansion, so that the management node may control the standby nodes of the corresponding number to implement the main node capacity expansion task corresponding to the node capacity expansion instruction.

When the management node determines that the standby nodes with the target quantity are needed to realize the capacity expansion of the main node, the management node controls the standby nodes with the target quantity to be in a standby state, namely, the standby nodes are used as the standby state of the main node. In actual implementation, the management node may send a control instruction for instructing the corresponding standby node to switch the node state to the standby state, and when the standby node receives the control instruction, the standby node controls the node state to be switched from the current state (for example, the offline state) to the standby state, so as to wait for a subsequent operation to implement conversion from the standby node to the main node, and achieve the purpose of capacity expansion of the main node. For example, the node capacity expansion instruction indicates that the number of nodes after capacity expansion of the master node is 2, the number of the original master nodes is 1, at this time, if the original master nodes need to be offline after capacity expansion, the target number of the standby nodes required for capacity expansion is 2, and if the original master nodes do not need to be offline after capacity expansion, the target number of the standby nodes required for capacity expansion is 1.

Step 202: in the process of acquiring a plurality of first messages, when a message write request is received, caching a second message corresponding to the message write request.

Here, in the process of host node capacity expansion, in the embodiment of the present application, a message write request from an external service side may still be received to continue to carry the write service from the service side, so as to implement that the write request is uninterrupted, and for the service side and the user, the effect that the write request service is not perceived in the process of host node capacity expansion is implemented.

At this time, if receiving a message write request from the service side, the master node performs a message write operation in response to the message write request, and performs message write corresponding to the message write request. In practical application, when the master node writes a message, firstly, the message corresponding to the message write request is written into the kafka message sequence, and after the writing of the kafka message sequence is completed, the message corresponding to the message write request is written into the local storage.

In the embodiment of the application, due to the separation of the read-write requests, that is, the master nodes perform message writing to respond to the write requests, and the standby nodes perform message reading to respond to the read requests, each master node corresponds to a corresponding standby node, and the standby nodes are equivalent to mirror nodes of the corresponding master nodes, so that data synchronization between the master nodes and the standby nodes is realized. Specifically, after the master node writes the message corresponding to the message write request into the kafka message sequence, the backup node corresponding to the master node (i.e., the mirror node of the master node) reads the message corresponding to the message write request from the kafka message sequence, and writes the read message corresponding to the message write request into the local storage, so as to implement data synchronization of the master node and its corresponding backup node. Therefore, when receiving a message reading request of a service side, the standby node (namely the mirror image node of the main node) can realize the service of the reading request, and return the corresponding message to be read, thereby realizing the separation of the reading and writing requests, avoiding the influence on the reading request in the main node capacity expansion process to the greatest extent, ensuring that the reading request is uninterrupted, and realizing the effect that the reading request service is not sensed in the main node capacity expansion process for the service side and users. It should be noted that the standby nodes in mirroring relationship with the primary node are other standby nodes in the system except for the target number of standby nodes.

Step 203: when first messages consistent with second messages exist in the plurality of first messages, the second messages are released, and when the number of the remaining messages cached by the standby node is lower than a first number threshold value, first notification information is sent to the management node, so that the management node switches the standby node sending the first notification information from a standby state to an online state based on the first notification information, and the standby node is used as a main node for writing messages.

The first notification information is used for indicating that the standby node in the standby state has the capability of being switched to the on-line state.

Here, when the standby node is in the standby state, it needs to acquire a plurality of first messages written by the master node (i.e., the original master node), where the plurality of first messages include a history message written before a node capacity expansion instruction is triggered and a message written in real time after the node capacity expansion instruction is triggered (i.e., during the node capacity expansion), and in the acquisition process, the plurality of first messages are specifically acquired according to the writing sequence of the messages, that is, according to the writing time point of the messages, the plurality of first messages are sequentially acquired from the first to the last. Therefore, the write request can be ensured to be uninterrupted in the host node capacity expansion process, and the effect that the write request service is not perceived in the host node capacity expansion process is realized for the service side and the user. Here, the reason that the standby node in the standby state acquires the plurality of first messages written by the main node (i.e., the original main node) is to catch up the messages, and the standby node can synchronize the messages written by the original main node to itself, so that the loss of information during the capacity expansion of the main node is avoided. In actual implementation, the plurality of first messages written by the master node (i.e., the original master node) may be stored in the master node itself, or may be stored in other node devices dedicated to data storage.

The standby node in the standby state also receives the message writing request in the process of acquiring a plurality of first messages, and at the moment, the system is in a double writing mode, namely, the original main node and the standby node in the standby state both receive the message writing request. Here, the original master node needs to perform message writing in response to the message writing request (i.e. execute the flow of writing the kafka message sequence first and then writing the message sequence to the local storage), that is, step 102; the standby node in the standby state needs to respond to the message write request, and does not perform the message write (i.e., does not need to perform the procedure of writing the kafka message sequence first and then writing the message sequence to the local storage), but caches the second message corresponding to the message write request in the local memory (e.g., a set container in the memory).

The standby node also determines whether a first message consistent with the second message exists in the acquired plurality of first messages in the process of acquiring the plurality of first messages. The step is performed in order to determine the progress of the backup node in the ready state relative to the additional message of the original master node by judging whether the acquired first message is consistent with the second message or not, because the first message is acquired in sequence according to the writing sequence of the messages when the first message is acquired, and the master node (namely the original master node) and the standby node in the ready state both receive the message writing request and perform corresponding operations in the capacity expansion process of the master node, and the messages written in the same message writing request are consistent.

If the first message consistent with the second message exists in the plurality of first messages, the standby node in the preparation state reads the history message written by the main node before the triggering of the node capacity expansion instruction, and the standby node starts to read the message written after the triggering of the node capacity expansion instruction. And based thereon it can be determined which second message the read first message corresponds to. Because the standby node in the ready state has already synchronously acquired the message corresponding to the current message write request by acquiring the first message written by the master node, at this time, the second message cached by the standby node is released, which may be specifically implemented by removing the message sequence number item stored in the set container by the second message.

Meanwhile, in the process of releasing the cached second message, the standby node in the preparation state needs to determine the remaining message quantity of the cached second message after the release. When the number of the remaining messages cached by the standby node is lower than a first number threshold, the standby node is characterized to be qualified for being switched from the standby state to the online state, and at this time, the standby node sends first notification information to the management node, where the first notification information is used for indicating that the standby node in the standby state has the capability of being switched to the online state.

And after receiving the first notification information sent by the standby node in the standby state, the management node controls the corresponding standby node to be switched from the standby state to the online state based on the first notification information. The standby node in the online state can be used as a main node to write messages, including writing the messages corresponding to the message write requests into the kafka message sequence, and after the writing of the kafka message sequence is completed, writing the messages corresponding to the message write requests into a local storage.

In practical application, when the target number of the standby nodes is greater than the original number of the main nodes, and if the management node determines that the standby nodes with the target number are all in an on-line state, because the data and the traffic of the original main node are synchronized to the standby nodes in the on-line state, in order to reduce the waste of node resources, the main node can be taken off line so as to be taken as the standby nodes to be used on line when a standby node capacity expansion instruction is received.

By applying the embodiment of the application, when the management node receives a node capacity expansion instruction, the standby nodes of the control target number are in the standby state, the main node writes messages based on the message writing request, the standby nodes in the standby state acquire a plurality of first messages written by the main node, and cache second messages corresponding to the received message writing request in the process of acquiring the plurality of first messages; when the standby node determines that the first message consistent with the second message exists in the acquired first message, releasing the second message, and when the number of the remaining messages cached by the standby node is lower than a first number threshold, sending first notification information to the management node; and the management node switches the standby node sending the first notification information from a standby state to an online state based on the received first notification information, so that the standby node is used as a main node to write in the information, and the capacity expansion of the main node is realized.

Here, in the whole host node capacity expansion flow, the message write request of the host node is uninterrupted and normal write is kept, and the node capacity expansion flow is not sensible to the service side; and the standby node acquires the first message after the first message corresponding to the message write request is written by the main node, so that the message synchronization of the standby node relative to the main node is realized, and when the acquired first message contains a message which is consistent with a second message for the message write request cache, the second message is released, so that whether the standby node can be used as the main node or not is determined based on whether the residual quantity of the second message is lower than a quantity threshold (if the residual quantity of the second message is lower than the quantity threshold, the message synchronization of the standby node relative to the main node is completed, and the standby node can be used as the main node), so that the problem of capacity expansion failure caused by data backup and migration failure is avoided, and the stability and the availability of node capacity expansion are improved.

An exemplary application of the embodiments of the present application in a practical application scenario will be described below.

Distributed storage computing services (such as redis) are typically deployed in a master + slave (i.e., a master node + a standby node, where the slave node is a mirror image of the master node and data remains consistent). When the master node is expanded (for example, the original master is N nodes, and is now expanded to 2N nodes), a machine needs to be prepared first, then data backup is performed, the data are migrated to the prepared machine one by keys, and the original master node is retained. However, in this master node capacity expansion mode, because the read-write requests are all completed at the master node, the read requests may not be requested at the original master node, and need to be routed to a new master node. Meanwhile, in a large value scene (when a single value is set or map), the requirements on the network (such as network bandwidth, network stability and the like) are very high, the capacity expansion time is relatively long, the overall system jitter is relatively obvious, once the migration of data fails, the final capacity expansion fails, and the operation and maintenance complexity is greatly increased.

Based on this, the embodiment of the present application provides a master node capacity expansion method, which enables external read-write requests to be serviced without interruption while capacity expansion of the master node is performed, and zero perception is provided for a service and a user, thereby ensuring high availability of the service and the user.

Before describing the master node capacity expansion method provided in the embodiment of the present application, first, the overall architecture of the distributed computing storage system provided in the embodiment of the present application is described below. Referring to fig. 5, fig. 5 is a schematic structural diagram of a distributed computing storage system according to an embodiment of the present application, where functions of respective modules included in the schematic structural diagram are described as follows:

1) Service side read-write service: the calling module of the service side has the main function of interacting the read-write request of the service side with the storage computing system.

2) proxyserver: the access layer service (i.e. the proxy node) is composed of a plurality of nodes, and the read-write request of the service side read-write service is borne among the nodes in a consistent hash manner so as to transmit the read-write request of the service side to the leaf server.

3) configserver: the cluster management service (i.e., the management node) is responsible for managing the state, heartbeat maintenance, master node selection, and the like of each node of the leaf server, and meanwhile, synchronizing cluster information (e.g., state transformation of the slave node) to the proxyserver in a heartbeat manner.

4) leaf server: the system comprises a master (namely a main node) and a slave (namely a standby node), is a real calculation and storage core module and is responsible for completing the processing of read-write requests of proxyserver. In practical application, when the master node processes a write request, a message is written into the kafka message queue, and then the message is written into the local storage of the master node, and if the messages are all successful, the messages are returned to the proxyserver. Here, the Slave nodes 1-N may have multiple rows, and are set according to system requirements.

5) kafka: the message queue exists as a relay of a message, and is used for storing a write-in message of a master node of a leaf server, and then the write-in message is consumed by a slave node of the leaf server, so that data stored by the slave node of the leaf server is consistent with data stored by the master node, specifically, after the master node writes the message into kafka, the slave node stores the message in the kafka to the local of the slave node by consuming the message in the kafka, and the data synchronization of the master and the slave is realized. Thus, the slave node can carry the processing of the read request.

6) Incremental index build service: is a processing jobof flink, which is mainly used for depositing the messages in the kafka message queue into hbase persistent storage.

7) hbase: based on the existence of the hbase, the system can process the increase and deletion censorship messages of the service side and persistently store the result in a table of the hbase.

8) And (3) constructing a full index: the method is a spark batch processing task, can regularly (generally every day) trigger index construction on the total data in hbase, and then push the index construction to an Object Storage (COS) Storage for loading and switching of an online leaf server, index switching is generally needed when a leaf server node is down, and frequent switching can be avoided under normal conditions.

9) Object Store (COS): the index is a distributed storage service which has no directory hierarchy structure and no data format limitation, can contain mass data and supports HTTP/HTTPS protocol access, and can be used for storing indexes constructed by a full-scale index construction task.

Based on the above description of the overall architecture of the distributed computing storage system provided in the embodiment of the present application, a description is next given of a master node capacity expansion method provided in the embodiment of the present application, that is, how to implement high-availability capacity expansion of a master node in a scenario where a read-write service is not stopped, so that a service and a user do not have perception. The capacity expansion process related to the master node capacity expansion method provided by the embodiment of the application is divided into 6 stages, and there are 6 corresponding states corresponding to the state transition of the system, which are respectively described below. Wherein, the o-master is the original master node, and the n-master is the new master node after capacity expansion.

Referring to fig. 6, fig. 6 is a schematic diagram of state 1 in a master node capacity expansion flow provided in the embodiment of the present application. Here, the state 1 is a preparation state before the master node is expanded, and it is necessary to complete the expansion of all slave nodes first. Specifically, in order to ensure that system data is clean, it is usually necessary to downline a corresponding slave node to clear previous data in a node, so when the slave node is expanded, it is necessary to downline one row of slave nodes first, and then expand one time of slave nodes, for example, the original master has only 2 nodes, and prepare to expand the slave node into 4 nodes, as shown in fig. 6, the slave node has expanded into 4 nodes.

Thus, in state 1 of the master node capacity expansion flow, wherein,

(1) There is only one row of master (i.e. 2 nodes) in the configserver cluster management information;

(2) The proxyserver cluster routes the write request to the corresponding master, and each proxyserver sends the write request to any node of the master because the proxyserver and the leafserver do not force 1 to 1, and specifically, the write request is sent to the corresponding master through a consistency hash algorithm. At the moment, the write request is not limited, the transmitted msg does not carry any flag, and the flag refers to the mark/identification of the msg and can be represented by one field;

(3) According to the data volume of the service, the kafka message queue needs to allocate enough partition partitions in advance to ensure that the subsequent leaf servers cannot cause discard consumption after capacity expansion (namely, it is ensured that data under a certain partition can be pulled, and then messages belonging to the kafka message queue are selected for consumption). In the embodiment of the present application, kafka partition can be 128, and then the leaver can be supported from 4 columns- >8 columns- >16 columns- > … … - >128 columns, and if the number of columns exceeds 128, the lost consumption will be caused.

Referring to fig. 7, fig. 7 is a schematic diagram of state 2 in a master node capacity expansion flow provided in the embodiment of the present application. Here, in state 2 in the master node capacity expansion flow, the configserver may be notified through an external command to start the master capacity expansion flow, select one row (i.e., the target number of the master node after capacity expansion) of the slave node, switch the state of the selected slave node to the master-try state (i.e., the trial state), and synchronize all proxyservers through heartbeat. At this time, there are a row of official master nodes "o-master (including o-master0 and o-master 1)" in the cluster, and a row of master nodes "n-master (including n-master0 to n-master 3)" in the master-try state "

Referring to fig. 8, fig. 8 is a schematic view of state 3 in a master node capacity expansion flow provided in the embodiment of the present application. Here, in state 3 in the master node capacity expansion flow, where,

(1) After the state switching in the state 2, the proxyserver receives the heartbeat of two rows of masters, switches the message sending mode into a slow mode, namely reduces the updating speed of the read-write request message of the service side, and sends msg with the flag1 to the o-master;

(2) n-master (try) consumes all write messages stored before the o-master node from the full index to realize the trace message, so as to ensure the data synchronization from the o-master to the n-master, and when the trace message lag (i.e. the number of the write messages remaining for consumption) is less than a certain value (such as 10), the n-master (try) feeds back to the configserver through heartbeat to indicate that the n-master (try) has the capability of converting from try state to pre state (i.e. the preparation state);

(3) The configserver synchronizes information (i.e., state information) of two rows of masters (o-master and n-master-pre) to all proxyservers through a heartbeat.

Referring to fig. 9, fig. 9 is a schematic diagram of state 4 in a master node capacity expansion flow provided in the embodiment of the present application. Here, in state 4 in the master node capacity expansion flow, where,

(1) After the proxyserver receives the information of the two rows of masters (o-master and n-master-pre) sent by the configserver, the process of double writing the two rows of masters (o-master and n-master-pre) is started, specifically, the process of writing the n-master and then writing the o-master can be carried out, wherein,

the o-master sends the received msg with flag2 to kafka and to the write local storage;

after receiving a message (namely msg with flag 2) with flag2 sent by proxyserver, the n-master does not execute local updating, but stores the message in a set of a memory, and consumes the message with flag1 and flag2 at the same time, namely continues to chase the message, and when consuming the message with flag2, the n-master needs to remove the corresponding item in the set (namely the sequence number of the message with flag2 in the set;

(2) When the number of the remaining messages carrying the flag2 in the set of the n-master is smaller than a certain value (such as 10), the n-master acquires the appointed heartbeat from the configserver, finishes the appointed, namely, the pre state is switched to the online state, and the o-master state continues to maintain the online state.

(3) Since the numbers of o-masters and n-masters are not identical, the o-master cannot go down line directly when there is an n-master being nominated at this time, since the n-master is likely to carry only a partial flow of the o-master.

Referring to fig. 10, fig. 10 is a schematic diagram of state 5 in a master node capacity expansion flow provided in the embodiment of the present application. Here, in state 5 in the master node capacity expansion flow, where,

(1) proxyserver gradually sends the traffic on o-master (msg with no flag) only to n-master, with an increasing number of n-master upgrades to online state. The n-master receiving the msg with no flag needs to search the message in the local set first to see whether the same mainkey message exists or not, if yes, the n-master returns failure to proxyserver to ensure the order preservation of the message, if not, the received message is written into a kafka message queue first and then written into the local storage, and then the message returns to proxyserver successfully;

(2) After all n-masters are switched to the online state and passed to the proxyserver nodes via the configserver heartbeat, each proxyserver node may send a write request to the n-master. At the moment, the o-master flow is cleared, the read message is treated the same as the write message in the latest 2 minutes, and the read request out of 2 minutes is routed to the rest slave nodes;

(3) And if the configserver finds that the n-masters in the cluster are all set to be in an online state through self-checking and lasts for 3 heartbeat cycles with proxyserver, all the o-masters are set to be offline.

(4) When the o-master of the whole cluster is in the offline state by calling the interface of the configserver, the o-master is offline, and when the machine needs to be expanded, the o-master can be online again as slave.

Referring to fig. 11, fig. 11 is a schematic diagram of state 6 in a host node capacity expansion process according to an embodiment of the present application. Here, the state 6 is a state after the master node is completely expanded, and in the state 6, the n-master stops pulling the kafka message along with the clearing of the message in the own internal set, and directly discards the message if the message other than the flag2 is pulled. And when the expansion of the master node is required, the o-master node is taken as a standby node and becomes a new master node again based on the expansion mode. The single write master of proxyserver can then revert to the non-slow write mode.

In practical application, the slow write strategy can be eliminated, and conventionally, the write traffic speed supported by the cluster is far less than the limit traffic of the system, so the slow write strategy is added to better and more quickly switch the n-master from the three states of try- > pre- > online so as to shorten the switching flow of the whole release. In addition, the try state of the n-master can be omitted, and two-state switching of pre- > online can be used.

By applying the embodiment of the application, the problems that external read-write service is transparent and zero perception is given to a business party when the master node expands capacity, and the situations of data relocation and reading of a new master and an old master are avoided can be effectively solved, the usability of storage calculation type service is greatly improved, and the capacity expansion operability is reduced.

Continuing with the exemplary structure of the node capacity expansion device 655 implemented as software modules provided by the present application, in some embodiments, as shown in fig. 2, the software modules stored in the node capacity expansion device 655 of the memory 650 may include:

an obtaining module 6551, configured to obtain, by a standby node in a standby state, a plurality of first messages written by the host node, where the standby node in the standby state is obtained by performing state switching on a target number of standby nodes in response to a node capacity expansion instruction for the management node;

a caching module 6552, configured to, in a process of acquiring a plurality of first messages, when a message write request is received, cache a second message corresponding to the message write request;

a sending module 6553, configured to release the second message when a first message consistent with the second message exists in the plurality of first messages, and send first notification information to the management node when the number of remaining messages cached by the standby node is lower than a first number threshold, so that the management node switches, based on the first notification information, the standby node that sends the first notification information from the standby state to an online state, so as to perform message writing as a master node;

the first notification information is used for indicating that the standby node in the standby state can be switched to the on-line state.

An embodiment of the present application further provides an electronic device, where the electronic device includes:

a memory for storing executable instructions;

Embodiments of the present application also provide a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and executes the computer instruction, so that the computer device executes the node capacity expansion method provided by the embodiment of the application.

The embodiment of the present application further provides a computer-readable storage medium, which stores executable instructions, and when the executable instructions are executed by a processor, the node capacity expansion method provided in the embodiment of the present application is implemented.

In some embodiments, the computer-readable storage medium may be memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

In some embodiments, executable instructions may be written in any form of programming language (including compiled or interpreted languages), in the form of programs, software modules, scripts or code, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

By way of example, executable instructions may correspond, but do not necessarily have to correspond, to files in a file system, and may be stored in a portion of a file that holds other programs or data, such as in one or more scripts in a hypertext Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub-programs, or portions of code).

By way of example, executable instructions may be deployed to be executed on one computing device or on multiple computing devices at one site or distributed across multiple sites and interconnected by a communication network.

The above description is only an example of the present application, and is not intended to limit the scope of the present application. Any modification, equivalent replacement, and improvement made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A node capacity expansion system, the system comprising: a management node, a primary node, and at least one backup node, wherein,

the management node is used for responding to the node capacity expansion instruction and switching the node state of the standby nodes with the target quantity into a preparation state;

the standby node is used for acquiring a plurality of first messages written by the main node when in a preparation state, and caching a second message corresponding to a message write request when receiving the message write request in the process of acquiring the plurality of first messages;

2. The system of claim 1, wherein the management node is further configured to switch the node status of a target number of standby nodes to a trial status in response to the node capacity expansion instruction;

the standby node is further configured to, when the standby node is in a trial state, acquire a plurality of first messages written by the master node, and in a process of acquiring the plurality of first messages, when it is determined that the number of first messages that are not acquired is lower than a second number threshold, send second notification information to the management node;

the management node is further configured to receive the second notification information, and switch the standby node that sends the second notification information from the trial state to the standby state.

3. The system of claim 1, wherein the system further comprises: a proxy node;

the management node is further configured to send state synchronization information to the proxy node when the standby node is switched over;

and the proxy node is used for receiving the state synchronization information and updating the node state of the corresponding standby node based on the state synchronization information.

4. The system of claim 3,

the proxy node is further configured to reduce a sending rate of the message write request when the state synchronization information indicates that the standby node is in a prepared state.

5. The system of claim 3,

the proxy node is further configured to send a message write request carrying a target identifier to the standby node when a message write instruction is received and the state synchronization information indicates that the standby node is in a standby state;

the target identifier is used for indicating a corresponding standby node to cache a message corresponding to a message write request carrying the target identifier;

and the standby node is further configured to cache a second message corresponding to the message write request when receiving the message write request carrying the target identifier.

6. The system of claim 5,

the proxy node is further configured to send a first message write request that does not carry the target identifier to the standby node when the number of standby nodes that receive the message write instruction and are in an online state reaches a third number threshold;

the standby node is further used for determining whether a target message corresponding to the first message write request exists in the standby node when the first message write request is received;

when determining that the target message corresponding to the first message writing request exists, canceling to execute message writing corresponding to the first message writing request;

and when determining that the target message corresponding to the first message writing request does not exist, writing the message corresponding to the first message writing request.

7. The system of claim 1,

the standby node is further configured to stop acquiring the plurality of first messages written by the master node when the number of the cached remaining messages is zero.

8. The system of claim 1, wherein the target number is greater than an original number of master nodes;

and the management node is further used for switching the node state of the main node to the offline state when the standby nodes of the target number are in the online state, so that the standby nodes can be used online when a standby node capacity expansion instruction is received.

9. The system of claim 1, wherein the number of the standby nodes is greater than the target number, and when the target number of the standby nodes performs message writing as a primary node, the standby nodes other than the target number of the standby nodes are mirror nodes of the primary node;

the standby node is used for responding to a message writing request when serving as a main node, writing a third message corresponding to the message writing request into a message queue, and writing the third message into a local storage;

the other standby nodes are used for acquiring the third message from the message queue and writing the third message into a local storage;

and when receiving the message reading request, returning a third message corresponding to the message reading request.

10. A node capacity expansion method is characterized by being applied to a node capacity expansion system comprising a management node, a main node and at least one standby node; the method comprises the following steps:

And the management node switches the standby node sending the first notification information from a standby state to an online state based on the first notification information, and the standby node is used as a main node to write in messages.