CN115118779A - Method, system, device and medium for building cluster based on centralized storage - Google Patents
Method, system, device and medium for building cluster based on centralized storage Download PDFInfo
- Publication number
- CN115118779A CN115118779A CN202210732771.4A CN202210732771A CN115118779A CN 115118779 A CN115118779 A CN 115118779A CN 202210732771 A CN202210732771 A CN 202210732771A CN 115118779 A CN115118779 A CN 115118779A
- Authority
- CN
- China
- Prior art keywords
- data
- centralized storage
- cluster
- node
- storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 73
- 230000015654 memory Effects 0.000 claims abstract description 77
- 238000004590 computer program Methods 0.000 claims description 14
- 230000008569 process Effects 0.000 claims description 9
- 238000012217 deletion Methods 0.000 claims description 2
- 230000037430 deletion Effects 0.000 claims description 2
- 238000010586 diagram Methods 0.000 description 12
- 238000013461 design Methods 0.000 description 11
- 230000001360 synchronised effect Effects 0.000 description 9
- 238000012545 processing Methods 0.000 description 8
- 230000006870 function Effects 0.000 description 6
- 238000004519 manufacturing process Methods 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 4
- PCTMTFRHKVHKIS-BMFZQQSSSA-N (1s,3r,4e,6e,8e,10e,12e,14e,16e,18s,19r,20r,21s,25r,27r,30r,31r,33s,35r,37s,38r)-3-[(2r,3s,4s,5s,6r)-4-amino-3,5-dihydroxy-6-methyloxan-2-yl]oxy-19,25,27,30,31,33,35,37-octahydroxy-18,20,21-trimethyl-23-oxo-22,39-dioxabicyclo[33.3.1]nonatriaconta-4,6,8,10 Chemical compound C1C=C2C[C@@H](OS(O)(=O)=O)CC[C@]2(C)[C@@H]2[C@@H]1[C@@H]1CC[C@H]([C@H](C)CCCC(C)C)[C@@]1(C)CC2.O[C@H]1[C@@H](N)[C@H](O)[C@@H](C)O[C@H]1O[C@H]1/C=C/C=C/C=C/C=C/C=C/C=C/C=C/[C@H](C)[C@@H](O)[C@@H](C)[C@H](C)OC(=O)C[C@H](O)C[C@H](O)CC[C@@H](O)[C@H](O)C[C@H](O)C[C@](O)(C[C@H](O)[C@H]2C(O)=O)O[C@H]2C1 PCTMTFRHKVHKIS-BMFZQQSSSA-N 0.000 description 3
- 230000009471 action Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000003993 interaction Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a method, a system, equipment and a medium for building a cluster based on centralized storage, wherein the method comprises the steps of allocating memory space for writing of data of each node in the cluster, controlling the writing of the data of the nodes into a memory and combining the written data for centralized storage; when the centralized storage is carried out, a second cache for controlling data reading and writing is added; the cache uses a local disk for storing incremental data. And when the nodes in the cluster need to delete data, directly deleting the data from the centralized storage. Based on the method, the system, the equipment and the medium for building the cluster based on the centralized storage are also provided. The invention ensures the consistency of data by configuring all nodes of the cluster to use the same centralized storage, solves the problem of data security, can ensure that the data used by each node is consistent by using the centralized storage, and can ensure the high availability and throughput of cluster services and the data security by using the same centralized storage for a plurality of nodes.
Description
Technical Field
The invention belongs to the technical field of cluster building, and particularly relates to a method, a system, equipment and a medium for building a cluster based on centralized storage.
Background
rabbitmq: message middleware is used for producing and consuming messages. After the client is connected with the rabbitmq service, the client can send a message to a server end of the rabbitmq and receive the message from the server end, wherein the sending of the message is production, and the receiving of the message is consumption. rabbitmq is used in many products as a mature, widely used messaging middleware. For high availability of services and throughput of rabbitmq services, rabbitmq services are typically deployed in clusters.
Currently used rabbitmq is a distributed storage architecture, with data stored on multiple nodes. Fig. 1 shows a data storage situation of a general rabbitmq cluster. Each node uses local storage data (local disk), and synchronization of data is required among each node of the cluster. Each node within the cluster may provide a service. This method has a disadvantage that when the node is abnormal, the respective data of the nodes in the cluster may be inconsistent. If the inconsistency cannot be repaired by the synchronization of the data in the cluster, the rabbitmq service cannot be started. This problem is the data security problem of rabbitmq.
Disclosure of Invention
In order to solve the technical problems, the invention provides a method, a system, equipment and a medium for building a cluster based on centralized storage, which ensure the security of cluster data on the basis of keeping the high available function of the cluster on the premise of not obviously reducing the throughput of the cluster.
In order to achieve the purpose, the invention adopts the following technical scheme:
a cluster building method based on centralized storage comprises the following steps:
allocating memory space for writing of each node data in the cluster, controlling the writing of the node data into the memory and combining the written data for centralized storage;
when the centralized storage is carried out, a second cache for controlling data reading and writing is added; the cache adopts a local disk for storing incremental data.
Further, the method also includes deleting data directly from the centralized storage when a node in the cluster needs to delete data.
Further, the data of each node in the cluster is located in the first cache, and the data of each node in the cluster adopts memory cache data.
Furthermore, in the process of controlling the data of the nodes to be written into the memory, global control is added, so that only one node is written in within a period of time, and the next node can be written in after the current node is written in.
Further, the method for merging the written data for centralized storage includes:
only one copy of the same part in the data of each node is stored in a centralized storage;
different parts of the data of each node are respectively stored in a centralized storage.
Further, after the data increment is stored, the data read-write control periodically controls data to be merged and then written into the centralized storage.
The invention also provides a system for building the cluster based on the centralized storage, which comprises the following steps: a write module and a storage module;
the write-in module is used for distributing memory space for writing in of each node data in the cluster, controlling the writing in of the node data into the memory and combining the written data for centralized storage;
the storage module is used for adding a second cache for data read-write control when performing centralized storage; the cache adopts a local disk for storing incremental data.
Further, the system also comprises a deleting module;
and the deleting module is used for directly deleting the data from the centralized storage when the nodes in the cluster need to delete the data.
The invention also proposes a device comprising:
a memory for storing a computer program;
a processor for implementing the method steps as described when executing the computer program.
The invention also proposes a readable storage medium having stored thereon a computer program which, when being executed by a processor, carries out the method steps.
The effect provided in the summary of the invention is only the effect of the embodiment, not all the effects of the invention, and one of the above technical solutions has the following advantages or beneficial effects:
the invention provides a method and a system for building a cluster based on centralized storage, wherein the method comprises the steps of allocating memory space for writing of data of each node in the cluster, controlling the writing of the data of the nodes into a memory and combining the written data for centralized storage; when the centralized storage is carried out, a second cache for controlling data reading and writing is added; the cache adopts a local disk for storing incremental data. The method also includes deleting directly from the centralized storage when a node in the cluster needs to delete data. Based on a method for building a cluster based on centralized storage, a system, equipment and a medium for building a cluster based on centralized storage are also provided. The invention ensures the consistency of data by configuring all nodes of the cluster to use the same centralized storage, and solves the problem of data security. The centralized storage can ensure that the data used by each node is consistent, and the multiple nodes using the same centralized storage can ensure high availability and throughput of cluster services and data security.
In the invention, the Rabbitmq service does not use local storage, uses cache data, and does not synchronize data among all nodes of the cluster any longer, thereby ensuring the security of cluster data on the basis of keeping the high available function of the cluster on the premise of not obviously reducing the throughput of the cluster.
After the data increment is stored, the data read-write control part periodically merges the data and writes the data into the centralized storage, and merges the data and writes the data into the centralized storage through time delay, so that the storage and writing times are reduced.
Drawings
FIG. 1 is a schematic diagram of a rabbitmq cluster structure commonly used in the prior art;
fig. 2 is a schematic diagram of a structure of a centrally stored rabbitmq cluster according to embodiment 1 of the present invention;
fig. 3 is a flowchart of a method for building a cluster based on centralized storage according to embodiment 1 of the present invention;
fig. 4 is a schematic structural diagram of writing and merging control node data into a memory according to embodiment 1 of the present invention;
fig. 5 is a schematic structural diagram of incremental saving in embodiment 1 of the present invention;
fig. 6 is a schematic diagram of a system for building a cluster based on centralized storage according to embodiment 2 of the present invention.
Detailed Description
In order to clearly explain the technical features of the present invention, the following detailed description of the present invention is provided with reference to the accompanying drawings. The following disclosure provides many different embodiments, or examples, for implementing different features of the invention. To simplify the disclosure of the present invention, specific example components and arrangements are described below. Furthermore, the present invention may repeat reference numerals and/or letters in the various examples. This repetition is for the purpose of simplicity and clarity and does not in itself dictate a relationship between the various embodiments and/or configurations discussed. It should be noted that the components illustrated in the figures are not necessarily drawn to scale. Descriptions of well-known components and processing techniques and procedures are omitted so as to not unnecessarily limit the invention.
Example 1
The embodiment 1 of the invention provides a method for building a cluster based on centralized storage, which ensures the consistency of data by configuring all nodes of the cluster and using the same centralized storage, and solves the problem of data security in the prior art.
Fig. 2 is a schematic diagram of a structure of a centrally stored rabbitmq cluster according to embodiment 1 of the present invention; the centralized storage can ensure that the data used by each node is consistent, and the multiple nodes using the same centralized storage can ensure high availability and throughput of cluster services and data security.
Fig. 3 is a flowchart of a method for building a cluster based on centralized storage according to embodiment 1 of the present invention.
In step S300, allocating a memory space for writing of each node data in the cluster, controlling writing of the node data into the memory and merging the written data for centralized storage;
in the application, the Rabbitmq service does not use local storage and uses cache data, data is not synchronized among all nodes in a cluster, and data synchronization is not performed among all nodes in the cluster.
In an original rabbitmq cluster architecture, each node uses local storage data (local disk), and data synchronization needs to be performed among the nodes of the cluster. Each node within the cluster may provide a service.
The invention designs that each node uses centralized storage, so that each node does not need to store data in a local disk, and each node does not need to synchronize the data (the data are all written into the centralized storage). A block of area may be allocated in the memory using the cache data, and the rabbitmq service reads and writes data from the memory.
Fig. 4 is a schematic structural diagram of writing and merging of control node data into a memory according to embodiment 1 of the present invention.
And in the process of controlling the data of the nodes to be written into the memory, global control is added to ensure that only one node is written in within a period of time, and the next node can be written in after the current node is written in.
The rabbitmq data of each node in the cluster needs to be written into the centralized storage. The data read-write control section is required to control the reading and writing of data. The use of centralized storage has a significant problem that data written by each node may generate conflicts, for example, data written by node 1 is ABC, data written by node 2 is BCD, and direct write storage may generate errors. The more general processing mode is to add a global control, when the node 1 writes, other nodes cannot write, after the node 1 finishes writing, the other node can write, and only one writing operation needs to be ensured in one time.
The method for solving the problem in the invention is to control the data writing through the data reading and writing control part. The data control part allocates the space (memory) written by each node, and after the data of each node is written, the data are merged.
The method for merging the written data for centralized storage comprises the following steps: only one copy of the same part in the data of each node is stored in a centralized storage; different parts of the data of each node are respectively stored in a centralized storage.
In fig. 4, data written in the node 1 is ABC, data written in the node 2 is BCD, and data written in the node 3 is ACD.
The data read-write control part allocates memory space in advance, stores the write-in data of 3 nodes, and then merges the data according to the write-in data of the nodes, wherein the merged data is ABCD.
Since rabbitmq is the message middleware, the message is composed of two parts, namely a message header and a message content. The message header contains some characteristics and attributes of the message, and the message content is the specific data of the message. The messages can be classified and combined according to the information of the message headers.
When the centralized storage is performed in step S310, a second cache for data read-write control is added; the cache adopts a local disk for storing incremental data.
Due to the production and consumption of the rabbitmq messages, the data is changed in real time in order to reduce the number of times data is written to the centralized storage. The design of a cache is introduced in the data read-write control part, and the data in the cache is written into the centralized storage after a certain delay time.
The cache design of the data read-write control part stores the data in a local disk, and the storage mode is increment storage.
Fig. 5 is a schematic structural diagram of incremental saving in embodiment 1 of the present invention; after the data increment is saved, the data read-write control part periodically combines the data and writes the data into the centralized storage.
The invention also includes deleting directly from the centralized storage when the nodes in the cluster need to delete data.
In fig. 5, if a and D need to be deleted in node 3, they are deleted in the centralized storage, and the BCE remains in the centralized storage.
The Rabbitmq uses a local disk to store data, the method does not use the local disk any more, and uses memory cache data which is read from a centralized storage.
In the method for building the cluster based on the centralized storage provided by embodiment 1 of the present invention, all nodes of the cluster are configured to use the same centralized storage to ensure the consistency of data, so as to solve the problem of data security. The centralized storage can ensure that the data used by each node is consistent, and the multiple nodes using the same centralized storage can ensure high availability and throughput of cluster services and data security.
In the method for building the cluster based on the centralized storage provided by embodiment 1 of the invention, the Rabbitmq service does not use local storage, uses cache data, and does not synchronize data among all nodes of the cluster any more, so that the safety of cluster data is ensured on the basis of keeping the high available function of the cluster on the premise of not obviously reducing the throughput of the cluster.
After data increment is stored in the method for building the cluster based on the centralized storage, which is provided by the embodiment 1 of the invention, the data read-write control part periodically merges the data and writes the merged data into the centralized storage, and writes the merged data into the centralized storage by delaying, so that the storage and writing times are reduced.
Example 2
Based on a method for building a cluster based on centralized storage provided by embodiment 1 of the present invention, embodiment 2 of the present invention provides a system for building a cluster based on centralized storage, and as shown in fig. 6, the system for building a cluster based on centralized storage provided by embodiment 2 of the present invention is schematically illustrated, and includes a write-in module and a storage module;
the write-in module is used for distributing memory space for writing in of each node data in the cluster, controlling the writing in of the node data into the memory and combining the written data for centralized storage;
the storage module is used for increasing a second cache for data read-write control when centralized storage is carried out; the cache adopts a local disk for storing incremental data.
The system also includes a deletion module; and the deleting module is used for directly deleting the data from the centralized storage when the nodes in the cluster need to delete the data.
The data of each node in the cluster in the write-in module is located in the first cache, and the data of each node in the cluster adopts memory cache data.
And in the process of controlling the data of the nodes to be written into the memory, global control is added to ensure that only one node is written in within a period of time, and the next node can be written in after the current node is written in.
The method for merging the written data for centralized storage comprises the following steps: only one copy of the same part in the data of each node is stored in a centralized storage; the different parts of the data of each node are respectively stored in a centralized storage
In the storage module, data is changed in real time due to the production and consumption of the rabbitmq message, so as to reduce the number of times data is written into the centralized storage. The design of a cache is introduced in the data read-write control part, and the data in the cache is written into the centralized storage after a certain delay time.
The cache design of the data read-write control part stores the data in a local disk, and the storage mode is increment storage. After the data increment is saved, the data read-write control part periodically combines the data and writes the data into the centralized storage.
The system for building the cluster based on the centralized storage, which is provided by the embodiment 2 of the invention, ensures the consistency of data by configuring all nodes of the cluster to use the same centralized storage, and solves the problem of data security. The centralized storage can ensure that the data used by each node is consistent, and the multiple nodes using the same centralized storage can ensure high availability and throughput of cluster services and data security.
In the system for building the cluster based on the centralized storage provided by embodiment 2 of the present invention, the Rabbitmq service does not use local storage, uses cache data, and does not synchronize data among nodes in the cluster any more, so that the security of cluster data is ensured on the basis of maintaining the high available function of the cluster on the premise of not significantly reducing the throughput of the cluster.
After data increment storage in the system for building the cluster based on the centralized storage, which is provided by the embodiment 2 of the invention, is performed, the data read-write control part periodically merges the data and writes the merged data into the centralized storage, and the merged data is written into the centralized storage by delaying, so that the storage and writing times are reduced.
Example 3
The invention also proposes a device comprising:
a memory for storing a computer program;
a processor for implementing the method steps when executing the computer program as follows:
fig. 3 is a flowchart of a method for building a cluster based on centralized storage according to embodiment 1 of the present invention.
In step S300, allocating a memory space for writing of each node data in the cluster, controlling writing of the node data into the memory and merging the written data for centralized storage;
in the application, the Rabbitmq service does not use local storage and uses cache data, data is not synchronized among all nodes in a cluster, and data synchronization is not performed among all nodes in the cluster.
In an original rabbitmq cluster architecture, each node uses local storage data (local disk), and data synchronization needs to be performed among the nodes of the cluster. Each node within the cluster may provide a service.
The invention designs that each node uses centralized storage, so that each node does not need to store data in a local disk, and each node does not need to synchronize the data (the data are all written into the centralized storage). A block of area may be allocated in the memory using the cache data, and the rabbitmq service reads and writes data from the memory.
Fig. 4 is a schematic structural diagram of writing and merging of control node data into a memory according to embodiment 1 of the present invention.
And in the process of controlling the data of the nodes to be written into the memory, global control is added to ensure that only one node is written in within a period of time, and the next node can be written in after the current node is written in.
The rabbitmq data of each node in the cluster needs to be written into the centralized storage. The data read-write control section is required to control the reading and writing of data. The use of centralized storage has a obvious problem that data written by each node can generate conflict, for example, data written by node 1 is ABC, data written by node 2 is BCD, and direct write storage can generate errors. The more general processing mode is to add a global control, when the node 1 writes, other nodes cannot write, after the node 1 finishes writing, the other node can write, and only one writing operation needs to be ensured in one time.
The method for solving the problem in the invention is to control the data writing through the data reading and writing control part. The data control part allocates the space (memory) written by each node, and after the data of each node is written, the data are merged.
The method for merging the written data for centralized storage comprises the following steps: only one copy of the same part in the data of each node is stored in a centralized storage; different parts of the data of each node are respectively stored in a centralized storage.
In fig. 4, data written in node 1 is ABC, data written in node 2 is BCD, and data written in node 3 is ACD.
The data read-write control part allocates memory space in advance, stores the write-in data of 3 nodes, and then merges the data according to the write-in data of the nodes, wherein the merged data is ABCD.
Since rabbitmq is the message middleware, the message is composed of two parts, namely a message header and a message content. The message header contains some characteristics and attributes of the message, and the message content is the specific data of the message. The messages can be classified and combined according to the information of the message headers.
When the centralized storage is performed in step S310, a second cache for data read-write control is added; the cache adopts a local disk for storing incremental data.
Due to the production and consumption of the rabbitmq messages, the data is changed in real time in order to reduce the number of times data is written to the centralized storage. The design of a cache is introduced in the data read-write control part, and the data in the cache is written into the centralized storage after a certain delay time.
The cache design of the data read-write control part stores the data in a local disk, and the storage mode is increment storage.
Fig. 5 is a schematic structural diagram of incremental saving in embodiment 1 of the present invention; after the data increment is stored, the data read-write control part periodically merges the data and writes the data into the centralized storage.
The invention also includes deleting directly from the centralized storage when the nodes in the cluster need to delete data.
In fig. 5, if a and D need to be deleted in the node 3, they are deleted in the centralized storage, and the BCE remains in the centralized storage.
The Rabbitmq uses a local disk to store data, the method does not use the local disk any more, and uses memory cache data which is read from a centralized storage.
The device provided in embodiment 3 of the present invention ensures data consistency by configuring all nodes of a cluster to use the same centralized storage, thereby solving the problem of data security. The centralized storage can ensure that the data used by each node is consistent, and the multiple nodes using the same centralized storage can ensure high availability and throughput of cluster services and data security.
In the apparatus provided in embodiment 3 of the present invention, the Rabbitmq service does not use local storage, uses cache data, and does not synchronize data among nodes in a cluster any more, so that on the premise of not significantly reducing cluster throughput, security of cluster data is ensured on the basis of maintaining high available functions of a cluster.
After the data increment is stored in the device provided in embodiment 3 of the present invention, the data read/write control portion periodically merges the data and writes the merged data into the centralized storage, and writes the merged data into the centralized storage by delaying, thereby reducing the number of times of storage and writing.
Need to explain: the technical solution of the present invention also provides an electronic device, including: the communication interface can carry out information interaction with other equipment such as network equipment and the like; and the processor is connected with the communication interface to realize information interaction with other equipment, and is used for executing the method for building the cluster based on the centralized storage provided by one or more technical schemes when running a computer program, and the computer program is stored on the memory. Of course, in practice, the various components in an electronic device are coupled together by a bus system. It will be appreciated that a bus system is used to enable the communication of the connections between these components. The bus system includes a power bus, a control bus, and a status signal bus in addition to the data bus. The memory in the embodiments of the present application is used to store various types of data to support the operation of the electronic device. Examples of such data include: any computer program for operating on an electronic device. It will be appreciated that the memory can be either volatile memory or nonvolatile memory, and can include both volatile and nonvolatile memory. Among them, the nonvolatile Memory may be a Read Only Memory (ROM), a Programmable Read Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), a magnetic random access Memory (FRAM), a Flash Memory (Flash Memory), a magnetic surface Memory, an optical disk, or a Compact Disc Read-Only Memory (CD-ROM); the magnetic surface storage may be disk storage or tape storage. The volatile memory may be a Random Access Memory (RAM) which serves as an external cache. By way of illustration and not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), Enhanced Synchronous Dynamic Random Access Memory (Enhanced DRAM), Synchronous Dynamic Random Access Memory (SLDRAM), Direct Memory (DRmb Access), and Random Access Memory (DRAM). The memories described in the embodiments of the present application are intended to comprise, without being limited to, these and any other suitable types of memory. The method disclosed in the embodiments of the present application may be applied to a processor, or may be implemented by a processor. The processor may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in a processor or instructions in the form of software. The processor may be a general purpose processor, a DSP (Digital Signal Processing, i.e., a chip capable of implementing Digital Signal Processing technology), or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc. The processor may implement or perform the methods, steps, and logic blocks disclosed in the embodiments of the present application. A general purpose processor may be a microprocessor or any conventional processor or the like. The steps of the method disclosed in the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium located in a memory where a processor reads the programs in the memory and in combination with its hardware performs the steps of the method as previously described. When the processor executes the program, corresponding processes in the methods of the embodiments of the present application are implemented, and for brevity, are not described herein again.
Example 4
The invention also proposes a readable storage medium on which a computer program is stored, which, when executed by a processor, implements the method steps of:
fig. 3 is a flowchart of a method for building a cluster based on centralized storage according to embodiment 1 of the present invention.
In step S300, allocating a memory space for writing of each node data in the cluster, controlling writing of the node data into the memory and merging the written data for centralized storage;
in the application, the Rabbitmq service does not use local storage and uses cache data, data is not synchronized among all nodes in a cluster, and data synchronization is not performed among all nodes in the cluster.
In an original rabbitmq cluster architecture, each node uses local storage data (local disk), and data synchronization needs to be performed among the nodes of the cluster. Each node within the cluster may provide a service.
The invention designs that each node uses centralized storage, so that each node does not need to store data in a local disk, and each node does not need to synchronize the data (the data are all written into the centralized storage). A block of area may be allocated in the memory using the cache data, and the rabbitmq service reads and writes data from the memory.
Fig. 4 is a schematic structural diagram of writing and merging of control node data into a memory according to embodiment 1 of the present invention.
And in the process of controlling the data of the nodes to be written into the memory, global control is added to ensure that only one node is written in within a period of time, and the next node can be written in after the current node is written in.
The rabbitmq data of each node in the cluster needs to be written into the centralized storage. The data read-write control section is required to control the reading and writing of data. The use of centralized storage has a obvious problem that data written by each node can generate conflict, for example, data written by node 1 is ABC, data written by node 2 is BCD, and direct write storage can generate errors. The more general processing mode is to add a global control, when the node 1 writes, other nodes cannot write, after the node 1 finishes writing, the other node can write, and only one writing operation needs to be ensured in one time.
The method for solving the problem in the invention is to control the data writing through the data reading and writing control part. The data control part allocates the space (memory) written by each node, and after the data of each node is written, the data are merged.
The method for merging the written data for centralized storage comprises the following steps: only one copy of the same part in the data of each node is stored in a centralized storage; different parts of the data of each node are respectively stored in a centralized storage.
In fig. 4, data written in node 1 is ABC, data written in node 2 is BCD, and data written in node 3 is ACD.
The data read-write control part allocates memory space in advance, stores the write-in data of 3 nodes, and then merges the data according to the write-in data of the nodes, wherein the merged data is ABCD.
Since rabbitmq is the message middleware, the message is composed of two parts, namely a message header and a message content. The message header contains some characteristics and attributes of the message, and the message content is the specific data of the message. The messages can be classified and combined according to the information of the message headers.
When the centralized storage is performed in step S310, a second cache for data read-write control is added; the cache adopts a local disk for storing incremental data.
Due to the production and consumption of the rabbitmq messages, the data is changed in real time in order to reduce the number of times data is written to the centralized storage. The design of a cache is introduced in a data read-write control part, and data in the cache is written into a centralized storage after a certain delay time.
The cache design of the data read-write control part stores the data in a local disk, and the storage mode is increment storage.
Fig. 5 is a schematic structural diagram of incremental saving in embodiment 1 of the present invention; after the data increment is saved, the data read-write control part periodically combines the data and writes the data into the centralized storage.
The invention also includes deleting directly from the centralized storage when the nodes in the cluster need to delete data.
In fig. 5, if a and D need to be deleted in the node 3, they are deleted in the centralized storage, and the BCE remains in the centralized storage.
The Rabbitmq uses a local disk to store data, the method does not use the local disk any more, uses memory cache data, and the memory cache data are read from centralized storage.
The storage medium provided in embodiment 4 of the present invention ensures data consistency by configuring all nodes of a cluster to use the same centralized storage, thereby solving the problem of data security. The centralized storage can ensure that the data used by each node is consistent, and the multiple nodes using the same centralized storage can ensure high availability and throughput of cluster services and data security.
In the storage medium provided in embodiment 4 of the present invention, the Rabbitmq service does not use local storage, uses cache data, and does not synchronize data among nodes in a cluster any more, so that on the premise of not significantly reducing cluster throughput, security of cluster data is ensured on the basis of maintaining high available functions of a cluster.
In the storage medium provided in embodiment 4 of the present invention, after the data increment is stored, the data read-write control portion periodically merges the data and writes the merged data into the centralized storage, and writes the merged data into the centralized storage by delaying, so as to reduce the number of times of storage and writing.
Embodiments of the present application further provide a storage medium, that is, a computer storage medium, specifically, a computer-readable storage medium, for example, a memory storing a computer program, where the computer program is executable by a processor to perform the steps of the foregoing method. The computer readable storage medium may be Memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface Memory, optical disk, or CD-ROM.
Those of ordinary skill in the art will understand that: all or part of the steps for implementing the method embodiments may be implemented by hardware related to program instructions, and the program may be stored in a computer readable storage medium, and when executed, the program performs the steps including the method embodiments; and the aforementioned storage medium includes: a removable storage device, a ROM, a RAM, a magnetic or optical disk, or various other media that can store program code. Alternatively, the integrated units described above in the present application may be stored in a computer-readable storage medium if they are implemented in the form of software functional modules and sold or used as independent products. Based on such understanding, the technical solutions of the embodiments of the present application may be essentially implemented or portions thereof that contribute to the prior art may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling an electronic device (which may be a personal computer, a server, or a network device) to execute all or part of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a removable storage device, a ROM, a RAM, a magnetic or optical disk, or various other media that can store program code.
For a description of a relevant part in a processing device and a storage medium for building a cluster based on centralized storage provided in an embodiment of the present application, reference may be made to a detailed description of a corresponding part in a method for building a cluster based on centralized storage provided in embodiment 1 of the present application, and details are not described here again.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Furthermore, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include elements inherent in the list. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element. In addition, parts of the above technical solutions provided in the embodiments of the present application, which are consistent with the implementation principles of corresponding technical solutions in the prior art, are not described in detail so as to avoid redundant description.
Although the embodiments of the present invention have been described with reference to the accompanying drawings, the scope of the present invention is not limited thereto. Various modifications and alterations will occur to those skilled in the art based on the foregoing description. And are neither required nor exhaustive of all embodiments. On the basis of the technical scheme of the invention, various modifications or changes which can be made by a person skilled in the art without creative efforts are still within the protection scope of the invention.
Claims (10)
1. A cluster building method based on centralized storage is characterized by comprising the following steps:
allocating memory space for writing of each node data in the cluster, controlling the writing of the node data into the memory and combining the written data for centralized storage;
when the centralized storage is carried out, a second cache for controlling data reading and writing is added; the cache adopts a local disk for storing incremental data.
2. The method for building the cluster based on the centralized storage according to claim 1, wherein the method further comprises deleting the data directly from the centralized storage when the nodes in the cluster need to delete the data.
3. The method for building the cluster based on the centralized storage according to claim 1, wherein the data of each node in the cluster is located in a first cache, and the data of each node in the cluster adopts memory cache data.
4. The method for building the cluster based on the centralized storage is characterized in that global control is added in the process of controlling the data of the nodes to be written into the memory, so that only one node can be written in one time, and the next node can be written in after the current node is written in.
5. The method for building a cluster based on centralized storage according to claim 1, wherein the method for merging written data for centralized storage comprises:
only one copy of the same part in the data of each node is stored in a centralized storage;
different parts of the data of each node are respectively stored in a centralized storage.
6. The method for building a cluster based on centralized storage according to claim 1, wherein after the data are incrementally stored, the data read-write control writes the periodically control data into the centralized storage after merging.
7. A system for building a cluster based on centralized storage, comprising: a write module and a storage module;
the write-in module is used for distributing memory space for writing in of each node data in the cluster, controlling the writing in of the node data into the memory and combining the written data for centralized storage;
the storage module is used for adding a second cache for data read-write control when carrying out centralized storage; the cache adopts a local disk for storing incremental data.
8. The system for building a cluster based on centralized storage according to claim 7, further comprising a deletion module;
and the deleting module is used for directly deleting the data from the centralized storage when the nodes in the cluster need to delete the data.
9. An apparatus, comprising:
a memory for storing a computer program;
a processor for implementing the method steps of any one of claims 1 to 6 when executing the computer program.
10. A readable storage medium, characterized in that the readable storage medium has stored thereon a computer program which, when being executed by a processor, carries out the method steps of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210732771.4A CN115118779B (en) | 2022-06-24 | 2022-06-24 | Method, system, equipment and medium for building cluster based on centralized storage |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210732771.4A CN115118779B (en) | 2022-06-24 | 2022-06-24 | Method, system, equipment and medium for building cluster based on centralized storage |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115118779A true CN115118779A (en) | 2022-09-27 |
| CN115118779B CN115118779B (en) | 2024-09-24 |
Family
ID=83330048
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210732771.4A Active CN115118779B (en) | 2022-06-24 | 2022-06-24 | Method, system, equipment and medium for building cluster based on centralized storage |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115118779B (en) |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103366016A (en) * | 2013-08-01 | 2013-10-23 | 南京大学 | Electronic file concentrated storing and optimizing method based on HDFS |
| CN103401705A (en) * | 2013-07-24 | 2013-11-20 | 佳都新太科技股份有限公司 | Centralized memory management method in video monitoring and control network |
| CN103442036A (en) * | 2013-08-09 | 2013-12-11 | 苏州蓝海彤翔系统科技有限公司 | System integrating design development, post production and data storage and based on cloud platform |
| US20150248352A1 (en) * | 2014-02-28 | 2015-09-03 | Dell Products, L.P. | Methods and systems for optimization of storage performance across multiple compute devices |
| CN107870982A (en) * | 2017-10-02 | 2018-04-03 | 深圳前海微众银行股份有限公司 | Data processing method, system and computer-readable recording medium |
| CN111488128A (en) * | 2019-12-30 | 2020-08-04 | 北京浪潮数据技术有限公司 | Method, device, equipment and medium for updating metadata |
| CN111638853A (en) * | 2020-05-08 | 2020-09-08 | 杭州海康威视系统技术有限公司 | Data storage method and device, storage cluster, gateway equipment and main equipment |
| US20200341855A1 (en) * | 2019-04-28 | 2020-10-29 | Synamedia | Object store specialized backup and point-in-time recovery architecture |
| CN112104729A (en) * | 2020-09-10 | 2020-12-18 | 华云数据控股集团有限公司 | Storage system and caching method thereof |
| CN112925788A (en) * | 2021-01-27 | 2021-06-08 | 上海商汤智能科技有限公司 | Data set management method, system, device, electronic equipment and storage medium |
| CN113138717A (en) * | 2021-04-09 | 2021-07-20 | 锐捷网络股份有限公司 | Node deployment method, device and storage medium |
| CN113868238A (en) * | 2021-10-12 | 2021-12-31 | 湖南快乐阳光互动娱乐传媒有限公司 | Cache refreshing method, related device and readable storage medium |
| US20220004480A1 (en) * | 2019-06-28 | 2022-01-06 | Boe Technology Group Co., Ltd. | Log data collection method, log data collection device, storage medium, and log data collection system |
-
2022
- 2022-06-24 CN CN202210732771.4A patent/CN115118779B/en active Active
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103401705A (en) * | 2013-07-24 | 2013-11-20 | 佳都新太科技股份有限公司 | Centralized memory management method in video monitoring and control network |
| CN103366016A (en) * | 2013-08-01 | 2013-10-23 | 南京大学 | Electronic file concentrated storing and optimizing method based on HDFS |
| CN103442036A (en) * | 2013-08-09 | 2013-12-11 | 苏州蓝海彤翔系统科技有限公司 | System integrating design development, post production and data storage and based on cloud platform |
| US20150248352A1 (en) * | 2014-02-28 | 2015-09-03 | Dell Products, L.P. | Methods and systems for optimization of storage performance across multiple compute devices |
| CN107870982A (en) * | 2017-10-02 | 2018-04-03 | 深圳前海微众银行股份有限公司 | Data processing method, system and computer-readable recording medium |
| US20200341855A1 (en) * | 2019-04-28 | 2020-10-29 | Synamedia | Object store specialized backup and point-in-time recovery architecture |
| US20220004480A1 (en) * | 2019-06-28 | 2022-01-06 | Boe Technology Group Co., Ltd. | Log data collection method, log data collection device, storage medium, and log data collection system |
| CN111488128A (en) * | 2019-12-30 | 2020-08-04 | 北京浪潮数据技术有限公司 | Method, device, equipment and medium for updating metadata |
| CN111638853A (en) * | 2020-05-08 | 2020-09-08 | 杭州海康威视系统技术有限公司 | Data storage method and device, storage cluster, gateway equipment and main equipment |
| CN112104729A (en) * | 2020-09-10 | 2020-12-18 | 华云数据控股集团有限公司 | Storage system and caching method thereof |
| CN112925788A (en) * | 2021-01-27 | 2021-06-08 | 上海商汤智能科技有限公司 | Data set management method, system, device, electronic equipment and storage medium |
| CN113138717A (en) * | 2021-04-09 | 2021-07-20 | 锐捷网络股份有限公司 | Node deployment method, device and storage medium |
| CN113868238A (en) * | 2021-10-12 | 2021-12-31 | 湖南快乐阳光互动娱乐传媒有限公司 | Cache refreshing method, related device and readable storage medium |
Non-Patent Citations (3)
| Title |
|---|
| ANANYA NAG; RAMI J. HADDAD: "A novel centralized storage model for distributed photovoltaic generation systems", 《2017 IEEE INTERNATIONAL CONFERENCE ON SMART GRID COMMUNICATIONS (SMARTGRIDCOMM)》, 19 April 2018 (2018-04-19) * |
| 王宇飞;刘丹;吴嘉生;: "基于OSGi的分布式系统集中日志管理方案", 计算机系统应用, no. 06, 15 June 2017 (2017-06-15) * |
| 王文丰;赵跃龙;曾文英;余斌;: "一种网络存储技术新方案――智能网络磁盘集群存储系统", 小型微型计算机系统, no. 07, pages 2 - 3 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115118779B (en) | 2024-09-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107533438B (en) | Data replication in a memory system | |
| US20190245919A1 (en) | Method and apparatus for information processing, server and computer readable medium | |
| CN103780638A (en) | Data synchronization method and system | |
| CN103577121A (en) | High-reliability linear file access method based on nand flash | |
| CN104156361A (en) | Method and system for achieving data synchronization | |
| CN110502523A (en) | Business datum storage method, device, server and computer readable storage medium | |
| CN106339387B (en) | Increase the method for data synchronization and device of server in a kind of data-base cluster newly | |
| CN112069260B (en) | Data storage and account checking method and system | |
| CN108595347B (en) | Cache control method and device and computer readable storage medium | |
| US20240289352A1 (en) | Data processing method, system, device, computer program product, and storage medium | |
| CN104750755A (en) | Method and system for recovering data after switching between main database and standby database | |
| CN111694863A (en) | Method, system and device for refreshing database cache | |
| CN109189326B (en) | Management method and device of distributed cluster | |
| CN114528029A (en) | Method, system, equipment and storage medium for realizing BIOS option setting | |
| WO2023241528A1 (en) | Data processing method and apparatus | |
| CN115443457A (en) | Transaction processing method, distributed database system, cluster and medium | |
| CN112612851A (en) | Multi-center data synchronization method and device | |
| CN115629901A (en) | Log playback method and device, data recovery method and device and electronic equipment | |
| CN114003181B (en) | Data write mirror image system, method, device, electronic equipment and storage medium | |
| US20230385155A1 (en) | Data Snapshot Method and Apparatus, Computer Device, and Storage Medium | |
| CN115118779A (en) | Method, system, device and medium for building cluster based on centralized storage | |
| CN110895545B (en) | Shared data synchronization method and device | |
| CN112131191B (en) | Management method, device and equipment of NAMENODE file system | |
| CN115129251A (en) | Method, system, equipment and storage medium for replacing storage volume on line | |
| CN117851304B (en) | Hard disk replacement method, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |