CN112988905A - Node memory synchronization method and device for cluster deployment - Google Patents
Node memory synchronization method and device for cluster deployment Download PDFInfo
- Publication number
- CN112988905A CN112988905A CN202110457235.3A CN202110457235A CN112988905A CN 112988905 A CN112988905 A CN 112988905A CN 202110457235 A CN202110457235 A CN 202110457235A CN 112988905 A CN112988905 A CN 112988905A
- Authority
- CN
- China
- Prior art keywords
- service data
- memory
- node
- synchronous
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a node memory synchronization method and a node memory synchronization device for cluster deployment, wherein the method comprises the following steps: after a secondary node logs in a primary node, acquiring synchronous service data required by the secondary node according to synchronous information, wherein the synchronous information comprises a service data type of the synchronous service data; carrying out data structure serialization processing on the synchronous service data to obtain serialized memory service data; and sending the serialized memory service data to the secondary node so that the secondary node can carry out service data synchronization according to the serialized memory service data. According to the service type, the invention uses the secondary node to initiate a synchronization request to the primary node, and then the primary node carries out snapshot on the service data structure, thereby synchronizing the data file to the secondary node through the network. The scheme can be used for various scenes, improves the reading and writing efficiency, and can complete the file transmission function only by using simple protocol constraint.
Description
Technical Field
The present invention relates to the field of data synchronization technologies, and in particular, to a node memory synchronization method and apparatus for cluster deployment.
Background
In a cluster deployment environment, all updates by clients are performed on Primary (Primary) nodes and queries are performed on Secondary (Secondary) nodes. When the secondary node initiates an update request, the latest service data may be acquired from the primary node.
In the prior art, cluster deployment design is mostly performed based on redundancy and fault tolerance of a master-slave structure, for example, Redis, data can be synchronized from a primary node to any number of secondary nodes, and full data synchronization and incremental synchronization are realized. However, the Redis synchronization mechanism is complex, and the application scenario is special, so that synchronization cannot be performed according to the service.
Therefore, a method and an apparatus for synchronizing node memories for cluster deployment are needed to solve the above problems.
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides a node memory synchronization method and device for cluster deployment.
The invention provides a node memory synchronization method for cluster deployment, which comprises the following steps:
after a secondary node logs in a primary node, acquiring synchronous service data required by the secondary node according to synchronous information, wherein the synchronous information comprises a service data type of the synchronous service data;
carrying out data structure serialization processing on the synchronous service data to obtain serialized memory service data;
and sending the serialized memory service data to the secondary node so that the secondary node can carry out service data synchronization according to the serialized memory service data.
According to the node memory synchronization method for cluster deployment provided by the present invention, before the synchronous service data required by the secondary node is acquired according to the synchronization information, the method further includes:
and obtaining login information sent by the secondary node so that the secondary node logs in the primary node, wherein the login information comprises the service data type of each service data in the secondary node.
According to the node memory synchronization method for cluster deployment provided by the invention, the data structure serialization processing is performed on the synchronization service data to obtain the serialized memory service data, and the method comprises the following steps:
judging the data volume of the synchronous service data, and if the data volume is greater than or equal to 4K data, performing data structure serialization processing on the synchronous service data according to the service data type corresponding to the synchronous service data;
and if the data volume is less than 4K data, directly sending the synchronous service data to a secondary node so that the secondary node can carry out service data synchronization according to the synchronous service data.
According to the node memory synchronization method for cluster deployment provided by the present invention, after performing data structure serialization processing on the synchronous service data according to the service data type corresponding to the synchronous service data, the method further includes:
and constructing an index file corresponding to each service data type based on the storage position, the data volume and the version information of the synchronous service data.
The invention also provides a node memory synchronization method for cluster deployment, which comprises the following steps:
according to the service data type of the service data in the memory, building login information;
sending the login information to a primary node to establish a login connection with the primary node;
according to the service data type of service data to be synchronized in a memory, constructing synchronization information, and sending the synchronization information to the primary node;
obtaining serialized memory service data sent by the primary node, and synchronizing service data to be synchronized according to the serialized memory service data, wherein the serialized memory service data is obtained by the primary node performing data structure serialization on synchronous service data required by a secondary node.
According to the node memory synchronization method for cluster deployment provided by the present invention, before the synchronization of the service data to be synchronized according to the serialized memory service data, the method further comprises:
and performing deserialization processing on the serialized memory service data to obtain corresponding synchronous service data so as to perform service data synchronization according to the synchronous service data.
The invention also provides a node memory synchronization device for cluster deployment, which comprises:
the synchronous data acquisition module is used for acquiring synchronous service data required by a secondary node according to synchronous information after the secondary node logs in a primary node, wherein the synchronous information comprises a service data type of the synchronous service data;
the serialization processing module is used for carrying out data structure serialization processing on the synchronous service data to obtain serialized memory service data;
and the synchronous data sending module is used for sending the serialized memory service data to the secondary node so that the secondary node can carry out service data synchronization according to the serialized memory service data.
The invention also provides a node memory synchronization device for cluster deployment, which comprises:
the login information construction module is used for constructing login information according to the service data type of the service data in the memory;
the login connection module is used for sending the login information to a primary node so as to establish login connection with the primary node;
the synchronous information construction module is used for constructing synchronous information according to the service data type of service data to be synchronized in the memory and sending the synchronous information to the primary node;
and the memory data synchronization module is used for acquiring the serialized memory service data sent by the primary node and synchronizing the service data to be synchronized according to the serialized memory service data, wherein the serialized memory service data is obtained by performing data structure serialization processing on the synchronous service data required by the secondary node by the primary node.
The invention further provides an electronic device, which includes a storage, a processor and a computer program stored on the storage and capable of running on the processor, and when the processor executes the program, the steps of any one of the above-mentioned node memory synchronization methods for cluster deployment are implemented.
The present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when being executed by a processor, implements the steps of the method for node memory synchronization for cluster deployment as described in any of the above.
According to the node memory synchronization method and device for cluster deployment, provided by the invention, a synchronization request is initiated to a primary node by using a secondary node according to a service type, and then a service data structure is snapshot by the primary node, so that a data file is synchronized to the secondary node through a network. The scheme can be used for various scenes, improves the reading and writing efficiency, and can complete the file transmission function only by using simple protocol constraint.
Drawings
In order to more clearly illustrate the technical solutions of the present invention or the prior art, the drawings needed for the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.
Fig. 1 is a schematic flow chart of a node memory synchronization method for cluster deployment according to the present invention;
fig. 2 is a second schematic flowchart of a node memory synchronization method for cluster deployment according to the present invention;
FIG. 3 is a diagram illustrating a format of login information provided by the present invention;
FIG. 4 is a diagram illustrating a format of a synchronization message provided by the present invention;
FIG. 5 is a diagram illustrating a synchronization format of a data file according to the present invention;
fig. 6 is a schematic structural diagram of a node memory synchronization apparatus for cluster deployment according to the present invention;
fig. 7 is a second schematic structural diagram of a node memory synchronization apparatus for cluster deployment according to the present invention;
fig. 8 is a schematic structural diagram of an electronic device provided in the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The invention initiates a synchronization request to the primary node through the secondary node according to the service data type, and after the primary node receives the synchronization request, the primary node carries out snapshot processing on the service data structure and synchronizes the data file to the secondary node through the network. In existing solutions, there is no general method to implement this process. Although the application scenarios exist in a partially open source component, such as Redis or Mysql, the implementation is relatively complex, and the service scenario is single. The application scenario of the invention is a master-slave cluster environment, the primary nodes and the secondary nodes are organized in a grouping mode, and one primary node and at least two secondary nodes form a group. And the synchronization operation is initiated by the secondary nodes, after the synchronization is completed, the data of each secondary node is consistent, and the query process of the user side is performed on the secondary nodes. In the invention, the secondary node and the primary node communicate through the TCP long connection, and the primary node and the secondary node are configured through the configuration file without automatic election.
Fig. 1 is a schematic flow chart of a node memory synchronization method for cluster deployment provided by the present invention, and as shown in fig. 1, the present invention provides a node memory synchronization method for cluster deployment, which includes:
In the invention, after the service of the secondary node is started, the secondary node establishes connection with the primary node and sends a login request to the primary node, and after the login is successful, the secondary node is in a state of receiving a message. The primary node is used as an execution main body for explanation, and after receiving the synchronization information, the primary node confirms the synchronization service data which needs to be updated by the secondary node according to the service data type contained in the information.
And 102, performing data structure serialization processing on the synchronous service data to obtain serialized memory service data.
In the invention, after confirming the service data which needs to synchronize the secondary node, the primary node carries out data structure serialization processing on the service data to obtain the serialized memory service data. The business data of the invention is memory data, the data structure of the memory can be represented by a structure body, and under the environment of C/C + +/golang programming language, a structure body is defined by using Struct keywords, so that the data in the structure body can be traversed according to the sequence defined in the structure body by carrying out serialization processing on the data structure, and then the data are sequentially stored in a continuous address space.
In the invention, a primary node opens an empty file from a hard disk in a binary mode, maps the empty file to a memory through an Mmap, serializes data used for synchronizing a secondary node in the memory, writes the serialized data into the empty file to form a data file, and sends the data file to the secondary node through network connection, so that the secondary node synchronizes according to the data file.
According to the node memory synchronization method for cluster deployment, a synchronization request is initiated to a primary node by using a secondary node according to the service type, and a service data structure is snapshot through the primary node, so that a data file is synchronized to the secondary node through a network. The scheme can be used for various scenes, improves the reading and writing efficiency, and can complete the file transmission function only by using simple protocol constraint.
On the basis of the above embodiment, before the acquiring, according to the synchronization information, the synchronization service data required by the secondary node, the method further includes:
and obtaining login information sent by the secondary node so that the secondary node logs in the primary node, wherein the login information comprises the service data type of each service data in the secondary node.
In the invention, when the secondary node is connected with the primary node, a 4-byte Magic-Number is sent for identity confirmation and communication protocol version identification, and when the communication protocol content is changed, the Magic-Number is added with 1. It should be noted that there is no additional authentication mechanism in the cluster of the present invention. After the secondary node is started, the secondary node actively establishes connection with the primary node through a configured TCP port, and after the connection is established, the secondary node sends a Login message to the primary node, wherein the Login message comprises the service data type of the secondary node, so that the primary node performs data synchronization according to the service data type when performing data synchronization subsequently, and the efficiency of the data synchronization is improved.
On the basis of the foregoing embodiment, the performing data structure serialization processing on the synchronous service data to obtain serialized memory service data includes:
judging the data volume of the synchronous service data, and if the data volume is greater than or equal to 4k of data, performing data structure serialization processing on the synchronous service data according to the service data type corresponding to the synchronous service data;
and if the data volume is less than 4k of data, directly sending the synchronous service data to a secondary node so that the secondary node can carry out service data synchronization according to the synchronous service data.
In the invention, when the data volume of the synchronous service data needing to be serialized in the memory of the primary node is less than 4K data, a serialized data file cannot be generated, and when the synchronization is carried out, the primary node directly sends the synchronous service data to the secondary node through the network. If the data volume needing to be serialized in the memory of the primary node is larger than 4K data, an empty file is opened from the hard disk in a binary mode, the empty file is mapped to the memory through Mmap, the data in the memory is serialized, then the data is written into the empty file to form a data file, and the data file is sent to the secondary node. In the present invention, the data structure of the memory can be represented by a structure, and the structure generally includes the following data types:
the method comprises the steps that firstly, types occupying the fixed memory size comprise int, float, double, pool and the like, and for the types with the fixed memory size, the types are written into a target address space in sequence according to the sequence defined in a structure body;
secondly, types occupying indefinite memory sizes comprise string types, dictionary (map) types, array types, slice types, structure types, pointer types and the like, and for the data of the types, the specific writing mode of each type is as follows: string type, stored with two bytes length + content; map type, paired key, value are stored in sequence; the array type is that all elements in the array are circularly traversed, and each element is processed according to the specific type of the element; the structure type sequentially traverses each element in the structure, can be realized by reflection in Golang, additionally records the definition sequence of the structure members in C/C + +, the size of the members in the structure can be calculated by using offset of, the type of the members can be determined by using type of, the members are stored in a target address space once according to the sequence and the size, and the members are processed according to the same sequence and size during deserialization; and obtaining the structure pointer type to obtain the corresponding structure, and converting according to the structure type.
On the basis of the foregoing embodiment, after performing data structure serialization processing on the synchronous service data according to the service data type corresponding to the synchronous service data, the method further includes:
and constructing an index file corresponding to each service data type based on the storage position, the data volume and the version information of the synchronous service data.
In the invention, when the primary node carries out serialization processing on synchronous service data, the service description information (including data volume), data version information, data file path and the like are written into another file at the same time, thereby generating a corresponding index file. In the present invention, each service type corresponds to an index file, and the index file indicates the location of data storage, the size of data volume, version information, and the like, and in the present invention, the contents included in the index file can refer to table 1:
further, when the data volume of a certain service structure is less than 4K of data, the data is directly stored in the memory, and when the data volume is greater than or equal to 4K of data, the data File is stored, the path is in the File field of the index File, and Size represents the data volume. It should be noted that, because the serialized and deserialized data structures need to be strictly consistent, the Version of the data structure is represented by a Version field, and Version is added by 1 each time the data structure changes. Since the data file is generated on the primary node and synchronized to the secondary node, in this application scenario, the structure of the data file is generated, which requires reading the database and then computationally intensive generation. After receiving the synchronization update message, the primary node compares the vid in the message with the last vid in the cache, and when the vids are different, the primary node executes the update process to the primary node to achieve the synchronization process, otherwise, the primary node directly uses the last data file to perform synchronization.
Fig. 2 is a second schematic flowchart of the node memory synchronization method for cluster deployment provided by the present invention, and as shown in fig. 2, the present invention provides a node memory synchronization method for cluster deployment, which includes:
In the present invention, a secondary node is used as an execution subject for explanation, first, after a service of the secondary node is started (for example, a client needs to perform an inquiry through the secondary node), a connection is established with a primary node, and a login request message is sent, and after the secondary node successfully logs in, the secondary node remains in a state of receiving a message. Specifically, after the secondary node is started, the secondary node actively establishes a connection with the primary node through a configured TCP port. After the connection is established, the secondary node sends a Login (Login) message to the primary node, fig. 3 is a schematic diagram of a format of Login information provided by the present invention, and as shown in fig. 3, the format of the Login information sequentially includes: length, 4 bytes, used to identify the total Length of the packet; cmd, 4 bytes, which is the command code of the data packet; the Cid is a string type, the length is identified by two bytes and is used for identifying one connection, and the hostname of the secondary node can be taken, so that the primary node is connected with different secondary nodes according to the identification; MsgNum represents the type of login message, and the size of the login message is an integer of 4 bytes; the size of the msgsversion is an 8 byte integer, and there are a total of MsgsNum versions. Each message represents a service and the primary node needs to perform a version check on the service data structure.
In the invention, when the service end sends the updating request, the service end can be simultaneously sent to the primary node and the secondary node. When the primary node receives data (new service data), the primary node updates the memory data to generate a data file; and after receiving the update request, the secondary node sends a synchronization request message to the primary node, and after the primary node acquires the memory file with the latest version, the primary node sends the memory file to the secondary node according to the synchronization request message, so that the secondary node completes data synchronization. Fig. 4 is a schematic diagram of a format of a synchronization message provided by the present invention, and as shown in fig. 4, the format of the synchronization message sequentially includes: length, Cmd (Length, Cmd are common fields, the same function as in the login message), and SyncMsg. The SyncMSg mainly comprises MsgId, Vid and Type which respectively represent the version of current data, the service Type to be synchronized and the message number, and some self-defined fields can be added according to the requirement.
In the invention, the secondary node acquires the latest data file sent by the primary node through TCP connection and completes synchronization. Fig. 5 is a schematic diagram of a synchronization format of a Data file provided by the present invention, and as shown in fig. 5, a Data format received by a secondary node is Length, MsgId, Type, and Data in sequence, in the Data format, the synchronization Data sent by a primary node does not have a Cmd field, Data represents custom Data, and the Length is controlled by Length.
On the basis of the above embodiment, before the synchronizing the service data to be synchronized according to the serialized memory service data, the method further includes:
and performing deserialization processing on the serialized memory service data to obtain corresponding synchronous service data so as to perform service data synchronization according to the synchronous service data.
In the invention, the synchronous data received by the secondary node is serialized data, and when the secondary node opens a file in which the serialized data is stored, in order to accelerate the reading speed, the secondary node is mapped to the memory through the Mmap to obtain the address P mapped by the file. For convenience of explanation, it is assumed that a structure is stored in the block memory area to which P points. Furthermore, according to a method for traversing the members of the structure body used in serialization, each member of the structure body is traversed in sequence, and a corresponding value is read from the address P according to the type and the size of the member. For the type occupying the indefinite memory size, the memory is read from the memory pointed by P in the above embodiment by writing the memory. For example, a string type data is stored in the address of P + i, and the actual content is read according to the length of two bytes by reading the length first.
Fig. 6 is a schematic structural diagram of a node memory synchronization apparatus for cluster deployment, and as shown in fig. 6, the present invention provides a node memory synchronization apparatus for cluster deployment, which includes a synchronization data acquisition module 601, a serialization processing module 602, and a synchronization data sending module 603, where the synchronization data acquisition module 601 is configured to acquire, after a secondary node logs in a primary node, synchronization service data required by the secondary node according to synchronization information, where the synchronization information includes a service data type of the synchronization service data; the serialization processing module 602 is configured to perform data structure serialization processing on the synchronous service data to obtain serialized memory service data; the synchronization data sending module 603 is configured to send the serialized memory service data to the secondary node, so that the secondary node performs service data synchronization according to the serialized memory service data.
In the invention, after the service of the secondary node is started, the secondary node establishes connection with the primary node and sends a login request to the primary node, and after the login is successful, the secondary node is in a state of receiving a message. Further, after receiving the synchronization information, the primary node determines, by using the synchronization data obtaining module 601, the synchronization service data that needs to be updated by the secondary node according to the service data type included in the synchronization information. In the present invention, after the synchronization data acquisition module 601 confirms the service data that needs to be synchronized with the secondary node, the serialization processing module 602 performs data structure serialization processing on the service data to obtain serialized memory service data. The business data of the invention is memory data, the data structure of the memory can be represented by a structure body, and under the environment of C/C + +/golang programming language, a structure body is defined by using Struct keywords, so that the data in the structure body can be traversed according to the sequence defined in the structure body by carrying out serialization processing on the data structure, and then the data are sequentially stored in a continuous address space. Specifically, the serialization processing module 602 opens an empty file from the hard disk in a binary manner, maps to the memory through the Mmap, serializes the data used for the synchronization of the secondary node in the memory, writes the serialized data into the empty file to form a data file, and finally, the synchronized data sending module 603 sends the data file to the secondary node through network connection, so that the secondary node performs synchronization according to the data file.
The node memory synchronization device for cluster deployment provided by the invention initiates a synchronization request to a primary node by using a secondary node according to the service type, and then carries out snapshot on a service data structure through the primary node, thereby synchronizing a data file to the secondary node through a network. The scheme can be used for various scenes, improves the reading and writing efficiency, and can complete the file transmission function only by using simple protocol constraint.
Fig. 7 is a second schematic structural diagram of the node memory synchronization device for cluster deployment according to the present invention, and as shown in fig. 7, the present invention provides a node memory synchronization device for cluster deployment, which includes a login information construction module 701, a login connection module 702, a synchronization information construction module 703 and a memory data synchronization module 704, where the login information construction module 701 is configured to construct login information according to a service data type of service data in a memory; the login connection module 702 is configured to send the login information to the primary node to establish a login connection with the primary node; the synchronization information construction module 703 is configured to construct synchronization information according to a service data type of service data to be synchronized in a memory, and send the synchronization information to the primary node; the memory data synchronization module 704 is configured to obtain serialized memory service data sent by the primary node, and synchronize service data to be synchronized according to the serialized memory service data, where the serialized memory service data is obtained by performing data structure serialization processing on synchronous service data required by a secondary node by the primary node.
In the present invention, after the service of the secondary node is started (for example, the client needs to query through the secondary node), a connection needs to be established with the primary node, a login request message is constructed through the login information construction module 701, and the secondary node is kept in a state of receiving a message when the login process of the secondary node is completed through the login connection module 702. Specifically, after the secondary node is started, the secondary node actively establishes a connection with the primary node through a configured TCP port. When the connection is established, the Login connection module 702 sends a Login message to the primary node. Further, when the service end sends the update request, the update request is sent to the primary node and the secondary node simultaneously. When the primary node receives data (new service data), the primary node updates the memory data to generate a data file; after the secondary node receives the update request, the synchronization information construction module 703 sends a synchronization request message to the primary node, and after the primary node obtains the memory file of the latest version, the memory data synchronization module 704 sends the memory file to the secondary node according to the synchronization request message, so that the secondary node completes data synchronization. In the invention, the secondary node acquires the latest data file sent by the primary node through TCP connection and completes synchronization.
The node memory synchronization device for cluster deployment provided by the invention initiates a synchronization request to a primary node by using a secondary node according to the service type, and then carries out snapshot on a service data structure through the primary node, thereby synchronizing a data file to the secondary node through a network. The scheme can be used for various scenes, improves the reading and writing efficiency, and can complete the file transmission function only by using simple protocol constraint.
The apparatus provided by the present invention is used for executing the above method embodiments, and for details and flow, reference is made to the above embodiments, which are not described herein again.
Fig. 8 is a schematic structural diagram of an electronic device provided in the present invention, and as shown in fig. 8, the electronic device may include: a processor (processor) 801, a communication interface (communication interface) 802, a memory (memory) 803 and a communication bus 804, wherein the processor 801, the communication interface 802 and the memory 803 complete communication with each other through the communication bus 804. The processor 801 may call logic instructions in the storage 803 to execute a method for node memory synchronization for cluster deployment, the method comprising: after a secondary node logs in a primary node, acquiring synchronous service data required by the secondary node according to synchronous information, wherein the synchronous information comprises a service data type of the synchronous service data; carrying out data structure serialization processing on the synchronous service data to obtain serialized memory service data; sending the serialized memory service data to the secondary node, so that the secondary node can carry out service data synchronization according to the serialized memory service data;
or, according to the service data type of the service data in the memory, building login information; sending the login information to a primary node to establish a login connection with the primary node; according to the service data type of service data to be synchronized in a memory, constructing synchronization information, and sending the synchronization information to the primary node; obtaining serialized memory service data sent by the primary node, and synchronizing service data to be synchronized according to the serialized memory service data, wherein the serialized memory service data is obtained by the primary node performing data structure serialization on synchronous service data required by a secondary node.
In addition, the logic instructions in the memory 803 may be implemented in the form of software functional units and stored in a computer readable storage medium when the logic instructions are sold or used as independent products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
In another aspect, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, where the computer program includes program instructions, and when the program instructions are executed by a computer, the computer can execute the node memory synchronization method for cluster deployment provided by the above methods, where the method includes: after a secondary node logs in a primary node, acquiring synchronous service data required by the secondary node according to synchronous information, wherein the synchronous information comprises a service data type of the synchronous service data; carrying out data structure serialization processing on the synchronous service data to obtain serialized memory service data; sending the serialized memory service data to the secondary node, so that the secondary node can carry out service data synchronization according to the serialized memory service data;
or, according to the service data type of the service data in the memory, building login information; sending the login information to a primary node to establish a login connection with the primary node; according to the service data type of service data to be synchronized in a memory, constructing synchronization information, and sending the synchronization information to the primary node; obtaining serialized memory service data sent by the primary node, and synchronizing service data to be synchronized according to the serialized memory service data, wherein the serialized memory service data is obtained by the primary node performing data structure serialization on synchronous service data required by a secondary node.
In yet another aspect, the present invention further provides a non-transitory computer-readable storage medium, on which a computer program is stored, where the computer program is implemented by a processor to execute the node memory synchronization method for cluster deployment provided in the foregoing embodiments, and the method includes: after a secondary node logs in a primary node, acquiring synchronous service data required by the secondary node according to synchronous information, wherein the synchronous information comprises a service data type of the synchronous service data; carrying out data structure serialization processing on the synchronous service data to obtain serialized memory service data; sending the serialized memory service data to the secondary node, so that the secondary node can carry out service data synchronization according to the serialized memory service data;
or, according to the service data type of the service data in the memory, building login information; sending the login information to a primary node to establish a login connection with the primary node; according to the service data type of service data to be synchronized in a memory, constructing synchronization information, and sending the synchronization information to the primary node; obtaining serialized memory service data sent by the primary node, and synchronizing service data to be synchronized according to the serialized memory service data, wherein the serialized memory service data is obtained by the primary node performing data structure serialization on synchronous service data required by a secondary node.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A node memory synchronization method for cluster deployment is characterized by comprising the following steps:
after a secondary node logs in a primary node, acquiring synchronous service data required by the secondary node according to synchronous information, wherein the synchronous information comprises a service data type of the synchronous service data;
carrying out data structure serialization processing on the synchronous service data to obtain serialized memory service data;
and sending the serialized memory service data to the secondary node so that the secondary node can carry out service data synchronization according to the serialized memory service data.
2. The method according to claim 1, wherein before the obtaining of the synchronous service data required by the secondary node according to the synchronization information, the method further comprises:
and obtaining login information sent by the secondary node so that the secondary node logs in the primary node, wherein the login information comprises the service data type of each service data in the secondary node.
3. The method according to claim 1, wherein the performing data structure serialization processing on the synchronous service data to obtain serialized memory service data comprises:
judging the data volume of the synchronous service data, and if the data volume is greater than or equal to 4K data, performing data structure serialization processing on the synchronous service data according to the service data type corresponding to the synchronous service data;
and if the data volume is less than 4K data, directly sending the synchronous service data to a secondary node so that the secondary node can carry out service data synchronization according to the synchronous service data.
4. The method according to claim 3, wherein after performing data structure serialization processing on the synchronous service data according to the service data type corresponding to the synchronous service data, the method further comprises:
and constructing an index file corresponding to each service data type based on the storage position, the data volume and the version information of the synchronous service data.
5. A node memory synchronization method for cluster deployment is characterized by comprising the following steps:
according to the service data type of the service data in the memory, building login information;
sending the login information to a primary node to establish a login connection with the primary node;
according to the service data type of service data to be synchronized in a memory, constructing synchronization information, and sending the synchronization information to the primary node;
obtaining serialized memory service data sent by the primary node, and synchronizing service data to be synchronized according to the serialized memory service data, wherein the serialized memory service data is obtained by the primary node performing data structure serialization on synchronous service data required by a secondary node.
6. The method according to claim 5, wherein before the synchronizing the service data to be synchronized according to the serialized memory service data, the method further comprises:
and performing deserialization processing on the serialized memory service data to obtain corresponding synchronous service data so as to perform service data synchronization according to the synchronous service data.
7. A node memory synchronization device for cluster deployment, comprising:
the synchronous data acquisition module is used for acquiring synchronous service data required by a secondary node according to synchronous information after the secondary node logs in a primary node, wherein the synchronous information comprises a service data type of the synchronous service data;
the serialization processing module is used for carrying out data structure serialization processing on the synchronous service data to obtain serialized memory service data;
and the synchronous data sending module is used for sending the serialized memory service data to the secondary node so that the secondary node can carry out service data synchronization according to the serialized memory service data.
8. A node memory synchronization device for cluster deployment, comprising:
the login information construction module is used for constructing login information according to the service data type of the service data in the memory;
the login connection module is used for sending the login information to a primary node so as to establish login connection with the primary node;
the synchronous information construction module is used for constructing synchronous information according to the service data type of service data to be synchronized in the memory and sending the synchronous information to the primary node;
and the memory data synchronization module is used for acquiring the serialized memory service data sent by the primary node and synchronizing the service data to be synchronized according to the serialized memory service data, wherein the serialized memory service data is obtained by performing data structure serialization processing on the synchronous service data required by the secondary node by the primary node.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the method for node memory synchronization for cluster deployment according to any one of claims 1 to 6 when executing the computer program.
10. A non-transitory computer readable storage medium, on which a computer program is stored, wherein the computer program, when being executed by a processor, implements the steps of the method for synchronizing the memory of nodes for cluster deployment according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110457235.3A CN112988905B (en) | 2021-04-27 | 2021-04-27 | Node memory synchronization method and device for cluster deployment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110457235.3A CN112988905B (en) | 2021-04-27 | 2021-04-27 | Node memory synchronization method and device for cluster deployment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112988905A true CN112988905A (en) | 2021-06-18 |
| CN112988905B CN112988905B (en) | 2021-08-10 |
Family
ID=76340335
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110457235.3A Active CN112988905B (en) | 2021-04-27 | 2021-04-27 | Node memory synchronization method and device for cluster deployment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112988905B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017088664A1 (en) * | 2015-11-26 | 2017-06-01 | 深圳市中博科创信息技术有限公司 | Data processing method and apparatus for cluster file system |
| CN107465709A (en) * | 2016-06-03 | 2017-12-12 | 阿里巴巴集团控股有限公司 | Distributed mirror image structure multitask method and device, system |
| CN111240737A (en) * | 2020-01-20 | 2020-06-05 | 杭州海兴电力科技股份有限公司 | Dynamic service parameter configuration method based on Redis |
-
2021
- 2021-04-27 CN CN202110457235.3A patent/CN112988905B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017088664A1 (en) * | 2015-11-26 | 2017-06-01 | 深圳市中博科创信息技术有限公司 | Data processing method and apparatus for cluster file system |
| CN107465709A (en) * | 2016-06-03 | 2017-12-12 | 阿里巴巴集团控股有限公司 | Distributed mirror image structure multitask method and device, system |
| CN111240737A (en) * | 2020-01-20 | 2020-06-05 | 杭州海兴电力科技股份有限公司 | Dynamic service parameter configuration method based on Redis |
Non-Patent Citations (2)
| Title |
|---|
| WEIXIN_39477597: "Redis主从复制", 《HTTPS://BLOG.CSDN.NET/WEIXIN_39477597/ARTICLE/DETAILS/114434831》 * |
| 幕布斯6054654: "Redis数据同步机制", 《HTTPS://WWW.IMOOC.COM/ARTICLE/264423》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112988905B (en) | 2021-08-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106874389B (en) | Data migration method and device | |
| CN108628942B (en) | Block chain node equipment and data reading and writing method of distributed database | |
| CN106815218B (en) | Database access method and device and database system | |
| CN111124755B (en) | Fault recovery method and device for cluster nodes, electronic equipment and storage medium | |
| WO2018045758A1 (en) | Data storage method and data centre | |
| CN109710185B (en) | Data processing method and device | |
| CN110209642A (en) | Method, apparatus, server and the computer-readable medium of information processing | |
| CN1988535A (en) | Synchronous method, system for file storage and customer terminal | |
| CN103268351A (en) | Method and device for data synchronism | |
| CN113360456B (en) | Data archiving method, device, equipment and storage medium | |
| CN113010549A (en) | Data processing method based on remote multi-active system, related equipment and storage medium | |
| CN112988905B (en) | Node memory synchronization method and device for cluster deployment | |
| US20210342334A1 (en) | Method, device, and computer program product for data management | |
| CN112000850A (en) | Method, device, system and equipment for data processing | |
| CN109165259B (en) | Index table updating method based on network attached storage, processor and storage device | |
| CN115104295A (en) | Data processing method, data processing device, electronic device and storage medium | |
| CN116303789A (en) | Parallel synchronization method and device for multi-fragment multi-copy database and readable medium | |
| CN108900347B (en) | Cluster initialization configuration method, device, system, equipment and readable storage medium | |
| CN111966650B (en) | Operation and maintenance big data sharing data table processing method and device and storage medium | |
| US20210374072A1 (en) | Augmenting storage functionality using emulation of storage characteristics | |
| CN113672591A (en) | Data migration method, system, storage medium and electronic device | |
| US20230353393A1 (en) | Blockchain-based methods and apparatuses for processing data, devices and readable storage mediums | |
| US11347406B2 (en) | Method, electronic device and computer program product for updating information | |
| CN114978950B (en) | Network algorithm calling method and system based on FPGA and CPU cooperation | |
| CN114205333B (en) | IP configuration method, cluster construction method, computer device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |