CN114866570A - Information processing method and device, electronic equipment and storage medium - Google Patents
Information processing method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN114866570A CN114866570A CN202210405323.3A CN202210405323A CN114866570A CN 114866570 A CN114866570 A CN 114866570A CN 202210405323 A CN202210405323 A CN 202210405323A CN 114866570 A CN114866570 A CN 114866570A
- Authority
- CN
- China
- Prior art keywords
- data storage
- node
- cluster
- data
- service
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000010365 information processing Effects 0.000 title claims abstract description 20
- 238000003672 processing method Methods 0.000 title claims abstract description 19
- 238000013500 data storage Methods 0.000 claims abstract description 214
- 238000000034 method Methods 0.000 claims abstract description 27
- 230000008569 process Effects 0.000 claims description 9
- 230000003993 interaction Effects 0.000 claims description 6
- 230000002159 abnormal effect Effects 0.000 description 12
- 238000012545 processing Methods 0.000 description 12
- 238000010586 diagram Methods 0.000 description 8
- 239000012634 fragment Substances 0.000 description 7
- 238000005192 partition Methods 0.000 description 6
- 230000008859 change Effects 0.000 description 4
- 230000004044 response Effects 0.000 description 4
- 238000007726 management method Methods 0.000 description 3
- 238000011161 development Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 239000000969 carrier Substances 0.000 description 1
- 230000008602 contraction Effects 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0803—Configuration setting
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The disclosure relates to an information processing method, an information processing device, an electronic device and a storage medium, which are applied to a server for controlling a cluster, wherein the method comprises the following steps: establishing connection with each node in the cluster; receiving configuration parameters; the configuration parameters comprise a cluster identifier of the cluster, a service identifier of a current service of the cluster, the number of data storage files and the number of backup copies of the data storage files; each data storage file is used for storing part of service data of the current service; the service data stored in different data storage files are different; generating the creation information of the data storage file and the backup copy according to the configuration parameters and the number of the nodes; the creation information is used for indicating the nodes where the data storage files and the backup copies are located; sending the creation information to the node; and the node is used for creating the data storage file and the backup copy according to the creation information.
Description
Technical Field
The present disclosure relates to the field of information processing, and in particular, to an information processing method and apparatus, an electronic device, and a storage medium.
Background
With the development of technology, many emerging application scenarios appear, and business processing is required to be performed in both the existing application scenarios and the emerging application scenarios, so as to realize the functions of the applications. As the application scenarios of service processing become wider and wider, the service processing modes in different application scenarios may be different, and the service processing modes of different services may also be different.
Disclosure of Invention
The disclosure provides an information processing method, an information processing apparatus, an electronic device, and a storage medium.
In a first aspect of the embodiments of the present disclosure, an information processing method is provided, which is applied to a server controlling a cluster, and the method includes: establishing connection with each node in the cluster; receiving configuration parameters; the configuration parameters comprise a cluster identifier of the cluster, a service identifier of a current service of the cluster, the number of data storage files and the number of backup copies of the data storage files; each data storage file is used for storing part of service data of the current service; the service data stored in different data storage files are different; generating the creation information of the data storage file and the backup copy according to the configuration parameters and the number of the nodes; the creation information is used for indicating nodes where the data storage files and the backup copies are located; sending the creation information to the node; and the node is used for creating the data storage file and the backup copy according to the creation information.
In some embodiments, the number of data storage files and the number of backup copies included in each of the nodes are the same.
In some embodiments, the business data stored in the data storage file included in the same node is different from the business data stored in the backup copy.
In some embodiments, the data storage file and the backup copy storing the service data of the same current service are located in the node in the same cluster.
In some embodiments, the method further comprises: saving a first location of each of the data storage files and a second location of each of the backup copies; after sending the change at the first position and the second position, sending an updating instruction of the first position and the second position to each node; wherein the update instructions are configured to control the node to update the data storage file and the backup copy in the node; the updating instruction comprises the updated first position and the updated second position.
In some embodiments, the configuration parameters include: a first configuration parameter for the number of data storage files and a second configuration parameter for the number of backup copies; the method further comprises the following steps: detecting a first configuration operation according to the number of the data storage files, and determining a first configuration parameter according to the first configuration operation; and detecting a second configuration operation according to the quantity of the configured backup files, and determining a second configuration parameter according to the second configuration operation.
In some embodiments, the first configuration operation is determined according to the total amount of data of the current service and the storage space of the data storage file for storing service data.
In some embodiments, the storage space for storing the service data is the same in size for different data storage files.
In some embodiments, the establishing connections with the nodes in the cluster includes: receiving an interface calling request sent by each node; responding to the interface calling request, and allowing the node to carry out interface calling; receiving a registration request sent by the node; and responding to the registration request, and allowing information interaction with the node.
In a second aspect of the embodiments of the present disclosure, an information processing apparatus is provided, which is applied to a server controlling a cluster, and the apparatus includes: the connection establishing module is used for establishing connection with each node in the cluster; the configuration parameter receiving module is used for receiving configuration parameters; the configuration parameters comprise a cluster identifier of the cluster, a service identifier of a current service of the cluster, the number of data storage files and the number of backup copies of the data storage files; each data storage file is used for storing part of service data of the current service; the service data stored in different data storage files are different; the creation information generation module is used for generating creation information of the data storage file and the backup copy according to the configuration parameters and the number of the nodes; the creation information is used for indicating the nodes where the data storage files and the backup copies are located; a sending module, configured to send the creation information to the node; and the node is used for creating the data storage file and the backup copy according to the creation information.
In a third aspect of the embodiments of the present disclosure, there is provided an electronic device, including: a processor and a memory for storing executable instructions operable on the processor, wherein: when the processor is used for executing the executable instructions, the executable instructions execute the method of any one of the above embodiments.
In a fourth aspect of the embodiments of the present disclosure, a non-transitory computer-readable storage medium is provided, in which computer-executable instructions are stored, and when executed by a processor, the computer-executable instructions implement the method according to any of the embodiments described above.
The technical scheme provided by the embodiment of the disclosure can have the following beneficial effects:
the embodiment of the disclosure is applied to a server for controlling a cluster, and receives configuration parameters after establishing connection with each node in the cluster; the configuration parameters comprise cluster identification of the cluster, service identification of current service of the cluster, the number of data storage files and the number of backup copies of the data storage files; each data storage file is used for storing part of service data of the current service; different data storage files store different service data. And then generating creating information of the data storage file and the backup copy according to the configuration parameters and the number of the nodes, wherein the creating information is used for indicating the nodes where the data storage file and the backup copy are located. Then sending the creation information to the node; the node is used for creating a data storage file and a backup copy according to the creation information.
According to the method, the creation information corresponding to each node in the cluster can be automatically generated according to the configuration parameters matched with the cluster and the current service of the cluster and by combining the number of the nodes in the cluster, the nodes can be used for creating the data storage file and the backup copy after receiving the creation information, so that the creation of the data storage file and the backup copy is realized, the data storage file is ensured to have the corresponding backup copy, when the data stored in the data storage file is lost or damaged, the lost or damaged data in the corresponding data storage file can be obtained through the backup copy, and the safety of the data is improved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.
FIG. 1 is a schematic flow diagram illustrating an information processing method in accordance with an exemplary embodiment;
FIG. 2 is a schematic diagram illustrating another information processing method in accordance with an illustrative embodiment;
FIG. 3 is a schematic diagram illustrating an establishment of a connection in accordance with an exemplary embodiment;
FIG. 4 is a schematic diagram of an information processing apparatus shown in accordance with an exemplary embodiment;
FIG. 5 is a schematic diagram illustrating another information process according to an example embodiment.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. The following description refers to the accompanying drawings in which the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of devices consistent with certain aspects of the present disclosure, as detailed in the appended claims.
Referring to fig. 1, a schematic flow chart of an information processing method provided in an embodiment of the present disclosure is shown, where the information processing method is at least applicable to a server controlling a cluster, the cluster includes a plurality of nodes, each node may be a terminal, and the server may control each node in the cluster, so as to control the whole cluster. The method comprises the following steps:
step S100, connection with each node in the cluster is established.
Step S200, receiving configuration parameters; the configuration parameters comprise a cluster identifier of the cluster, a service identifier of a current service of the cluster, the number of data storage files and the number of backup copies of the data storage files; each data storage file is used for storing part of service data of the current service; the service data stored in different data storage files are different.
Step S300, generating the creation information of the data storage file and the backup copy according to the configuration parameters and the number of the nodes; wherein the creation information is used to indicate a node where the data storage file and the backup copy are located.
Step S400, sending the creation information to the node; and the node is used for creating the data storage file and the backup copy according to the creation information.
For step S100, each node in the cluster may establish a connection with the server, each node may send a connection request to the server, the server processes the connection request after receiving the connection request, and in response to the connection request, a connection with each node may be established according to the connection request sent by each node. The form of the connection request sent by the node to the server is not limited, and may include, for example, a call request for calling a data interface of the server. The node may be capable of establishing a connection with the server upon a request sent to the server.
After the server establishes connection with each node, information interaction can be carried out with each node.
The server may be connected to multiple clusters, i.e. establish connections to individual nodes in multiple clusters, which may differ from one cluster to another.
In one embodiment, the connection request includes a cluster identifier of a cluster to which the node belongs, the cluster identifier carried in the connection request is used for determining the cluster to which the node belongs, and the number of nodes establishing connection with the server in each cluster can be determined according to the connection request and the cluster identifier.
For step S200, the server may further receive configuration parameters, where the configuration parameters at least include a cluster identifier of a cluster to which the node establishing the connection belongs, a service identifier of a current service of the cluster, the number of data storage files, and the number of backup copies of each data storage file.
The current traffic may be different for each cluster, i.e. the current traffic may be different for the various nodes comprised in each cluster for the different clusters. The number of nodes included in each cluster may be determined based on actual traffic demands. Each cluster has a respective cluster identifier, different clusters have different cluster identifiers, and the cluster identifiers are used for identifying the corresponding clusters, and the clusters can be distinguished according to the cluster identifiers.
The current service of each cluster may also be different, the current service of each cluster also has a corresponding service identifier, and when the current service is different, the corresponding service identifiers are also different. The service processed by each node in the same cluster is the same, and the current service of the cluster is the current service of each node. The cluster mark and the service mark have a corresponding relation, the service mark corresponding to the current service can be determined according to the cluster mark, and the corresponding cluster mark can also be determined according to the service mark.
The data storage file is a file for storing data of the current service of the cluster, and the file may be in a folder form, or may be in a file form capable of storing data, such as a compressed package form. Each data storage file has a corresponding backup copy, and for a data storage file, there may be at least one backup copy.
The configuration parameters may also include the total data amount of the current service. The number of the data storage files and the number of the backup copies of the data storage files can be determined according to business requirements, and can also be determined according to the size of the total data amount of the current business. Each data storage file may be used to store part of the service data of the current service, and different data storage files store different service data. The sum of the service data stored in each data storage file is the sum of the number of the current services, that is, each data storage file stores a part of data, and all the data storage files can store all the data of the current services.
For example, the total amount of data of the current service is a, the number of data storage files is 10, and the data storage files 1, 2, 3, 4, 5, 6, 7, 8, 9, and 10 are stored. Data storage file 1 stores data a1, data storage file 2 stores data a2, data storage file 3 stores data A3, data storage file 4 stores data a4, data storage file 5 stores data a5, data storage file 6 stores data a6, data storage file 7 stores data a7, data storage file 8 stores data A8, data storage file 9 stores data a9, and data storage file 10 stores data a 10. The total data amount of the current service is A and is divided into 10 parts, and each data storage file stores a part of data.
Therefore, the situation that the total data amount of the current service is stored in one data storage file can be reduced, and the total data amount of the current service is abnormal when the data storage file is abnormal or the data in the storage file is lost or damaged. If one or more of the data storage files are abnormal or the storage processing is abandoned or damaged, the data in other data storage files are not affected, so that the loss degree of the data is reduced, and the safety of the data is improved.
The data stored in each data storage file is the same as the data stored in the backup copy of the data storage file, so that when the data storage file is abnormal or the data in the data storage file is abnormal, the data in the data storage file can be recovered through the backup copy, and the safety of the data is improved.
The configuration parameters may be obtained according to configuration operations, and the server may receive configuration operations configuring the cluster identifier, the service identifier, the total data amount of the current service, the number of the data storage files, and the number of backup copies of each data storage file, and determine the configuration parameters according to the configuration operations.
In another embodiment, the server may further determine the configuration parameter corresponding to each cluster according to the current service of each cluster. After the nodes included in each cluster are connected with the server, the server can determine the cluster identifier of each cluster, the service identifier of the current service of each cluster and the total data amount of the current service, and predict the number of data storage files and the number of backup copies according to the total number of the current service. The prediction can be performed according to a prediction relation table, wherein the prediction relation table comprises mapping relations among the cluster identifier, the service identifier of the current service, the total data amount of the current service, the number of the data storage files and the number of the backup copies.
For step S300, after the server establishes connection with the nodes in the cluster, the number of the nodes that establish connection may be determined. The server can obtain the cluster identification of the cluster to which the node belongs when establishing connection with the node, so that the node which establishes connection with the server in each cluster can be determined according to the cluster identification of the cluster to which each node belongs, and the number of the nodes which establish connection with the server in each cluster can be determined at the same time. The number of nodes establishing a connection with the server may also be determined by the connection request.
After the configuration parameters and the number of the nodes establishing connection with the server are obtained, the creation information of the data storage file and the backup copy can be generated according to the configuration parameters and the number of the nodes. The creation information is used to indicate the node where the data storage file and the backup copy are located.
The configuration parameters comprise cluster identifiers, service identifiers, the number of data storage files and the number of backup copies, so that a cluster for processing the current service can be determined according to the cluster identifiers, and creation information is generated according to the number of nodes connected with the server in the cluster, the number of the data storage files and the number of the backup copies and used for indicating the nodes where the data storage files and the backup copies are located.
For example, the number of nodes establishing connection with the server in the cluster a is 20, and the 20 nodes process the current service a. The number of data storage files is 10, the number of backup copies of each data storage file is 1, and 10 backup copies are provided in total. The creation information may indicate the node where each data storage file and backup copy is located, generated based on the number of nodes and the number of data storage files. For example, the creation information may indicate that data storage file 1 is located at node 1, data storage file 2 is located at node 2, data storage file 3 is node 4, data storage file 4 is located at node 6, data storage file 5 is located at node 9, data storage file 6 is located at node 10, data storage file 7 is located at node 15, data storage file 8 is located at node 12, data storage file 9 is located at node 20, and data storage file 10 is located at node 18. Likewise, so does the backup copy.
The node where the data storage file and the backup copy indicated in the creation information are located may be a target node, and the target node may be all nodes that establish connection with the server in the cluster that processes the current service, or may be a part of nodes.
And step S400, sending the creation information to each node, wherein the nodes are used for creating data storage files and backup copies according to the creation information. After receiving the creation information, each node determines whether the node belongs to the node indicating the location of the data storage file and the backup copy in the creation information, and creates the data storage file and/or the backup copy when determining that the node belongs to the node indicating the location of the data storage file and the backup copy in the creation information.
And when determining that the node does not belong to the node indicating the data storage file and the backup copy in the creation information, not creating the data storage file and/or the backup copy.
In another embodiment, the server may further send creation information to the target node according to the creation information, where the target node is the node where the data storage file and the backup copy indicated in the creation information are located.
In another embodiment, when the number of nodes where the data storage files and the backup copies indicated in the creation information are located is less than the number of data storage files, a plurality of different data storage files are created in the same node indicated in the creation information. I.e. a plurality of different data storage files are created in the same target node.
Or when the number of the nodes where the data storage files and the backup copies indicated in the creation information are located is smaller than the number of the backup copies, creating a plurality of different backup copies in the same node indicated in the creation information. I.e. multiple different backup copies are created in the same target node.
In another embodiment, the number of data storage files and the number of backup copies included in each node indicated in the creation information are the same. Thus, the storage amount of each node indicated in each creation information and the load in the creation process can be balanced.
In another embodiment, the business data stored in the data storage file included in the same node is different from the business data stored in the backup copy.
For any node indicated in the creation information, the business data stored in the data storage file created in the node is different from the business data stored in the backup copy created in the node. That is, the data storage file in the same node does not correspond to the backup copy, and the backup copy of the data storage file in the node is no longer located in the node but in another node. A data storage file and a backup copy of the data storage file are not located in the same node. Therefore, when the node is abnormal, the abnormal condition of the node comprises that the data storage file and the backup copy in the node are abnormal at the same time, and if the data stored in the data storage file and the data stored in the backup copy are the same, all the data are abnormal.
Referring to FIG. 2, data storage file 1 is located at node 1 and backup copy 1 of data storage file 1 is located at node 3. Data storage file 2 is located at node 3 and backup copy 2 of data storage file 2 is located at node 2. A data storage file 3 is located at node 2 and a backup copy 3 of the data storage file 3 is located at node 1.
When one node is abnormal, the other nodes also comprise backup copies of the data storage files in the abnormal node or the data storage files corresponding to the backup copies in the abnormal node, so that the safety of the data is improved.
In another embodiment, the data storage file and the backup copy storing the service data of the same current service are located in nodes in the same cluster.
And for the service data of the same current service, storing the service data on the node for processing the current service, wherein the nodes indicated by the creation information are all the nodes in the cluster for processing the current service. For example, the cluster 1 processes the current service 1, and the cluster 1 includes the node 1, the node 2 to the node 10, so that the service data of the current service 1 is stored on the nodes in the cluster 1. The nodes where the data storage files and backup copies storing the service data of the current service 1 indicated in the creation information are located are all nodes in the cluster 1.
Therefore, the data can be conveniently stored and matched, the node processing the current service stores the data of the service, the condition of confusion of data storage can be reduced, and the service processing is convenient.
In another embodiment, referring to fig. 3, a schematic diagram of another information processing method is shown, where the method further includes:
step S10, saving the first location of each data storage file and the second location of each backup copy;
step S20, after the first position and the second position are changed, sending updating instructions of the first position and the second position to each node;
the updating instruction is used for controlling the node to update the data storage file and the backup copy in the node; the update instruction includes the updated first location and the updated second location.
After the creation information is sent to the node, the node creates the data storage files and the backup copies according to the creation information, then saves the positions of the data storage files and the backup copies, records the positions of the data storage files as first positions, and records the positions of the backup copies as second positions. And after the first position and the second position are changed, sending updating instructions of the first position and the second position to each node. The update instructions are used to control the nodes to update the data storage files and backup copies. The update instruction comprises an updated first position and an updated second position, after the update instruction is sent to each node, each node receives the update instruction, and each node can create a data storage file and a backup copy according to the updated first position and the updated second position which are included in the update instruction.
For example, for a data storage file 1, a first location saved by a server is a node 1, and after the first location is changed, the changed first location is a node 2, the server sends an update instruction to each node establishing a connection with the server in a cluster where the node 1 is located, where the update instruction includes the updated first location. After receiving the update instruction, the node creates a data storage file 1 according to the updated first position in the update instruction, the update instruction controls the node 2 to create the data storage file 1, and after the node 2 creates the data storage file 1, the node 1 is controlled to delete the data storage file 1. When the first position and/or the second position are updated, the situation that the data storage file and/or the backup copy are deleted before the updating is not finished can be reduced, and the data safety is improved. Meanwhile, the accuracy of the first position and the second position can be improved, and the data in the data storage file and the data in the backup copy can be conveniently read by the node.
In one embodiment, the sending of the change of the first position and the second position may be driven by a monitor in the node, the monitor in the node monitors the first position and the second position in the node, and after the change of the first position and the second position, the changed first position and/or second position is sent to a server, and the server stores the changed first position and second position.
In another embodiment, the configuration parameters include: a first configuration parameter for the number of data storage files.
The method further comprises the following steps:
a first configuration operation according to the number of configuration data storage files is detected, and a first configuration parameter is determined according to the first configuration operation. The first configuration parameter may be determined according to the first configuration operation, and the first configuration parameter may be determined after the first configuration operation is detected.
In another embodiment, the configuration parameters include: a second configuration parameter for the number of backup copies.
The method further comprises the following steps:
and detecting a second configuration operation according to the quantity of the configuration backup files, and determining a second configuration parameter according to the second configuration operation. The second configuration parameter may be determined according to the second configuration operation, and the second configuration parameter may be determined after the second configuration operation is detected.
In another embodiment, other ones of the configuration parameters may also be determined by corresponding configuration operations.
In another embodiment, the first configuration operation is determined according to the total amount of data of the current service and the storage space of the data storage file for storing the service data.
The storage space of the data storage file for storing the service data and the total data amount of the current service determine the number of the data storage files. When the total amount of data is fixed, the larger the storage space of the data storage file for storing the business data is, the smaller the number of the data storage files is, and the smaller the storage space of the data storage file for storing the business data is, the larger the number of the data storage files is.
The first configuration operation for configuring the number of the data storage files can be determined according to the total data amount of the current service and the number of the data storage files, and a first configuration parameter can be determined according to the first configuration operation, so that the number of the data storage files can be determined.
In another embodiment, the storage spaces for storing the service data in different data storage files are the same in size, so that the data amount of the service data stored in each node can be balanced, and the balance of the storage spaces of each node is improved.
In another embodiment, referring to fig. 4, which is a schematic diagram illustrating establishment of connections, step S100, establishing connections with each node in a cluster includes:
step S101, receiving an interface calling request sent by each node;
step S102, responding to the interface calling request, allowing the node to carry out interface calling;
step S103, receiving a registration request sent by a node;
and step S104, responding to the registration request, and allowing information interaction with the node.
When the connection with each node is established, each node sends an interface calling request to the server, and after the server receives the interface calling request, the server responds to the interface calling request and then allows the node to carry out interface calling. After allowing the node to carry out interface call, the node sends a registration request to the server, and after receiving the registration request, the server responds to the registration request and allows the node to carry out information interaction with the server.
In another embodiment, referring to fig. 5, a schematic diagram of an information processing apparatus applied to a server controlling a cluster includes:
the connection establishing module 1 is used for establishing connection with each node in the cluster;
a configuration parameter receiving module 2, configured to receive configuration parameters; the configuration parameters comprise a cluster identifier of the cluster, a service identifier of a current service of the cluster, the number of data storage files and the number of backup copies of the data storage files; each data storage file is used for storing part of service data of the current service; the service data stored in different data storage files are different;
a creation information generation module 3, configured to generate creation information of the data storage file and the backup copy according to the configuration parameters and the number of nodes; the creation information is used for indicating the nodes where the data storage files and the backup copies are located;
a sending module 4, configured to send the creation information to the node; and the node is used for creating the data storage file and the backup copy according to the creation information.
In another embodiment, the number of the data storage files and the number of the backup copies included in each of the nodes are the same.
In another embodiment, the business data stored in the data storage file included in the same node is different from the business data stored in the backup copy.
In another embodiment, the data storage file and the backup copy storing the same service data of the current service are located in the node in the same cluster.
In another embodiment, the apparatus further comprises:
the storage module is used for storing a first position of each data storage file and a second position of each backup copy;
an update instruction sending module, configured to send, to each node, an update instruction of the first location and the second location after sending a change between the first location and the second location;
wherein the update instructions are configured to control the node to update the data storage file and the backup copy in the node; the updating instruction comprises the updated first position and the updated second position.
In another embodiment, the configuration parameters include: a first configuration parameter for the number of data storage files and a second configuration parameter for the number of backup copies;
the device further comprises:
the first processing module is used for detecting a first configuration operation according to the quantity of the data storage files, and determining a first configuration parameter according to the first configuration operation;
and the second processing module is used for detecting second configuration operation according to the quantity of the configured backup files and determining second configuration parameters according to the second configuration operation.
In another embodiment, the first configuration operation is determined according to the total amount of data of the current service and the storage space of the data storage file for storing service data.
In another embodiment, the storage space for storing the service data is the same in size for different data storage files.
In another embodiment, the connection establishing module 1 comprises:
an interface call request receiving unit, configured to receive an interface call request sent by each node;
a first response unit, configured to allow the node to perform interface call in response to the interface call request;
a registration request receiving unit, configured to receive a registration request sent by the node;
and the second response unit is used for responding to the registration request and allowing information interaction with the node.
In another embodiment, there is also provided an electronic device including:
a processor and a memory for storing executable instructions operable on the processor, wherein:
when the processor is used for executing the executable instructions, the executable instructions execute the method of any one of the above embodiments.
In another embodiment, a non-transitory computer-readable storage medium is also provided, having stored therein computer-executable instructions that, when executed by a processor, implement the method of any of the above embodiments.
It should be noted that "first" and "second" in the embodiments of the present disclosure are merely for convenience of description and distinction, and have no other specific meaning.
In another embodiment, an example of an information processing method is also provided. Refer to fig. 2.
Apache Helix is an open source cluster management framework designed for partitioned and replicated distributed resources. As a general cluster resource management framework, it can be used as a distributed resource with replicas that automatically manages partitions residing on cluster nodes.
Helix can automatically reallocate resources after events such as node stop work, restart, cluster capacity expansion and contraction, configuration update and the like occur.
The core idea of the scheme is to create a resource interface through Helix, divide each data storage file segment of the rocksDB cluster into multiple partitions and multiple copies, and provide such routing and read-write support inside the cluster.
The cluster uses different data storage file segments for different services for data isolation. segment is divided and scheduled as the resource of Helix service. The specific process of creating segment is as follows:
the node service needs to implement a callback interface of the Helix server, so that the callback message during segment division and scheduling can be processed.
When the nodes of the cluster are started, the cluster names are required to be registered with the Helix server, and the nodes of the same cluster are used as carriers for dividing segment fragments and copies of the same data storage file by the Helix.
When a service applies for establishing a new data storage file segment, the segment number of the segment is determined by storing about 2GB data in each segment according to the total data amount estimated by the service.
When creating the data storage file segment, information including sharestore cluster name, data storage file segment name, number of fragments, number of copies, and the like needs to be sent to the Helix server. The Helix server divides the fragments and the copies of each node as uniformly as possible according to the node number of the sharestore cluster. And then, the division result is sent to each sharestore node, and after the nodes receive the message, the corresponding data storage file can be created. The number of the fragments is the number of the data storage files, and the number of the copies is the number of the backup copies.
The helli server records the data storage file (including fragments) and the backup copy (such as copy information) after each segmen is divided in a memory, for example, in zokeyer, a monitor (such as zum service) deployed in a node monitors a corresponding zokeyer path, the value is pulled to the local after being changed, and the node in the sharestore cluster analyzes the value, so that the fragment fragments and copy information of the whole cluster can be obtained, and the read-write request routing of the client is realized.
The scheme has the advantages that:
the realization is simple: the existing open source framework is directly utilized, only a small amount of adaptive development is needed, the complex cluster management function can be realized, and the labor and time cost is greatly saved.
The scalability is strong: after the segment is divided into multiple partitions, the cluster can increase and decrease the nodes according to the request quantity and the pressure of the nodes, and then the Helix server can automatically balance the number of the nodes and the number of the partitions, so that the number of the segments of each node is approximately equal.
The data security is high: helix can set each partition of segment as multiple copies, so that the service can ensure the security of data as long as the service realizes the data synchronization among different copies of the same partition.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims (12)
1. An information processing method applied to a server controlling a cluster, the method comprising:
establishing connection with each node in the cluster;
receiving configuration parameters; the configuration parameters comprise a cluster identifier of the cluster, a service identifier of a current service of the cluster, the number of data storage files and the number of backup copies of the data storage files; each data storage file is used for storing part of service data of the current service; the service data stored in different data storage files are different;
generating creation information of the data storage file and the backup copy according to the configuration parameters and the number of the nodes; the creation information is used for indicating the nodes where the data storage files and the backup copies are located;
sending the creation information to the node; and the node is used for creating the data storage file and the backup copy according to the creation information.
2. The process of claim 1, wherein the number of said data storage files and the number of said backup copies included in each of said nodes are the same.
3. The process of claim 1, wherein the business data stored in the data storage file included in the same node is different from the business data stored in the backup copy.
4. The processing method according to claim 1, wherein the data storage file and the backup copy storing the same service data of the current service are located in the node in the same cluster.
5. The processing method according to claim 1, characterized in that the method further comprises:
saving a first location of each of the data storage files and a second location of each of the backup copies;
after the first position and the second position are changed, sending updating instructions of the first position and the second position to each node;
wherein the update instructions are configured to control the node to update the data storage file and the backup copy in the node; the updating instruction comprises the updated first position and the updated second position.
6. The processing method according to claim 1,
the configuration parameters include: a first configuration parameter for the number of data storage files and a second configuration parameter for the number of backup copies;
the method further comprises the following steps:
detecting a first configuration operation according to the number of the configured data storage files, and determining a first configuration parameter according to the first configuration operation;
and detecting a second configuration operation according to the quantity of the configured backup files, and determining a second configuration parameter according to the second configuration operation.
7. The processing method according to claim 6, wherein the first configuration operation is determined according to the total amount of data of the current service and the storage space of the data storage file for storing service data.
8. The processing method according to any one of claims 1, 6 or 7, wherein the storage spaces for storing the service data of different data storage files are the same in size.
9. The processing method according to claim 1, wherein the establishing of the connection with each node in the cluster comprises:
receiving an interface calling request sent by each node;
responding to the interface calling request, and allowing the node to carry out interface calling;
receiving a registration request sent by the node;
and responding to the registration request, and allowing information interaction with the node.
10. An information processing apparatus applied to a server that controls a cluster, the apparatus comprising:
the connection establishing module is used for establishing connection with each node in the cluster;
the configuration parameter receiving module is used for receiving configuration parameters; the configuration parameters comprise a cluster identifier of the cluster, a service identifier of a current service of the cluster, the number of data storage files and the number of backup copies of the data storage files; each data storage file is used for storing part of service data of the current service; the service data stored in different data storage files are different;
the creation information generation module is used for generating creation information of the data storage file and the backup copy according to the configuration parameters and the number of the nodes; the creation information is used for indicating the nodes where the data storage files and the backup copies are located;
a sending module, configured to send the creation information to the node; and the node is used for creating the data storage file and the backup copy according to the creation information.
11. An electronic device, comprising:
a processor and a memory for storing executable instructions operable on the processor, wherein:
the processor is configured to execute the executable instructions, when the executable instructions are executed, to perform the method of any of the preceding claims 1 to 9.
12. A non-transitory computer-readable storage medium having stored thereon computer-executable instructions that, when executed by a processor, perform the method of any of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210405323.3A CN114866570A (en) | 2022-04-18 | 2022-04-18 | Information processing method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210405323.3A CN114866570A (en) | 2022-04-18 | 2022-04-18 | Information processing method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114866570A true CN114866570A (en) | 2022-08-05 |
Family
ID=82630870
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210405323.3A Pending CN114866570A (en) | 2022-04-18 | 2022-04-18 | Information processing method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114866570A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116991562A (en) * | 2023-09-28 | 2023-11-03 | 宁波银行股份有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN116991543A (en) * | 2023-09-26 | 2023-11-03 | 阿里云计算有限公司 | Host, virtualized instance introspection method and storage medium |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102622185A (en) * | 2011-01-27 | 2012-08-01 | 北京东方广视科技股份有限公司 | Method for storing document in plurality of storage units and storage allocation method |
| US20150088827A1 (en) * | 2013-09-26 | 2015-03-26 | Cygnus Broadband, Inc. | File block placement in a distributed file system network |
| US20170339008A1 (en) * | 2016-05-17 | 2017-11-23 | Microsoft Technology Licensing, Llc | Distributed operational control in computing systems |
| CN108156040A (en) * | 2018-01-30 | 2018-06-12 | 北京交通大学 | A kind of central control node in distribution cloud storage system |
| CN111752892A (en) * | 2019-03-27 | 2020-10-09 | 北京京东尚科信息技术有限公司 | Distributed file system, method for implementing the same, management system, device, and medium |
| CN111831232A (en) * | 2020-07-22 | 2020-10-27 | 浙江大华技术股份有限公司 | Data storage method and device, storage medium and electronic device |
| CN112948354A (en) * | 2021-03-01 | 2021-06-11 | 北京金山云网络技术有限公司 | Method and device for creating copy cluster, electronic device and storage medium |
| CN113722275A (en) * | 2021-09-08 | 2021-11-30 | 重庆紫光华山智安科技有限公司 | Object storage space management method, device, server and storage medium |
| CN114138737A (en) * | 2022-02-08 | 2022-03-04 | 亿次网联(杭州)科技有限公司 | File storage method, device, equipment and storage medium |
-
2022
- 2022-04-18 CN CN202210405323.3A patent/CN114866570A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102622185A (en) * | 2011-01-27 | 2012-08-01 | 北京东方广视科技股份有限公司 | Method for storing document in plurality of storage units and storage allocation method |
| US20150088827A1 (en) * | 2013-09-26 | 2015-03-26 | Cygnus Broadband, Inc. | File block placement in a distributed file system network |
| US20170339008A1 (en) * | 2016-05-17 | 2017-11-23 | Microsoft Technology Licensing, Llc | Distributed operational control in computing systems |
| CN108156040A (en) * | 2018-01-30 | 2018-06-12 | 北京交通大学 | A kind of central control node in distribution cloud storage system |
| CN111752892A (en) * | 2019-03-27 | 2020-10-09 | 北京京东尚科信息技术有限公司 | Distributed file system, method for implementing the same, management system, device, and medium |
| CN111831232A (en) * | 2020-07-22 | 2020-10-27 | 浙江大华技术股份有限公司 | Data storage method and device, storage medium and electronic device |
| CN112948354A (en) * | 2021-03-01 | 2021-06-11 | 北京金山云网络技术有限公司 | Method and device for creating copy cluster, electronic device and storage medium |
| CN113722275A (en) * | 2021-09-08 | 2021-11-30 | 重庆紫光华山智安科技有限公司 | Object storage space management method, device, server and storage medium |
| CN114138737A (en) * | 2022-02-08 | 2022-03-04 | 亿次网联(杭州)科技有限公司 | File storage method, device, equipment and storage medium |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116991543A (en) * | 2023-09-26 | 2023-11-03 | 阿里云计算有限公司 | Host, virtualized instance introspection method and storage medium |
| CN116991543B (en) * | 2023-09-26 | 2024-02-02 | 阿里云计算有限公司 | Host, virtualized instance introspection method and storage medium |
| CN116991562A (en) * | 2023-09-28 | 2023-11-03 | 宁波银行股份有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN116991562B (en) * | 2023-09-28 | 2023-12-26 | 宁波银行股份有限公司 | Data processing method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114866570A (en) | Information processing method and device, electronic equipment and storage medium | |
| CN110737442A (en) | edge application management method and system | |
| CN113778623B (en) | Resource processing method and device, electronic equipment and storage medium | |
| EP3442201B1 (en) | Cloud platform construction method and cloud platform | |
| CN110674095B (en) | CTDB cluster expansion method, device and equipment and readable storage medium | |
| CN113382077B (en) | Micro-service scheduling method, micro-service scheduling device, computer equipment and storage medium | |
| US20190004844A1 (en) | Cloud platform construction method and cloud platform | |
| CN111399764A (en) | Data storage method, data reading device, data storage equipment and data storage medium | |
| CN112783646A (en) | Stateful application containerization deployment method and device | |
| CN111464603A (en) | Server capacity expansion method and system | |
| CN112470119B (en) | Service upgrading method and device in distributed system and distributed system | |
| CN110113217B (en) | Micro-service management method, device, management platform and storage medium | |
| CN114546493A (en) | Core sharing method and device, processing core, electronic device and medium | |
| CN109002263B (en) | Method and device for adjusting storage capacity | |
| CN106534758B (en) | Conference backup method and device | |
| CN109032674B (en) | Multi-process management method, system and network equipment | |
| CN111026425A (en) | Service updating method, device, server and medium | |
| CN112035579A (en) | Graph management method, data storage method, data query method, device and storage medium | |
| CN113032107B (en) | Resource management method, device and system of cloud database | |
| CN112153093B (en) | Cluster-based task scheduling method, device, equipment and readable storage medium | |
| CN113093995A (en) | Cloud disk data migration method and system | |
| CN112822792A (en) | Distribution method, device, equipment and storage medium of bearing identification | |
| CN112839068B (en) | Management method and device for network nodes deployed by NFV (network File management) components | |
| CN116112499B (en) | Construction method of data acquisition system and data acquisition method | |
| CN111431951A (en) | Data processing method, node equipment, system and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |