CN113225362B - Server cluster system and implementation method thereof - Google Patents
Server cluster system and implementation method thereof Download PDFInfo
- Publication number
- CN113225362B CN113225362B CN202010081707.5A CN202010081707A CN113225362B CN 113225362 B CN113225362 B CN 113225362B CN 202010081707 A CN202010081707 A CN 202010081707A CN 113225362 B CN113225362 B CN 113225362B
- Authority
- CN
- China
- Prior art keywords
- information
- layer cluster
- cluster
- storage
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 25
- 239000012634 fragment Substances 0.000 claims abstract description 85
- 239000003795 chemical substances by application Substances 0.000 claims abstract description 49
- 238000004364 calculation method Methods 0.000 claims abstract description 19
- 230000005012 migration Effects 0.000 claims description 21
- 238000013508 migration Methods 0.000 claims description 21
- 238000012545 processing Methods 0.000 claims description 11
- 238000013467 fragmentation Methods 0.000 claims description 9
- 238000006062 fragmentation reaction Methods 0.000 claims description 9
- 238000010586 diagram Methods 0.000 description 12
- 230000006870 function Effects 0.000 description 10
- 230000000694 effects Effects 0.000 description 5
- 230000007246 mechanism Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 230000008447 perception Effects 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 238000004590 computer program Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000000087 stabilizing effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Abstract
The invention discloses a server cluster system and an implementation method of the server cluster system, and relates to the technical field of computers. The system of the method comprises: agent layer clusters, management layer clusters, computation layer clusters, storage layer clusters; the agent layer cluster is used for receiving the request information, acquiring the slicing information corresponding to the request information from the management layer cluster according to the request information, and sending the slicing information to the calculation layer cluster; the storage layer cluster is used for storing document data information; the computing layer cluster is used for computing according to the fragment information, and determining data request result information corresponding to the request information from the storage layer cluster. According to the embodiment, an ES server architecture in the prior art can be broken, a platform server cluster system is established, resources can be used in a larger mode, the system can be balanced in load better, and the system is stable.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a server cluster system and a method for implementing the server cluster system.
Background
The elastiscearch is a distributed, high-expansion, high-real-time search server, abbreviated as ES server, capable of document searching. In large corporations, multiple ES server clusters are often used simultaneously, and data between these server clusters is not interworking.
In the process of implementing the present invention, the inventor finds that at least the following problems exist in the prior art:
because multiple ES server clusters are used and data among the server clusters are not communicated, the clusters cannot share resources and the resources cannot be used more optimally, and a problem of large load difference among the clusters often occurs.
Disclosure of Invention
In view of this, the embodiments of the present invention provide a server cluster system and a method for implementing the server cluster system, which can break through the ES server architecture in the prior art, and build a platform server cluster system, so that resources can be used more greatly, and the system can be better load-balanced, and further the system is more stable.
To achieve the above object, according to an aspect of an embodiment of the present invention, there is provided a server cluster system.
The server cluster system of the embodiment of the invention comprises: agent layer clusters, management layer clusters, computation layer clusters, storage layer clusters; wherein,
the agent layer cluster is used for receiving the request information, acquiring the slicing information corresponding to the request information from the management layer cluster according to the request information, and sending the slicing information to the calculation layer cluster;
the storage layer cluster is used for storing document data information;
the computing layer cluster is used for computing according to the fragment information, and determining data request result information corresponding to the request information from the storage layer cluster.
Optionally, the proxy layer cluster includes a plurality of proxy nodes; the management layer cluster comprises a plurality of management nodes; the computing layer cluster comprises a plurality of computing nodes; the storage layer cluster comprises a plurality of storage nodes.
Optionally, the storage node includes a sliced storage unit; the document data is divided into at least one fragment data and stored in the fragment storage unit; wherein, a fragment data is stored in a fragment storage unit; at least one fragment data is stored in one fragment storage unit.
Optionally, the agent layer cluster acquires a slicing message corresponding to the request information from the management layer cluster according to the request information, including:
determining a target proxy node according to the load condition of the proxy layer cluster;
the target agent node determines a target management node from the management layer cluster according to the request information; and determining the slicing message according to the target management node.
Optionally, the document data corresponding to the request information is taken as a target document, and the slicing information at least comprises: index information of each fragment data in the target document, wherein the index information stores a corresponding relation between a management node and a calculation node; the distribution information of each fragment data in the target document in the fragment storage unit is stored with the corresponding relation between the computing node and the storage node;
the computing layer cluster is configured to perform computing according to the fragmentation information, determine data request result information corresponding to the request information from the storage layer cluster, and include:
determining a target computing node from the computing layer cluster according to the index information;
and the target computing node performs route computation according to the distribution information, determines a target storage node where the target document is located from the storage layer cluster, and determines the data request result information from the target storage node.
Optionally, after determining the data request result information from the storage layer cluster, the computing layer cluster further includes: returning the data request result information to the proxy layer cluster;
the proxy layer cluster is further configured to: and carrying out merging processing on the received data request result information.
Optionally, when the system is upgraded, the method includes:
writing new computing nodes after upgrading and copying old computing nodes before upgrading based on the agent layer cluster to determine an upgraded computing layer cluster; and updating the fragmentation information after confirming that the updated computing layer cluster is established.
Optionally, when the system expands, the method includes:
writing new computing nodes after capacity expansion and copying old computing nodes before capacity expansion based on the agent layer cluster so as to determine a capacity-expanded computing layer cluster; and updating the fragment information after confirming that the establishment of the expanded computing layer cluster is completed.
Optionally, when the system expands, the method further comprises:
for the agent layer cluster, adding a new agent node for receiving request information and/or data request result information; and/or the number of the groups of groups,
for the management layer cluster, adding a new management node for managing and updating the fragment information; and/or the number of the groups of groups,
for the storage layer cluster, a new storage node is added for storing the document data information.
Optionally, the storage layer cluster is further used for data migration, including:
copying the data in the old storage node before migration to the new storage node after migration by using the fragmented storage units with access quantity lower than a preset threshold value in the storage nodes; and updating the fragment information after confirming that the data migration is completed.
According to still another aspect of the embodiment of the present invention, a method for implementing a server cluster system is provided.
The embodiment of the invention provides a method for realizing a server cluster system, which comprises the following steps: agent layer clusters, management layer clusters, computation layer clusters, storage layer clusters; the method comprises the following steps:
the agent layer cluster receives the request information, acquires the slicing information corresponding to the request information from the management layer cluster according to the request information, and sends the slicing information to the calculation layer cluster;
and the computing layer cluster performs computation according to the fragment information, and determines data request result information corresponding to the request information from the storage layer cluster storing the document data information.
Optionally, the proxy layer cluster includes a plurality of proxy nodes; the management layer cluster comprises a plurality of management nodes; the computing layer cluster comprises a plurality of computing nodes; the storage layer cluster comprises a plurality of storage nodes.
Optionally, the storage node includes a sliced storage unit; the document data is divided into at least one fragment data and stored in the fragment storage unit; wherein, a fragment data is stored in a fragment storage unit; at least one fragment data is stored in one fragment storage unit.
Optionally, the agent layer cluster acquires a slicing message corresponding to the request information from the management layer cluster according to the request information, including:
determining a target proxy node according to the load condition of the proxy layer cluster;
the target agent node determines a target management node from the management layer cluster according to the request information; and determining the slicing message according to the target management node.
Optionally, the document data corresponding to the request information is taken as a target document, and the slicing information at least comprises: index information of each fragment data in the target document, wherein the index information stores a corresponding relation between a management node and a calculation node; the distribution information of each fragment data in the target document in the fragment storage unit is stored with the corresponding relation between the computing node and the storage node;
the computing layer cluster performs computing according to the fragment information, determines data request result information corresponding to the request information from the storage layer cluster, and comprises the following steps:
determining a target computing node from the computing layer cluster according to the index information;
and the target computing node performs route computation according to the distribution information, determines a target storage node where the target document is located from the storage layer cluster, and determines the data request result information from the target storage node.
Optionally, after determining the data request result information from the storage layer cluster, the computing layer cluster further includes:
returning the data request result information to the proxy layer cluster;
and the proxy layer cluster performs merging processing on the received data request result information.
Optionally, when the system is upgraded, the method includes:
writing new computing nodes after upgrading and copying old computing nodes before upgrading based on the agent layer cluster to determine an upgraded computing layer cluster; and updating the fragmentation information after confirming that the updated computing layer cluster is established.
Optionally, when the system expands, the method includes:
writing new computing nodes after capacity expansion and copying old computing nodes before capacity expansion based on the agent layer cluster so as to determine a capacity-expanded computing layer cluster; and updating the fragment information after confirming that the establishment of the expanded computing layer cluster is completed.
Optionally, when the system expands, the method further comprises:
for the agent layer cluster, adding a new agent node for receiving request information and/or data request result information; and/or the number of the groups of groups,
for the management layer cluster, adding a new management node for managing and updating the fragment information; and/or the number of the groups of groups,
for the storage layer cluster, a new storage node is added for storing the document data information.
Optionally, the implementation method further includes performing data migration by using the storage layer cluster, including:
copying the data in the old storage node before migration to the new storage node after migration by using the fragmented storage units with access quantity lower than a preset threshold value in the storage nodes; and updating the fragment information after confirming that the data migration is completed.
One embodiment of the above invention has the following advantages or benefits: the server cluster system comprises a proxy layer cluster, a management layer cluster, a calculation layer cluster and a storage layer cluster, wherein the proxy layer cluster faces to a client and can provide a stable receiving platform for a request from the client; therefore, the technical problems that a plurality of clusters cannot share resources and the load difference among the clusters is large in the prior art are solved, and the technical effects that resources can be used more greatly, the system can be balanced in load better and the system is more stable are achieved by establishing a platform server cluster system.
Further effects of the above-described non-conventional alternatives are described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
FIG. 1 is a schematic diagram of clusters of a server cluster system in accordance with an embodiment of the invention;
FIG. 2 is a schematic diagram of an architecture for fragment management in a server cluster system according to an embodiment of the invention;
FIG. 3 is a schematic architecture diagram of a server cluster system according to an embodiment of the invention;
FIG. 4 is a schematic diagram of the main steps of a method for implementing a server cluster system according to an embodiment of the invention;
Detailed Description
Exemplary embodiments of the present invention will now be described with reference to the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding, and are to be considered merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Fig. 1 is a schematic diagram of clusters of a server cluster system according to an embodiment of the present invention, and as shown in fig. 1, a server cluster system according to an embodiment of the present invention includes: agent layer cluster 101, management layer cluster 102, computation layer cluster 103, storage layer cluster 104; wherein,
the agent layer cluster 101 is configured to receive request information, acquire fragment information corresponding to the request information from the management layer cluster 102 according to the request information, and send the fragment information to the calculation layer cluster;
the storage layer cluster 104 is used for storing document data information;
the computing layer cluster 103 is configured to perform computing according to the slice information, and determine data request result information corresponding to the request information from the storage layer cluster 104.
The request information may be issued by a client machine; the agent layer cluster 101 is set facing the client, and can provide a stable receiving platform for the request from the client;
the management layer cluster 102 may be used to load-balance manage various other clusters in the system;
the computing layer cluster 103 and the storage layer cluster 104 are two separated parts, and the computing function and the storage function are separated, so that the computing layer cluster 103 is more concentrated in computing, and the computing capability of the system is further enhanced;
the storage layer cluster 104 can be used for storing all document data information for access, and can make the storage of the document data more stable and can manage the document data more conveniently.
By the aid of the arrangement, the server cluster system can be built into a platform, so that resources can be used more greatly, the system can be balanced in load better, and the technical effect of stabilizing the system is achieved.
In some embodiments, the proxy layer cluster 101 includes a plurality of proxy nodes; the management layer cluster 102 comprises a plurality of management nodes; the computing layer cluster 103 comprises a plurality of computing nodes; the storage layer cluster 104 includes a plurality of storage nodes.
In the server cluster system provided by the embodiment of the invention, all clusters are mutually related and are all high-availability clusters composed of a plurality of nodes;
in the prior art, the clusters are isolated from each other, and each cluster has only one master node, which is similar to the management node in the present invention; in the prior art, the master nodes of each cluster cannot be managed uniformly, but in the invention, a plurality of management nodes in the management layer cluster can be managed uniformly, so that the whole server cluster system can be managed uniformly.
In some embodiments, the storage node includes a tiled memory element therein; the document data is divided into at least one fragment data and stored in the fragment storage unit; wherein, a fragment data is stored in a fragment storage unit; at least one fragment data is stored in one fragment storage unit.
The storage nodes in the storage layer cluster 104 may be storage media with storage functions, such as floppy disks and hard disks;
the document data can be stably stored in the storage layer cluster, and the storage layer cluster comprises the storage nodes, so that the capacity expansion and the capacity reduction are easy, and the system is applicable to document data of various magnitudes; the storage node comprises a fragment storage unit, and the document data can be divided into at least one fragment data, and the fragment data can be stored in the fragment storage unit in the storage node.
In some embodiments, the agent layer cluster 101 obtains, from the management layer cluster 102, a slicing message corresponding to the request information according to the request information, including:
determining a target proxy node according to the load condition of the proxy layer cluster 101;
the target agent node determines a target management node from the management layer cluster 102 according to the request information; and determining the slicing message according to the target management node.
The request information can be sent by a client, after the client sends the request information to the proxy layer cluster, the load condition information of the current proxy layer cluster can be obtained, and further, a proxy node for processing the request information can be determined according to the load condition information to serve as a target proxy node;
the document data corresponding to the request information is taken as a target document, the request information can contain the type information of the target document, a target management node can be determined in a management layer cluster according to the type information, and the target management node contains necessary information, namely the fragmentation information, required by processing the request information in a subsequent step.
In some embodiments, the document data corresponding to the request information is taken as a target document, and the slicing information at least comprises: index information of each fragment data in the target document, wherein the index information stores a corresponding relation between a management node and a calculation node; the distribution information of each fragment data in the target document in the fragment storage unit is stored with the corresponding relation between the computing node and the storage node;
the computing layer cluster 103 is configured to perform computing according to the slice information, determine data request result information corresponding to the request information from the storage layer cluster 104, and include:
determining a target computing node from the computing layer cluster 103 according to the index information;
the target computing node performs route calculation according to the distribution information, determines a target storage node where the target document is located from the storage layer cluster 104, and determines the data request result information from the target storage node.
The slicing information can be determined when the document data is written into the storage layer cluster, and the slicing information and the type information of the document data can also have a corresponding relation;
the index information of each piece of data in the target document can be regarded as the corresponding relation between the management node and the computing node, and the distribution information of each piece of data in the target document in the fragment storage unit can be regarded as the corresponding relation between the computing node and the storage node;
in some embodiments, a management node may correspond to managing at least one computing node; one management node may manage at least one sharded storage unit correspondingly, and a sharded storage unit managed by one management node may not exist in one storage node.
FIG. 2 is a schematic diagram of an architecture for fragment management in a server cluster system according to an embodiment of the invention; as shown in fig. 2, includes: a management node 201, a first storage node 202, a second storage node 203, a third storage node 204; wherein,
the management node 201 manages: the sliced storage unit 1-2 in the first storage node 202, the sliced storage unit 2-2 in the second storage node 203, and the sliced storage unit 3-2 in the third storage node 204.
In some embodiments, after determining the data request result information from the storage layer cluster 104, the computing layer cluster 103 further includes: returning the data request result information to the agent layer cluster 101;
the proxy layer cluster 101 is further configured to: and carrying out merging processing on the received data request result information.
The target proxy node in the proxy layer cluster 101 may perform merging processing on the received data request result information.
In some embodiments, when the system is upgraded, it comprises:
writing new computing nodes after upgrading and copying old computing nodes before upgrading based on the agent layer cluster 101 to determine an upgraded computing layer cluster; and after confirming that the updated computing layer cluster is established, updating the slicing information in the management layer cluster 102.
Writing in new computing nodes after upgrading and copying old computing nodes before upgrading, namely upgrading the system by adopting a double-writing mechanism, so that the upgrading can be completed transparently to a user without perception of the user, and the user experience is improved.
In some embodiments, when the system expands, it comprises:
writing new computing nodes after capacity expansion and copying old computing nodes before capacity expansion based on the agent layer cluster 101 to determine a capacity-expanded computing layer cluster; and after confirming that the establishment of the expanded computing layer cluster is completed, updating the slicing information in the management layer cluster 102.
Writing in new computing nodes after capacity expansion and copying old computing nodes before capacity expansion, namely adopting a double-writing mechanism to expand the capacity of the system, thereby achieving the purposes of being transparent to users and finishing upgrading under the condition that users do not feel, and improving user experience.
In some embodiments, when the system expands, further comprising:
for the agent layer cluster, adding a new agent node for receiving request information and/or data request result information; and/or the number of the groups of groups,
for the management layer cluster, adding a new management node for managing and updating the fragment information; and/or the number of the groups of groups,
for the storage layer cluster, a new storage node is added for storing the document data information.
In some embodiments, the storage layer cluster is further for data migration, comprising:
copying the data in the old storage node before migration to the new storage node after migration by using the fragmented storage units with access quantity lower than a preset threshold value in the storage nodes; and updates the shard information in the management layer cluster 102 after confirming that the data migration is complete.
In some embodiments, when a storage medium has a need for replacement, data migration may be performed, such as: data can be migrated from the ssd hard disk to the mechanical hard disk to save costs.
Fig. 3 is a schematic architecture diagram of a server cluster system according to an embodiment of the present invention, as shown in fig. 3, including: client cluster 301, proxy layer cluster 302, management layer cluster 303, computing layer cluster 304, storage layer cluster 305; wherein,
client cluster 301 may consist of client 1, client 2, clients 3, … …, client n, which may send request information to proxy layer cluster 302;
after receiving the request information, the agent layer cluster 302 can acquire the slicing information from the management layer cluster 303, and then determine a request result corresponding to the request information from the storage layer cluster 305 by using the computing layer cluster 304 according to the slicing information;
the proxy layer cluster 302 may merge the request results and return the merged results to the client cluster 301.
Client cluster 301, proxy layer cluster 302, compute layer cluster 304, and storage layer cluster 305 may all be managed by management layer cluster 303.
When the server cluster system of the present invention is upgraded, and/or expanded, and/or data migrated, there may be corresponding changes in the management layer cluster 303.
Fig. 4 is a schematic diagram of main steps of an implementation method of a server cluster system according to an embodiment of the present invention, and as shown in fig. 4, the main steps of the implementation method of the server cluster system according to the embodiment of the present invention may include:
step S401: the agent layer cluster 101 receives request information, acquires fragment information corresponding to the request information from the management layer cluster 102 according to the request information, and sends the fragment information to the calculation layer cluster;
step S402: the computing layer cluster 103 performs computation according to the piece information, and determines data request result information corresponding to the request information from the storage layer clusters 104 storing document data information.
The request information may be issued by a client machine; the agent layer cluster is arranged facing the client, and can provide a stable receiving platform for the request from the client;
the management layer clusters can be used for managing each other cluster in the system in a load balancing manner;
the computing layer cluster and the storage layer cluster are two separated parts, and the computing function and the storage function are separated, so that the computing layer cluster is more concentrated in computing, and the computing capability of the system is stronger;
the storage layer cluster can be used for storing all document data information for access, can enable the storage of the document data to be more stable, and can manage the document data more conveniently.
By the adoption of the implementation method of the server cluster system, a platform server cluster system can be established, so that resources can be used more greatly, the system can be balanced in load better, and further the technical effect of enabling the system to be more stable is achieved.
In some embodiments, the proxy layer cluster includes a plurality of proxy nodes; the management layer cluster comprises a plurality of management nodes; the computing layer cluster comprises a plurality of computing nodes; the storage layer cluster comprises a plurality of storage nodes.
In the server cluster system provided by the embodiment of the invention, all clusters are mutually related and are all high-availability clusters composed of a plurality of nodes;
in the prior art, the clusters are isolated from each other, and each cluster has only one master node, which is similar to the management node in the present invention; in the prior art, the master nodes of each cluster cannot be managed uniformly, but in the invention, a plurality of management nodes in the management layer cluster can be managed uniformly, so that the whole server cluster system can be managed uniformly.
In some embodiments, the storage node includes a tiled memory element therein; the document data is divided into at least one fragment data and stored in the fragment storage unit; wherein, a fragment data is stored in a fragment storage unit; at least one fragment data is stored in one fragment storage unit.
The storage nodes in the storage layer cluster can be storage media with a storage function, such as floppy disks and hard disks;
the document data can be stably stored in the storage layer cluster, and the storage layer cluster comprises the storage nodes, so that the capacity expansion and the capacity reduction are easy, and the system is applicable to document data of various magnitudes; the storage node comprises a fragment storage unit, and the document data can be divided into at least one fragment data, and the fragment data can be stored in the fragment storage unit in the storage node.
In some embodiments, the agent layer cluster obtains, from the management layer cluster according to the request information, a fragment message corresponding to the request information, including:
determining a target proxy node according to the load condition of the proxy layer cluster;
the target agent node determines a target management node from the management layer cluster according to the request information; and determining the slicing message according to the target management node.
The request information can be sent by a client, after the client sends the request information to the proxy layer cluster, the load condition information of the current proxy layer cluster can be obtained, and further, a proxy node for processing the request information can be determined according to the load condition information to serve as a target proxy node;
the document data corresponding to the request information is taken as a target document, the request information can contain the type information of the target document, a target management node can be determined in a management layer cluster according to the type information, and the target management node contains necessary information, namely the fragmentation information, required by processing the request information in a subsequent step.
In some embodiments, the document data corresponding to the request information is taken as a target document, and the slicing information at least comprises: index information of each fragment data in the target document, wherein the index information stores a corresponding relation between a management node and a calculation node; the distribution information of each fragment data in the target document in the fragment storage unit is stored with the corresponding relation between the computing node and the storage node;
the computing layer cluster performs computing according to the fragment information, determines data request result information corresponding to the request information from the storage layer cluster, and comprises the following steps:
determining a target computing node from the computing layer cluster according to the index information;
and the target computing node performs route computation according to the distribution information, determines a target storage node where the target document is located from the storage layer cluster, and determines the data request result information from the target storage node.
The slicing information can be determined when the document data is written into the storage layer cluster, and the slicing information and the type information of the document data can also have a corresponding relation;
the index information of each piece of data in the target document can be regarded as the corresponding relation between the management node and the computing node, and the distribution information of each piece of data in the target document in the fragment storage unit can be regarded as the corresponding relation between the computing node and the storage node;
in some embodiments, a management node may correspond to managing at least one computing node; one management node may manage at least one sharded storage unit correspondingly, and a sharded storage unit managed by one management node may not exist in one storage node.
In some embodiments, after determining the data request result information from the storage layer cluster, the computing layer cluster further includes:
returning the data request result information to the proxy layer cluster;
and the proxy layer cluster performs merging processing on the received data request result information.
The target proxy node in the proxy layer cluster 101 may perform merging processing on the received data request result information.
In some embodiments, when the system is upgraded, it comprises:
writing new computing nodes after upgrading and copying old computing nodes before upgrading based on the agent layer cluster to determine an upgraded computing layer cluster; and updating the fragmentation information after confirming that the updated computing layer cluster is established.
Writing in new computing nodes after upgrading and copying old computing nodes before upgrading, namely upgrading the system by adopting a double-writing mechanism, so that the upgrading can be completed transparently to a user without perception of the user, and the user experience is improved.
In some embodiments, when the system expands, it comprises:
writing new computing nodes after capacity expansion and copying old computing nodes before capacity expansion based on the agent layer cluster so as to determine a capacity-expanded computing layer cluster; and updating the fragment information after confirming that the establishment of the expanded computing layer cluster is completed.
Writing in new computing nodes after capacity expansion and copying old computing nodes before capacity expansion, namely adopting a double-writing mechanism to expand the capacity of the system, thereby achieving the purposes of being transparent to users and finishing upgrading under the condition that users do not feel, and improving user experience.
In some embodiments, when the system expands, further comprising:
for the agent layer cluster, adding a new agent node for receiving request information and/or data request result information; and/or the number of the groups of groups,
for the management layer cluster, adding a new management node for managing and updating the fragment information; and/or the number of the groups of groups,
for the storage layer cluster, a new storage node is added for storing the document data information.
In some embodiments, the implementation method further includes performing data migration with the storage layer cluster, including:
copying the data in the old storage node before migration to the new storage node after migration by using the fragmented storage units with access quantity lower than a preset threshold value in the storage nodes; and updating the fragment information after confirming that the data migration is completed.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The clusters involved in the embodiments of the present invention may be implemented in software or in hardware. The described clusters may also be provided in a processor, e.g. may be described as: a processor includes a proxy layer cluster, a management layer cluster, a compute layer cluster, and a storage layer cluster. Wherein the names of the clusters do not constitute a limitation of the clusters themselves in some cases.
According to the technical scheme of the embodiment of the invention, the proxy layer cluster faces the client, a stable receiving platform can be provided for a request from the client, a plurality of management nodes are arranged in the management layer cluster, each other cluster and the piece information in the system can be managed in a load-balanced manner, the calculation layer cluster and the storage layer cluster are two separated parts, the calculation layer cluster can be used for calculation only, so that the calculation capability of the system is stronger, the storage layer cluster is used for storing all document data information for access, the storage of the document data is more stable, and the document data can be managed more conveniently; therefore, the technical problems that a plurality of clusters cannot share resources and the load difference among the clusters is large in the prior art are solved, and the technical effects that resources can be used more greatly, the system can be balanced in load better and the system is more stable are achieved by establishing a platform server cluster system.
The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives can occur depending upon design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.
Claims (11)
1. A server cluster system, the system comprising: agent layer clusters, management layer clusters, computation layer clusters, storage layer clusters; wherein,
the agent layer cluster is used for receiving the request information, acquiring the slicing information corresponding to the request information from the management layer cluster according to the request information, and sending the slicing information to the calculation layer cluster; wherein, the document data corresponding to the request information is taken as a target document, and the slicing information at least comprises: index information of each fragment data in the target document, wherein the index information stores a corresponding relation between a management node and a calculation node; the distribution information of each fragment data in the target document in the fragment storage unit is stored with the corresponding relation between the computing node and the storage node;
the storage layer cluster is used for storing document data information;
the computing layer cluster is used for computing according to the fragment information, and determining data request result information corresponding to the request information from the storage layer cluster.
2. The system according to claim 1, characterized in that it comprises:
the agent layer cluster comprises a plurality of agent nodes;
the management layer cluster comprises a plurality of management nodes;
the computing layer cluster comprises a plurality of computing nodes;
the storage layer cluster comprises a plurality of storage nodes.
3. The system according to claim 2, characterized in that it comprises:
the storage node comprises a fragmentation storage unit; the document data is divided into at least one fragment data and stored in the fragment storage unit; wherein, a fragment data is stored in a fragment storage unit; at least one fragment data is stored in one fragment storage unit.
4. The system of claim 3, wherein the agent layer cluster obtaining, from the management layer cluster according to the request information, a slicing message corresponding to the request information, includes:
determining a target proxy node according to the load condition of the proxy layer cluster;
the target agent node determines a target management node from the management layer cluster according to the request information; and determining the slicing message according to the target management node.
5. The system of claim 4, wherein the computing layer cluster is configured to perform computation according to the fragmentation information, and determining data request result information corresponding to the request information from the storage layer cluster includes:
determining a target computing node from the computing layer cluster according to the index information;
and the target computing node performs route computation according to the distribution information, determines a target storage node where the target document is located from the storage layer cluster, and determines the data request result information from the target storage node.
6. The system of claim 1, wherein the computing layer cluster, after determining the data request result information from the storage layer cluster, further comprises: returning the data request result information to the proxy layer cluster;
the proxy layer cluster is further configured to: and carrying out merging processing on the received data request result information.
7. The system of any of claims 1-6, wherein when the system is upgraded, comprising:
writing new computing nodes after upgrading and copying old computing nodes before upgrading based on the agent layer cluster to determine an upgraded computing layer cluster; and updating the fragmentation information after confirming that the updated computing layer cluster is established.
8. The system of any one of claims 1-6, wherein when the system expands, comprising:
writing new computing nodes after capacity expansion and copying old computing nodes before capacity expansion based on the agent layer cluster so as to determine a capacity-expanded computing layer cluster; and updating the fragment information after confirming that the establishment of the expanded computing layer cluster is completed.
9. The system of claim 8, wherein when the system expands, further comprising:
for the agent layer cluster, adding a new agent node for receiving request information and/or data request result information; and/or the number of the groups of groups,
for the management layer cluster, adding a new management node for managing and updating the fragment information; and/or the number of the groups of groups,
for the storage layer cluster, a new storage node is added for storing the document data information.
10. The system of any of claims 1-6, wherein the storage layer cluster is further configured for data migration, comprising:
copying the data in the old storage node before migration to the new storage node after migration by using the fragmented storage units with access quantity lower than a preset threshold value in the storage nodes; and updating the fragment information after confirming that the data migration is completed.
11. The implementation method of the server cluster system is characterized in that the server cluster system comprises the following steps: agent layer clusters, management layer clusters, computation layer clusters, storage layer clusters; the method comprises the following steps:
the agent layer cluster receives the request information, acquires the slicing information corresponding to the request information from the management layer cluster according to the request information, and sends the slicing information to the calculation layer cluster; wherein, the document data corresponding to the request information is taken as a target document, and the slicing information at least comprises: index information of each fragment data in the target document, wherein the index information stores a corresponding relation between a management node and a calculation node; the distribution information of each fragment data in the target document in the fragment storage unit is stored with the corresponding relation between the computing node and the storage node;
and the computing layer cluster performs computation according to the fragment information, and determines data request result information corresponding to the request information from the storage layer cluster storing the document data information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010081707.5A CN113225362B (en) | 2020-02-06 | 2020-02-06 | Server cluster system and implementation method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010081707.5A CN113225362B (en) | 2020-02-06 | 2020-02-06 | Server cluster system and implementation method thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113225362A CN113225362A (en) | 2021-08-06 |
| CN113225362B true CN113225362B (en) | 2024-04-05 |
Family
ID=77085546
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010081707.5A Active CN113225362B (en) | 2020-02-06 | 2020-02-06 | Server cluster system and implementation method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113225362B (en) |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102479207A (en) * | 2010-11-29 | 2012-05-30 | 阿里巴巴集团控股有限公司 | Information search method, system and device |
| CN103077245A (en) * | 2013-01-18 | 2013-05-01 | 浪潮电子信息产业股份有限公司 | Method for expanding parallel file system by free hard disk space of cluster computing node |
| CN104320401A (en) * | 2014-10-31 | 2015-01-28 | 北京思特奇信息技术股份有限公司 | Big data storage and access system and method based on distributed file system |
| CN104580503A (en) * | 2015-01-26 | 2015-04-29 | 浪潮电子信息产业股份有限公司 | Efficient dynamic load balancing system and method for processing large-scale data |
| US9697213B1 (en) * | 2013-06-12 | 2017-07-04 | EMC IP Holding Company LLC | Cluster file system comprising object storage server tier and scale-out network attached storage tier |
| CN107872404A (en) * | 2016-09-23 | 2018-04-03 | 中兴通讯股份有限公司 | Business burst processing method, device and software defined network controller |
| CN109033130A (en) * | 2018-06-04 | 2018-12-18 | 温州市图盛科技有限公司 | A kind of block chain electric power data storage system |
| CN109729129A (en) * | 2017-10-31 | 2019-05-07 | 华为技术有限公司 | Configuration modification method, storage cluster and the computer system of storage cluster |
| CN109828960A (en) * | 2018-12-14 | 2019-05-31 | 平安科技（深圳）有限公司 | Log library expansion method, system, computer installation and readable storage medium storing program for executing |
| CN109933631A (en) * | 2019-03-20 | 2019-06-25 | 江苏瑞中数据股份有限公司 | Distributed parallel database system and data processing method based on Infiniband network |
| CN109992196A (en) * | 2017-12-29 | 2019-07-09 | 杭州海康威视数字技术股份有限公司 | Storage method and device, the storage system of index data |
| CN110018878A (en) * | 2018-01-09 | 2019-07-16 | 阿里巴巴集团控股有限公司 | A kind of distributed system data load method and device |
| CN110427284A (en) * | 2019-07-31 | 2019-11-08 | 中国工商银行股份有限公司 | Data processing method, distributed system, computer system and medium |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7617292B2 (en) * | 2001-06-05 | 2009-11-10 | Silicon Graphics International | Multi-class heterogeneous clients in a clustered filesystem |
| US8812453B2 (en) * | 2010-01-28 | 2014-08-19 | Hewlett-Packard Development Company, L.P. | Database archiving using clusters |
| US10209982B2 (en) * | 2017-05-16 | 2019-02-19 | Bank Of America Corporation | Distributed storage framework information server platform architecture |
-
2020
- 2020-02-06 CN CN202010081707.5A patent/CN113225362B/en active Active
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102479207A (en) * | 2010-11-29 | 2012-05-30 | 阿里巴巴集团控股有限公司 | Information search method, system and device |
| CN103077245A (en) * | 2013-01-18 | 2013-05-01 | 浪潮电子信息产业股份有限公司 | Method for expanding parallel file system by free hard disk space of cluster computing node |
| US9697213B1 (en) * | 2013-06-12 | 2017-07-04 | EMC IP Holding Company LLC | Cluster file system comprising object storage server tier and scale-out network attached storage tier |
| CN104320401A (en) * | 2014-10-31 | 2015-01-28 | 北京思特奇信息技术股份有限公司 | Big data storage and access system and method based on distributed file system |
| CN104580503A (en) * | 2015-01-26 | 2015-04-29 | 浪潮电子信息产业股份有限公司 | Efficient dynamic load balancing system and method for processing large-scale data |
| CN107872404A (en) * | 2016-09-23 | 2018-04-03 | 中兴通讯股份有限公司 | Business burst processing method, device and software defined network controller |
| CN109729129A (en) * | 2017-10-31 | 2019-05-07 | 华为技术有限公司 | Configuration modification method, storage cluster and the computer system of storage cluster |
| CN109992196A (en) * | 2017-12-29 | 2019-07-09 | 杭州海康威视数字技术股份有限公司 | Storage method and device, the storage system of index data |
| CN110018878A (en) * | 2018-01-09 | 2019-07-16 | 阿里巴巴集团控股有限公司 | A kind of distributed system data load method and device |
| CN109033130A (en) * | 2018-06-04 | 2018-12-18 | 温州市图盛科技有限公司 | A kind of block chain electric power data storage system |
| CN109828960A (en) * | 2018-12-14 | 2019-05-31 | 平安科技（深圳）有限公司 | Log library expansion method, system, computer installation and readable storage medium storing program for executing |
| CN109933631A (en) * | 2019-03-20 | 2019-06-25 | 江苏瑞中数据股份有限公司 | Distributed parallel database system and data processing method based on Infiniband network |
| CN110427284A (en) * | 2019-07-31 | 2019-11-08 | 中国工商银行股份有限公司 | Data processing method, distributed system, computer system and medium |
Non-Patent Citations (5)
| Title |
|---|
| 一种集群服务器配置和管理工具;王华, 万木春, 彭勤科, 胡保生;计算机应用(01);全文 * |
| 云计算集群服务器系统监控方法的研究;董波;沈青;肖德宝;;计算机工程与科学(10);全文 * |
| 基于智能存储设备的共享存储集群系统研究;黄蓉;俞建新;;计算机工程与设计(16);全文 * |
| 基于服务器负载状况分类的负载均衡调度算法;郭平;李琪;;华中科技大学学报(自然科学版)(S1);全文 * |
| 基于服务器集群的云检索系统的研究与示范;安俊秀;;计算机科学(07);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113225362A (en) | 2021-08-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109729129B (en) | Configuration modification method of storage cluster system, storage cluster and computer system | |
| US9904599B2 (en) | Method, device, and system for data reconstruction | |
| US9052962B2 (en) | Distributed storage of data in a cloud storage system | |
| US8495013B2 (en) | Distributed storage system and method for storing objects based on locations | |
| US9639590B2 (en) | Database system and method for searching database | |
| JP5303038B2 (en) | Storage system that eliminates duplicate data | |
| US11296940B2 (en) | Centralized configuration data in a distributed file system | |
| CN107085539B (en) | cloud database system and dynamic cloud database resource adjustment method | |
| CN102708165B (en) | Document handling method in distributed file system and device | |
| CN105814544B (en) | System and method for supporting persistent partition recovery in a distributed data grid | |
| US9659038B2 (en) | Efficient snapshot read of a database in a distributed storage system | |
| EP3018593B1 (en) | Data storage method and device for distributed database | |
| CN111212111B (en) | Object storage service management method and electronic equipment | |
| KR20120018178A (en) | Swarm-based synchronization over a network of object stores | |
| CN111078121A (en) | Data migration method, system and related components of distributed storage system | |
| CN113268472B (en) | Distributed data storage system and method | |
| JP2009259007A (en) | Distributed storage method, distributed storage system and distributed storage device | |
| CN111078119B (en) | Data reconstruction method, system, device and computer readable storage medium | |
| US11341009B1 (en) | Directing placement of data in cloud storage nodes | |
| CN109254873B (en) | Data backup method, related device and system | |
| US10931750B1 (en) | Selection from dedicated source volume pool for accelerated creation of block data volumes | |
| CN106951443B (en) | Method, equipment and system for synchronizing copies based on distributed system | |
| CN113225362B (en) | Server cluster system and implementation method thereof | |
| US11537619B1 (en) | Replica group modification in a distributed database | |
| US10956442B1 (en) | Dedicated source volume pool for accelerated creation of block data volumes from object data snapshots |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |