CN116684416A - Mirror image distribution method, device and system in network element cluster - Google Patents
Mirror image distribution method, device and system in network element cluster Download PDFInfo
- Publication number
- CN116684416A CN116684416A CN202310823182.1A CN202310823182A CN116684416A CN 116684416 A CN116684416 A CN 116684416A CN 202310823182 A CN202310823182 A CN 202310823182A CN 116684416 A CN116684416 A CN 116684416A
- Authority
- CN
- China
- Prior art keywords
- mirror image
- hash
- edge node
- edge
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 70
- 230000005540 biological transmission Effects 0.000 claims abstract description 176
- 238000012423 maintenance Methods 0.000 claims abstract description 14
- 238000012544 monitoring process Methods 0.000 claims description 17
- 238000012545 processing Methods 0.000 claims description 9
- 238000003709 image segmentation Methods 0.000 claims description 5
- 238000010586 diagram Methods 0.000 description 8
- 238000005516 engineering process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L45/00—Routing or path finding of packets in data switching networks
- H04L45/74—Address processing for routing
- H04L45/745—Address table lookup; Address filtering
- H04L45/7453—Address table lookup; Address filtering using hashing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The embodiment of the invention discloses a mirror image distribution method, device and system in a network element cluster. The method comprises the following steps: dividing the image files, constructing a hash list by the hash value of each image block, and sending the hash list to all edge nodes for independent maintenance; issuing all mirror blocks to a plurality of different edge nodes; analyzing a hash list carried by the received mirror image transmission request, searching a set of edge nodes with corresponding mirror image blocks for each hash value in the hash list one by one, determining the edge node with the minimum network delay according to the network quality of the edge nodes in the set which is dynamically monitored, forwarding the mirror image transmission request to the edge node, transmitting the corresponding mirror image blocks by the edge node, and updating the mirror image block information owned by each edge node after the transmission is completed. The invention utilizes the bandwidth of each edge node to transmit the mirror image, improves the downloading speed of the mirror image, reduces the transmission time, improves the expandability of the distribution of the mirror image, and ensures the reliability and success rate of the transmission of the mirror image.
Description
Technical Field
The present invention relates to the field of network technologies, and in particular, to a method, an apparatus, and a system for distributing images in a network element cluster.
Background
With the development of mobile communication networks, network element clusters have become a common way of deploying computing resources. A network element cluster may be considered to be made up of one central storage node and several edge nodes. Due to the large number of nodes in the cluster, it can be very cumbersome and time consuming if the operations of manually deploying software, configuring the environment, etc. on each node are required. Therefore, automation and standardization of cluster management and deployment becomes particularly important.
Mirroring, also known as mirroring files, is a form of file storage. The image may be considered as a copy of the original file. The mirror image distribution technology can help to realize rapid deployment and configuration of nodes in the cluster, and improve the efficiency and reliability of cluster management.
The existing mirror image distribution scheme mainly comprises the following modes:
the first is a traditional mirror distribution scheme, mirrors are distributed on a central storage node, each node requests to download the mirror from the central storage node, and the central storage node sequentially transmits the mirror file for each node. In the scheme, each node downloads the mirror image from the central node, the bandwidth requirement on the central storage node is too high, network bottlenecks are easily caused, and the distribution efficiency is low due to centralized downloading.
The second type constructs a P2P network between cluster nodes, then splits the image into several blocks, and publishes it into the P2P network. And each node downloads the image block from other nodes, and finally, the downloaded image block is reconstructed. The proposal can reduce the load and bandwidth consumption of the source server and can improve the downloading speed, but the P2P network has certain requirements on the number of nodes for efficient mirror image distribution, and if the number of the nodes is too small, the availability and reliability of the mirror image can be affected.
The third is a binary tree-based mirror distribution scheme, which organizes mirror nodes by constructing a binary tree structure while dividing a mirror file into a plurality of blocks and distributing the blocks to a plurality of mirror nodes, respectively. The implementation of the scheme is complex, the binary tree structure, block level transmission and the like are required to be studied and optimized in depth, and network transmission efficiency is reduced when the binary tree depth is too large.
In general, the existing mirror image distribution scheme has higher complexity in managing the nodes in the network element cluster, and has certain requirements on the number scale of the nodes. Meanwhile, the existing scheme cannot well utilize the bandwidth transmission mirror image of each node, and performance is poor under the condition of high load.
Disclosure of Invention
The embodiment of the invention provides a method, a device and a system for distributing images in a network element cluster, which can solve the problems that the existing image distribution scheme has higher complexity in managing the nodes in the network element cluster, has certain requirements on the number and the scale of the nodes, can not well utilize the bandwidth of each node to transmit images, and has poor performance under the condition of high load.
According to a first aspect of the present invention, there is provided a method for distributing images in a network element cluster, the method being applied to a central storage node and comprising:
dividing a pre-distributed image file into a plurality of blocks, calculating a hash value of each image block, constructing all hash values into a hash list, and transmitting the hash list to all edge nodes for independent maintenance;
distributing all the mirror blocks to a plurality of different edge nodes according to the network quality of the edge nodes, and recording mirror block information owned by each edge node after the distribution of all the mirror blocks is completed;
when a mirror image transmission request of a first edge node is received, wherein the mirror image transmission request carries a first hash list which is independently maintained by the first edge node, and the first hash list only records hash values corresponding to mirror image blocks which are lack by the first edge node; analyzing the mirror image transmission request to obtain the first hash list, searching a set of edge nodes with corresponding mirror image blocks one by one for each hash value in the first hash list, determining a second edge node with minimum network delay according to the network quality of the edge nodes in the dynamically monitored set, forwarding the mirror image transmission request to the second edge node, transmitting the corresponding mirror image blocks to the first edge node by the second edge node, and updating the mirror image block information owned by each edge node after the transmission is completed.
Preferably, the method further comprises:
judging a busy state or an idle state of the edge node according to the network rate, wherein when the network rate of the edge node is smaller than a set network rate value, the edge node is judged to be in the busy state, otherwise, the edge node is judged to be in the idle state;
when receiving the mirror image transmission request of the first edge node, firstly judging the busy state or idle state of each edge node in the set for the set of the searched edge nodes with the corresponding mirror image blocks, and if all the edge nodes in the set are busy states, directly transmitting the corresponding mirror image blocks to the first edge node by the central storage node, and not forwarding the mirror image transmission request to other edge nodes.
Preferably, the method further comprises:
when mirror image transmission requests of a plurality of different edge nodes are received at the same time, resolving to obtain hash lists carried by the mirror image transmission requests, and processing the mirror image transmission requests according to the sequence from the large length to the small length of the hash lists. .
According to a second aspect of the present invention, there is provided another method of image distribution in a network element cluster, the method being applied to an edge node and comprising:
Receiving a hash list sent by a central storage node, wherein the hash list records a plurality of hash values, and each hash value corresponds to one of all image blocks separated by a pre-distributed image file;
when a certain mirror image block is received, calculating a hash value of the mirror image block, comparing the hash value with a locally stored hash list, deleting the corresponding hash value in the hash list if the same hash value exists in the hash list, and deleting the received mirror image block if the same hash value does not exist in the hash list;
when a complete mirror image file is needed, sending a mirror image transmission request to the central storage node, wherein the mirror image transmission request carries a locally stored hash list, and the hash list only records hash values corresponding to missing mirror image blocks; when the hash list stored locally is empty, combining all the received image blocks into a complete image file; and
when receiving the mirror image transmission request forwarded by the central storage node, the forwarded mirror image transmission request carries a target edge node identifier and a hash value, and a mirror image block corresponding to the hash value is transmitted to the target edge node according to the forwarded mirror image transmission request.
According to a third aspect of the present invention, there is provided an apparatus for distributing images in a network element cluster, the apparatus being applied to a central storage node, comprising:
the network monitoring module is used for dynamically monitoring the network quality of all the edge nodes;
the image segmentation module is used for dividing the pre-distributed image file into a plurality of blocks, calculating the hash value of each image block, constructing all the hash values into a hash list, and transmitting the hash list to all edge nodes for independent maintenance;
the mirror image issuing module is used for issuing all mirror image blocks to a plurality of different edge nodes according to the network quality of the edge nodes;
the transmission management module is used for recording mirror image block information owned by each edge node after all the mirror image blocks are issued, and when a mirror image transmission request of a first edge node is received, wherein the mirror image transmission request carries a first hash list which is independently maintained by the first edge node, the first hash list only records hash values corresponding to the mirror image blocks which are lack by the first edge node, the mirror image transmission request is analyzed to obtain the first hash list, each hash value in the first hash list is searched one by one, a set of edge nodes which have corresponding mirror image blocks is searched, a second edge node with minimum network delay is determined according to the network quality of the edge nodes in the set, the mirror image transmission request is forwarded to the second edge node, the second edge node transmits corresponding mirror image blocks to the first edge node, and the mirror image block information owned by each edge node is updated after the transmission is completed.
Preferably, the network monitoring module is further configured to determine a busy state or an idle state of the edge node according to the network rate, where when the network rate of the edge node is less than a set network rate value, it is determined that the edge node is in the busy state, and otherwise, it is in the idle state;
the transmission management module is further configured to, when receiving a mirror image transmission request of a first edge node, first determine a busy state or an idle state of each edge node in the set for the set of the searched edge nodes having a corresponding mirror image block, and if all the edge nodes in the set are busy states, directly transmit the corresponding mirror image block to the first edge node by the central storage node, and no longer forward the mirror image transmission request to other edge nodes.
The transmission management module is further configured to, when receiving mirror image transmission requests of a plurality of different edge nodes at the same time, parse and obtain hash lists carried by the mirror image transmission requests, and process the mirror image transmission requests according to the sequence from the large length to the small length of the hash lists.
According to a fourth aspect of the present invention, there is provided an apparatus for distributing images in a cluster of network elements, the apparatus being applied to an edge node, comprising:
The hash list maintenance module is used for receiving a hash list sent by the central storage node, wherein the hash list records a plurality of hash values, and each hash value corresponds to one of all image blocks separated by the pre-distributed image file; when a certain mirror image block is received, calculating a hash value of the mirror image block, comparing the hash value with a locally stored hash list, deleting the corresponding hash value in the hash list if the same hash value exists in the hash list, and deleting the received mirror image block if the same hash value does not exist in the hash list;
the mirror image merging module is used for sending a mirror image transmission request to the central storage node when a complete mirror image file is needed, wherein the mirror image transmission request carries a locally stored hash list, and the hash list only records hash values corresponding to missing mirror image blocks; when the hash list stored locally is empty, combining all the received image blocks into a complete image file; the method comprises the steps of,
the mirror image transmission module is used for carrying a target edge node identifier and a hash value in a mirror image transmission request forwarded by the central storage node when the mirror image transmission request is received; and transmitting the mirror image block corresponding to the hash value to the target edge node according to the forwarded mirror image transmission request.
According to a fifth aspect of the present invention, there is provided a mirror distribution system in a network element cluster, the system comprising a central storage node and a number of edge nodes, wherein,
the central storage node comprises a network monitoring module, an image segmentation module, an image issuing module and a transmission management module, and is used for executing the image distribution method;
each edge node comprises a hash list receiving module, a hash list maintaining module, an image merging module and an image transmission module, and the edge node is used for executing the image distribution method.
The beneficial effects of the embodiments of the invention are as follows:
according to the mirror image distribution method, device and system in the network element cluster, at the side of the central storage node, the mirror image file is divided into a plurality of blocks, the hash value of each mirror image block is calculated, all the hash values are constructed into hash lists and sent to all the edge nodes, so that each edge node independently maintains the locally stored hash list, the received mirror image blocks are checked according to the hash values, and the reliability of mirror image transmission is ensured; by distributing all the mirror blocks to a plurality of different edge nodes according to the network quality of the edge nodes, all the mirror blocks can be rapidly distributed to the plurality of edge nodes, so that the network bandwidth of the plurality of edge nodes is utilized, the mirror downloading speed is improved, and the transmission time is shortened; by recording and updating the mirror image block information owned by each edge node, the mirror image transmission request can be redirected to other edge nodes, and the expandability of mirror image distribution is improved; and by dynamically monitoring the network quality of the edge nodes, the network quality of the selected edge nodes can be ensured to be better when the edge nodes are selected for mirror image transmission, and the success rate of the mirror image transmission is improved.
Therefore, the scheme provided by the embodiments of the invention does not need to carry out complex management on the nodes in the network element cluster, does not have requirements on the number and the scale of the nodes, can well utilize the bandwidth of each edge node to transmit the mirror image under the condition of high load of the central storage node, improves the downloading speed of the mirror image, reduces the transmission time, can improve the expandability of the distribution of the mirror image, and ensures the reliability and the success rate of the transmission of the mirror image.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments described in the present invention, and other drawings may be obtained according to these drawings for those of ordinary skill in the art. In the drawings:
fig. 1 is a flow chart illustrating a mirror image distribution method in a network element cluster according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a hash table construction process according to one embodiment of the present invention;
fig. 3 is a flow chart illustrating a mirror image distribution method in a network element cluster according to another embodiment of the present invention;
Fig. 4 is a schematic structural diagram of a mirror image distribution apparatus in a network element cluster according to an embodiment of the present invention;
fig. 5 shows a schematic structural diagram of a mirror image distribution device in a network element cluster according to another embodiment of the present invention;
fig. 6 is a schematic structural diagram of a mirror image distribution system in a network element cluster according to an embodiment of the present invention.
Detailed Description
Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. These embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. While exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention may be embodied in various forms and should not be limited to the embodiments set forth herein.
Fig. 1 is a flow chart of a method for distributing images in a network element cluster according to an embodiment of the present invention, where the method is applied to a central storage node. The mirror image distribution method in the network element cluster shown in fig. 1 includes the following steps S110 to S130:
in step S110, the central storage node divides the pre-distributed image file into a plurality of blocks, calculates the hash value of each image block, constructs all the hash values into a hash list, and sends the hash list to all the edge nodes for independent maintenance.
The image file is typically relatively large, e.g., on the order of Gigabytes (GB). To achieve fast distribution, the pre-distributed image file needs to be split into blocks, e.g. into N blocks, where N is an integer greater than or equal to 2. The dividing mode can be equal division or unequal division.
Fig. 2 is a schematic diagram of a process for constructing a hash table according to an embodiment of the present invention. As shown in fig. 2, a Hash value is first calculated for each mirror block, and then all the obtained Hash values Hash1, hash2, … Hashn are constructed as a Hash list [ Hash1, hash2, … Hashn ]. There are various ways to calculate the hash value, such as square method, divide Yu Fa, multiplication method, random number method, etc. The embodiment of the invention does not limit the way of calculating the hash value.
The hash value is an identification of the image block and is used to uniquely identify the image block. And constructing hash values corresponding to all the image blocks of the image file into a list to obtain a hash list of the image file. The hash list is sent to all edge nodes, and each edge node can independently maintain a locally stored hash list, for example, to ensure that only the hash values of the missing mirror blocks are recorded in the locally stored hash list.
Step S120, distributing all the mirror blocks to a plurality of different edge nodes according to the network quality of the edge nodes, and recording the mirror block information owned by each edge node after the distribution of all the mirror blocks is completed.
The central storage node dynamically monitors the network quality of all the edge nodes, and the monitoring frequency is preset. Before the mirror image is issued, the central storage node needs to acquire the network quality condition of all the edge nodes so as to issue all the mirror image blocks to a plurality of different edge nodes according to the network quality of the edge nodes.
The dispensing modes can be as follows:
1) Dispensing one by one: and selecting an edge node with the minimum network delay each time, issuing a mirror image block to the edge node, reselecting the edge node with the minimum network delay after transmission is completed, and issuing another mirror image block to the newly selected edge node until all the mirror image blocks are issued to a plurality of different edge nodes. Wherein each newly selected edge node is determined by dynamically monitored network quality, and the newly selected edge node can be the same as or different from the previously selected edge node.
2) Packet issuing: and selecting the front N edge nodes with low network delay, which are equal to the number N of the mirror image blocks, and if the number of the edge nodes is smaller than N, assuming that K mirror image blocks remained after the first round of distribution, carrying out a new round of distribution on the K mirror image blocks. In particular, the method comprises the steps of,
Under the condition that the number of the edge nodes is not smaller than the number N of the mirror image blocks, selecting the first N edge nodes with small network delay, which are equal to the number N of the mirror image blocks, respectively issuing a mirror image block to each selected edge node, and after all the mirror image blocks are issued, enabling the edge nodes with small network delay to respectively have different mirror image blocks;
and under the condition that the number of the edge nodes is smaller than the number N of the mirror image blocks, selecting the edge nodes with small network delay of the preset number in two or more rounds, respectively issuing one mirror image block to each selected edge node, and performing next round of edge node selection and issuing operation after the previous round of issuing is finished until all the mirror image blocks are issued, so that a plurality of different edge nodes have one or more mirror image blocks.
After all the mirror blocks are issued to a plurality of different edge nodes, the central storage node records the mirror block information owned by each edge node so as to redirect the transmission request of a certain mirror block to the appropriate node owning the mirror block.
Step S130, when a mirror image transmission request of a first edge node is received, wherein the mirror image transmission request carries a first hash list independently maintained by the first edge node, and the first hash list only records hash values corresponding to mirror image blocks lacking by the first edge node; the central storage node analyzes the mirror image transmission request to obtain the first hash list, searches a set of edge nodes with corresponding mirror image blocks one by one for each hash value in the first hash list, determines a second edge node with minimum network delay according to the network quality of the edge nodes in the dynamically monitored set, forwards the mirror image transmission request to the second edge node, transmits the corresponding mirror image blocks to the first edge node by the second edge node, and updates the mirror image block information owned by each edge node after the transmission is completed.
When the edge node needs a complete mirror image file, a mirror image transmission request is sent to the central storage node, and the request carries a hash list of a mirror image block requested by the edge node. When the central storage node receives the request, the central storage node firstly analyzes the request to obtain a hash list carried by the edge node, distributes proper transmission nodes for the request according to a certain rule, and updates mirror image block information owned by each edge node after the transmission is completed, so that the request can be accurately redirected to the proper node later, and the expandability of mirror image distribution is improved.
Taking an example that a first edge node sends a mirror image transmission request to a central storage node, the mirror image transmission request carries a first hash list which is independently maintained by the first edge node, and the first hash list only records hash values corresponding to mirror image blocks lacking by the first edge node. When the central storage node receives the mirror image transmission request of the first edge node, analyzing the mirror image transmission request to obtain the first hash list, and then distributing proper nodes for the mirror image transmission request according to a first processing rule formulated by the embodiment of the invention:
the central storage node searches a set of edge nodes with corresponding mirror blocks for each hash value in the first hash list one by one, determines a second edge node with minimum network delay according to the network quality of the edge nodes in the dynamically monitored set, forwards the mirror transmission request to the second edge node, and transmits the corresponding mirror blocks to the first edge node by the second edge node.
Specifically, a first hash value in the first hash list is firstly searched for a set of edge nodes with corresponding mirror blocks, then an edge node with the minimum network delay, such as a second edge node, is determined according to the network quality of the edge nodes in the dynamically monitored set, a central storage node forwards a request to the second edge node, and the second edge node transmits the mirror blocks corresponding to the first hash values to the first edge node. Then, for a second hash value in the first hash list, searching out a set of edge nodes with corresponding mirror blocks, determining an edge node with the minimum network delay, such as a third edge node, according to the network quality of the edge nodes in the dynamically monitored set, forwarding a request to the third edge node by a central storage node, and transmitting the mirror blocks corresponding to the second hash value to the first edge node by the third edge node. … and so on until the mirror transmission request of the last hash value in the first hash list is also forwarded by the central storage node to the appropriate edge node.
However, there is a problem in that only the first processing rule: when the central storage node processes a certain hash value in the first hash list, if the searched set with the corresponding mirror image blocks, the network quality of all edge nodes is poor. At this time, although the edge node with the minimum network delay, such as the fourth edge node, is determined according to the dynamically monitored network quality, the transmission of the corresponding mirror image block cannot be completed according to the current network quality of the fourth edge node, and the transmission of the mirror image block cannot be abandoned, so that the transmission of the mirror image block fails, and the first edge node cannot receive all the mirror image blocks in time.
To solve the above problem, a second processing rule further formulated by one embodiment of the present invention allocates an appropriate node to the mirror transmission request:
judging a busy state or an idle state of the edge node according to the network rate, wherein when the network rate of the edge node is smaller than a set network rate value, the edge node is judged to be in the busy state, otherwise, the edge node is judged to be in the idle state;
when receiving a mirror image transmission request of the first edge node, a central storage node judges the searched set of edge nodes with corresponding mirror image blocks, and firstly, the busy state or the idle state of each edge node in the set is judged, if all the edge nodes in the set are busy states, the central storage node directly transmits the corresponding mirror image blocks to the first edge node, and the mirror image transmission request is not forwarded to other edge nodes.
Taking the above situation as an example, when the central storage node processes a certain hash value in the first hash list, if the searched set with the corresponding mirror blocks, the network quality of all the edge nodes is poor. Aiming at the situation, according to a second processing rule, a set of the searched edge nodes with the corresponding mirror blocks is firstly judged, if all the edge nodes in the set are busy, the corresponding mirror blocks are directly transmitted to the first edge node by the central storage node, and the mirror transmission request is not forwarded to other edge nodes. Therefore, in the above situation, the mirror image transmission request is not forwarded to the fourth edge node with the determined minimum network delay according to the first processing rule, so that the transmission failure of the corresponding mirror image block is effectively avoided, and even if the network quality of all the edge nodes in the set is poor, the first edge node can be guaranteed to receive all the mirror image blocks in time.
In a real situation, the central storage node may also receive mirror image transmission requests of a plurality of different edge nodes at the same time, and for this situation, a third processing rule further formulated by an embodiment of the present invention allocates an appropriate node to the mirror image transmission request:
when mirror image transmission requests of a plurality of different edge nodes are received at the same time, the center storage node analyzes and obtains hash lists carried by the mirror image transmission requests, and the mirror image transmission requests are processed according to the sequence from the large length to the small length of the hash lists.
The long hash list length represents that the number of image block files requested to be sent is large. The third processing rule preferentially processes the requests of the edge nodes with long hash list length, and can ensure load balancing to a certain extent.
Fig. 3 is a flow chart of a method for distributing images in a network element cluster according to another embodiment of the present invention, where the method is applied to an edge node. The mirror image distribution method in the network element cluster shown in fig. 3 includes the following steps S310 to S340:
in step S310, the edge node receives a hash list sent by the central storage node, where the hash list records a plurality of hash values, and each hash value corresponds to one of all the image blocks separated by the pre-distributed image file.
As previously described, the central storage node separates the pre-distributed image file into several blocks, calculates the hash value of each image block, constructs all hash values into a hash list, and sends the hash list to all edge nodes, so that each edge node maintains a locally stored hash list separately, for example, to ensure that only the hash values of the missing image blocks are recorded in the locally stored hash list.
Step S320, when a certain image block is received, calculating a hash value of the image block, comparing the hash value with a locally stored hash list, deleting the corresponding hash value in the hash list if the same hash value exists in the hash list, and deleting the received image block if the same hash value does not exist in the hash list.
The edge node adopts this step S320 to verify the received mirror image block with the hash value, and separately maintains the locally stored hash list.
The method for testing is as follows: calculating a hash value for each received image block and comparing to a locally stored hash list: if the same hash value exists in the hash list, indicating that the received image block is complete and belongs to one of the image blocks in the image file, and deleting the corresponding hash value in the hash list for marking that the image block is completely received; if the same hash value does not exist in the hash list, this corresponds to two cases, one in which the received image block has been previously received, and the received image block is repeated, and the other in which the received image block is incomplete or erroneous, and does not belong to one of the image blocks in the image file, for both cases it is necessary to delete the received image block.
Step S330, when a complete image file is needed, an image transmission request is sent to the central storage node, wherein the image transmission request carries a locally stored hash list, and the hash list only records hash values corresponding to missing image blocks; and when the locally stored hash list is empty, merging all the received image blocks into a complete image file.
The edge node independently maintains the locally stored hash list to ensure that only the hash value corresponding to the missing mirror image block is recorded in the locally stored hash list, so that when the complete mirror image file is needed, only a transmission request of the locally missing mirror image block is sent to the central storage node, and the mirror image block is prevented from being repeatedly received.
When the hash list stored locally is empty, it indicates that all the image blocks of the image file have been received completely, and each received image block is checked for integrity and repeatability, where all the received image blocks can be combined into a complete image file.
Step S340, when receiving the mirror image transmission request forwarded by the central storage node, the forwarded mirror image transmission request carries a target edge node identifier and a hash value, and according to the forwarded mirror image transmission request, a mirror image block corresponding to the hash value is transmitted to the target edge node.
As described above, when the central storage node receives the mirror image transmission request of the first edge node, the second edge node with the minimum network delay is determined according to the network quality of the edge nodes in the dynamically monitored set, the mirror image transmission request is forwarded to the second edge node, and once the second edge node receives the mirror image transmission request forwarded by the central storage node, the mirror image block corresponding to the hash value is transmitted to the first edge node according to the first edge node identifier and the hash value carried in the forwarded mirror image transmission request. Therefore, the network bandwidth of the edge node is utilized, the mirror image downloading speed is improved, and the transmission time is shortened.
It should be noted that, the steps S320, S330, and S340 are not sequentially performed, and when the condition of a certain step is satisfied, the corresponding step is executed.
In summary, in the mirror image distribution method in the network element cluster provided by the invention, at the central storage node side, the mirror image file is divided into a plurality of blocks, the hash value of each mirror image block is calculated, all the hash values are constructed into hash lists and sent to all the edge nodes, so that each edge node independently maintains the locally stored hash list, and the received mirror image block is checked according to the hash values, thereby ensuring the reliability of mirror image transmission; by distributing all the mirror blocks to a plurality of different edge nodes according to the network quality of the edge nodes, all the mirror blocks can be rapidly distributed to the plurality of edge nodes, so that the network bandwidth of the plurality of edge nodes is utilized, the mirror downloading speed is improved, and the transmission time is shortened; by recording and updating the mirror image block information owned by each edge node, the mirror image transmission request can be redirected to other edge nodes, and the expandability of mirror image distribution is improved; and by dynamically monitoring the network quality of the edge nodes, the network quality of the selected edge nodes can be ensured to be better when the edge nodes are selected for mirror image transmission, and the success rate of the mirror image transmission is improved.
Therefore, the scheme provided by the embodiments of the invention does not need to carry out complex management on the nodes in the network element cluster, does not have requirements on the number and the scale of the nodes, can well utilize the bandwidth of each edge node to transmit the mirror image under the condition of high load of the central storage node, improves the downloading speed of the mirror image, reduces the transmission time, can improve the expandability of the distribution of the mirror image, and ensures the reliability and the success rate of the transmission of the mirror image.
The method belongs to a technical conception similar to the method for distributing the mirror images in the network element cluster shown in fig. 1, and one embodiment of the invention also provides a device for distributing the mirror images in the network element cluster. Fig. 4 is a schematic structural diagram of a mirror distribution apparatus in a network element cluster according to an embodiment of the present invention, where the mirror distribution apparatus 400 in a network element cluster shown in fig. 4 is applied to a central storage node, and includes:
the network monitoring module 401 is configured to dynamically monitor network quality of all edge nodes;
the image splitting module 402 is configured to split a pre-distributed image file into a plurality of blocks, calculate a hash value of each image block, construct all hash values into a hash list, and send the hash list to all edge nodes for independent maintenance;
A mirror issuing module 403, configured to issue all mirror blocks to a plurality of different edge nodes according to network quality of the edge nodes;
the transmission management module 404 is configured to record mirror block information owned by each edge node after all the mirror blocks are issued; and when receiving a mirror image transmission request of a first edge node, wherein the mirror image transmission request carries a first hash list which is independently maintained by the first edge node, the first hash list only records hash values corresponding to mirror image blocks which are lack by the first edge node, the mirror image transmission request is analyzed to obtain the first hash list, each hash value in the first hash list is searched one by one, a set of edge nodes with corresponding mirror image blocks is searched out, a second edge node with minimum network delay is determined according to the network quality of the edge nodes in the set, the mirror image transmission request is forwarded to the second edge node, the second edge node transmits the corresponding mirror image blocks to the first edge node, and mirror image block information of each edge node is updated after the transmission is completed.
In a preferred embodiment, the network monitoring module 401 is further configured to determine a busy state or an idle state of the edge node according to the network rate, where when the network rate of the edge node is less than the set network rate value, it is determined that the edge node is in the busy state, and otherwise, it is in the idle state;
The transmission management module 404 is further configured to, when receiving a mirror image transmission request of a first edge node, first determine a busy state or an idle state of each edge node in the set for the set of the searched edge nodes having a corresponding mirror image block, and if all the edge nodes in the set are busy states, directly transmit the corresponding mirror image block to the first edge node by the central storage node, and no longer forward the mirror image transmission request to other edge nodes.
In another preferred embodiment, the transmission management module 404 is further configured to, when receiving mirror image transmission requests of a plurality of different edge nodes at the same time, parse and obtain hash lists carried by the mirror image transmission requests, and process the mirror image transmission requests according to the order from the large length to the small length of the hash lists.
The implementation process of each module in the apparatus shown in fig. 4 may refer to the method embodiment shown in fig. 1, which is not described herein.
The method belongs to a technical conception similar to the method for distributing the mirror images in the network element cluster shown in fig. 3, and one embodiment of the invention also provides another device for distributing the mirror images in the network element cluster. Fig. 5 is a schematic structural diagram of a mirror image distribution apparatus in a network element cluster according to another embodiment of the present invention, where a mirror image distribution apparatus 500 in a network element cluster shown in fig. 5 is applied to an edge node, and includes:
A hash list maintenance module 501, configured to receive a hash list sent by a central storage node, where the hash list records a plurality of hash values, where each hash value corresponds to one of all image blocks separated by a pre-distributed image file; when a certain mirror image block is received, calculating a hash value of the mirror image block, comparing the hash value with a locally stored hash list, deleting the corresponding hash value in the hash list if the same hash value exists in the hash list, and deleting the received mirror image block if the same hash value does not exist in the hash list;
the image merging module 502 is configured to send an image transmission request to a central storage node when a complete image file is required, where the image transmission request carries a locally stored hash list, and the hash list only records hash values corresponding to missing image blocks; when the hash list stored locally is empty, combining all the received image blocks into a complete image file;
a mirror image transmission module 503, configured to, when a mirror image transmission request forwarded by a central storage node is received, carry a target edge node identifier and a hash value in the forwarded mirror image transmission request; and transmitting the mirror image block corresponding to the hash value to the target edge node according to the forwarded mirror image transmission request.
The implementation process of each module in the apparatus shown in fig. 5 may refer to the method embodiment shown in fig. 3, which is not described herein.
The method and the device for distributing the mirror images in the network element cluster belong to the same technical conception, and one embodiment of the invention also provides a mirror image distribution system in the network element cluster. Fig. 6 is a schematic structural diagram of a mirror image distribution system in a network element cluster according to an embodiment of the present invention. The mirror image distribution system in the network element cluster shown in fig. 6 includes a central storage node 610 and a plurality of edge nodes 620, where the central storage node 610 includes a network monitoring module, a mirror image splitting module, a mirror image issuing module, and a transmission management module, and is configured to perform the mirror image distribution method shown in fig. 1; each edge node 620 includes a hash list maintenance module, an image merging module, and an image transmission module for performing the image distribution method shown in fig. 3. In particular, the method comprises the steps of,
on the side of central storage node 610: the network monitoring module is used for dynamically monitoring the network quality of all the edge nodes. The image segmentation module is used for dividing the pre-distributed image file into a plurality of blocks, calculating the hash value of each image block, constructing all the hash values into a hash list, and sending the hash list to all the edge nodes for independent maintenance. The mirror image issuing module is used for issuing all mirror image blocks to a plurality of different edge nodes according to the network quality of the edge nodes. The transmission management module is used for recording mirror image block information owned by each edge node after all the mirror image blocks are issued, and when mirror image transmission requests of a first edge node are received, wherein the mirror image transmission requests carry a first hash list which is independently maintained by the first edge node, the first hash list only records hash values corresponding to the mirror image blocks which are lack by the first edge node, the mirror image transmission requests are analyzed to obtain the first hash list, each hash value in the first hash list is searched one by one, a set of edge nodes which have the corresponding mirror image blocks is searched, a second edge node with the minimum network delay is determined according to the network quality of the edge nodes in the set, the mirror image transmission requests are forwarded to the second edge node, the second edge node transmits the corresponding mirror image blocks to the first edge node, and the mirror image block information owned by each edge node is updated after the transmission is completed.
Preferably, the above network monitoring module is further configured to determine a busy state or an idle state of the edge node according to the network rate, where when the network rate of the edge node is less than a set network rate value, it is determined that the edge node is in the busy state, and otherwise, it is in the idle state. The above transmission management module is further configured to, when receiving a mirror image transmission request of a first edge node, first determine a busy state or an idle state of each edge node in the set for the set of the searched edge nodes having a corresponding mirror image block, and if all the edge nodes in the set are busy states, directly transmit the corresponding mirror image block to the first edge node by the central storage node, and no longer forward the mirror image transmission request to other edge nodes.
Preferably, the above transmission management module is further configured to, when receiving mirror image transmission requests of a plurality of different edge nodes at the same time, parse and obtain hash lists carried by the mirror image transmission requests, and process the mirror image transmission requests according to the sequence from the large length to the small length of the hash lists.
On the edge node 620 side: the hash list maintenance module receives a hash list sent by a central storage node, wherein the hash list records a plurality of hash values, and each hash value corresponds to one of all image blocks separated by the pre-distributed image file; and when a certain mirror image block is received, calculating a hash value of the mirror image block, comparing the hash value with a locally stored hash list, deleting the corresponding hash value in the hash list if the same hash value exists in the hash list, and deleting the received mirror image block if the same hash value does not exist in the hash list. The image merging module is used for sending an image transmission request to the central storage node when a complete image file is needed, wherein the image transmission request carries a locally stored hash list, and the hash list only records hash values corresponding to missing image blocks; and when the locally stored hash list is empty, merging all the received image blocks into a complete image file. The mirror image transmission module is used for carrying a target edge node identifier and a hash value in a mirror image transmission request forwarded by the central storage node when the mirror image transmission request is received; and transmitting the mirror image block corresponding to the hash value to the target edge node according to the forwarded mirror image transmission request.
For the implementation process of each module in the system shown in fig. 6, reference may be made to the method embodiments shown in fig. 1 and 3, which are not described herein.
Finally it is pointed out that the term "comprising," "including," or any other variation thereof, is intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises an element.
The foregoing is merely exemplary of the present invention and is not intended to limit the present invention. Various modifications and variations of the present invention will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. which come within the spirit and principles of the invention are to be included in the scope of the claims of the present invention.
Claims (9)
1. A method for distributing images in a network element cluster, wherein the method is applied to a central storage node and comprises the following steps:
Dividing a pre-distributed image file into a plurality of blocks, calculating a hash value of each image block, constructing all hash values into a hash list, and transmitting the hash list to all edge nodes for independent maintenance;
distributing all the mirror blocks to a plurality of different edge nodes according to the network quality of the edge nodes, and recording mirror block information owned by each edge node after the distribution of all the mirror blocks is completed;
when a mirror image transmission request of a first edge node is received, wherein the mirror image transmission request carries a first hash list which is independently maintained by the first edge node, and the first hash list only records hash values corresponding to mirror image blocks which are lack by the first edge node; analyzing the mirror image transmission request to obtain the first hash list, searching a set of edge nodes with corresponding mirror image blocks one by one for each hash value in the first hash list, determining a second edge node with minimum network delay according to the network quality of the edge nodes in the dynamically monitored set, forwarding the mirror image transmission request to the second edge node, transmitting the corresponding mirror image blocks to the first edge node by the second edge node, and updating the mirror image block information owned by each edge node after the transmission is completed.
2. The method according to claim 1, wherein the method further comprises:
judging a busy state or an idle state of the edge node according to the network rate, wherein when the network rate of the edge node is smaller than a set network rate value, the edge node is judged to be in the busy state, otherwise, the edge node is judged to be in the idle state;
when receiving the mirror image transmission request of the first edge node, firstly judging the busy state or idle state of each edge node in the set for the set of the searched edge nodes with the corresponding mirror image blocks, and if all the edge nodes in the set are busy states, directly transmitting the corresponding mirror image blocks to the first edge node by the central storage node, and not forwarding the mirror image transmission request to other edge nodes.
3. The method according to claim 1 or 2, characterized in that the method further comprises:
when mirror image transmission requests of a plurality of different edge nodes are received at the same time, resolving to obtain hash lists carried by the mirror image transmission requests, and processing the mirror image transmission requests according to the sequence from the large length to the small length of the hash lists.
4. A method for distributing images in a network element cluster, wherein the method is applied to an edge node and comprises the following steps:
Receiving a hash list sent by a central storage node, wherein the hash list records a plurality of hash values, and each hash value corresponds to one of all image blocks separated by a pre-distributed image file;
when a certain mirror image block is received, calculating a hash value of the mirror image block, comparing the hash value with a locally stored hash list, deleting the corresponding hash value in the hash list if the same hash value exists in the hash list, and deleting the received mirror image block if the same hash value does not exist in the hash list;
when a complete mirror image file is needed, sending a mirror image transmission request to the central storage node, wherein the mirror image transmission request carries a locally stored hash list, and the hash list only records hash values corresponding to missing mirror image blocks; when the hash list stored locally is empty, combining all the received image blocks into a complete image file; and
when receiving the mirror image transmission request forwarded by the central storage node, the forwarded mirror image transmission request carries a target edge node identifier and a hash value, and a mirror image block corresponding to the hash value is transmitted to the target edge node according to the forwarded mirror image transmission request.
5. An image distribution apparatus in a network element cluster, wherein the apparatus is applied to a central storage node, and comprises:
the network monitoring module is used for dynamically monitoring the network quality of all the edge nodes;
the image segmentation module is used for dividing the pre-distributed image file into a plurality of blocks, calculating the hash value of each image block, constructing all the hash values into a hash list, and transmitting the hash list to all edge nodes for independent maintenance;
the mirror image issuing module is used for issuing all mirror image blocks to a plurality of different edge nodes according to the network quality of the edge nodes;
the transmission management module is used for recording mirror image block information owned by each edge node after all the mirror image blocks are issued, and when a mirror image transmission request of a first edge node is received, wherein the mirror image transmission request carries a first hash list which is independently maintained by the first edge node, the first hash list only records hash values corresponding to the mirror image blocks which are lack by the first edge node, the mirror image transmission request is analyzed to obtain the first hash list, each hash value in the first hash list is searched one by one, a set of edge nodes which have corresponding mirror image blocks is searched, a second edge node with minimum network delay is determined according to the network quality of the edge nodes in the set, the mirror image transmission request is forwarded to the second edge node, the second edge node transmits corresponding mirror image blocks to the first edge node, and the mirror image block information owned by each edge node is updated after the transmission is completed.
6. The apparatus of claim 5, wherein the device comprises a plurality of sensors,
the network monitoring module is further configured to determine a busy state or an idle state of the edge node according to the network rate, where when the network rate of the edge node is less than a set network rate value, it is determined that the edge node is in the busy state, and if not, it is in the idle state;
the transmission management module is further configured to, when receiving a mirror image transmission request of a first edge node, first determine a busy state or an idle state of each edge node in the set for the set of the searched edge nodes having a corresponding mirror image block, and if all the edge nodes in the set are busy states, directly transmit the corresponding mirror image block to the first edge node by the central storage node, and no longer forward the mirror image transmission request to other edge nodes.
7. The apparatus according to claim 5 or 6, wherein the transmission management module is further configured to, when receiving mirror image transmission requests of a plurality of different edge nodes at the same time, parse to obtain hash lists carried by the mirror image transmission requests, and process the mirror image transmission requests according to a sequence from a length of the hash lists to a length of the hash lists.
8. An image distribution apparatus in a network element cluster, wherein the apparatus is applied to an edge node, and comprises:
the hash list maintenance module is used for receiving a hash list sent by the central storage node, wherein the hash list records a plurality of hash values, and each hash value corresponds to one of all image blocks separated by the pre-distributed image file; when a certain mirror image block is received, calculating a hash value of the mirror image block, comparing the hash value with a locally stored hash list, deleting the corresponding hash value in the hash list if the same hash value exists in the hash list, and deleting the received mirror image block if the same hash value does not exist in the hash list;
the mirror image merging module is used for sending a mirror image transmission request to the central storage node when a complete mirror image file is needed, wherein the mirror image transmission request carries a locally stored hash list, and the hash list only records hash values corresponding to missing mirror image blocks; when the hash list stored locally is empty, combining all the received image blocks into a complete image file; the method comprises the steps of,
The mirror image transmission module is used for carrying a target edge node identifier and a hash value in a mirror image transmission request forwarded by the central storage node when the mirror image transmission request is received; and transmitting the mirror image block corresponding to the hash value to the target edge node according to the forwarded mirror image transmission request.
9. A mirror distribution system in a network element cluster, said system comprising a central storage node and a number of edge nodes, characterized in that,
the central storage node comprises a network monitoring module, a mirror image segmentation module, a mirror image issuing module and a transmission management module, and is used for executing the mirror image distribution method according to any one of claims 1-3;
each of the edge nodes includes a hash list maintenance module, an image merging module, and an image transmission module for executing the image distribution method of claim 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310823182.1A CN116684416A (en) | 2023-07-05 | 2023-07-05 | Mirror image distribution method, device and system in network element cluster |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310823182.1A CN116684416A (en) | 2023-07-05 | 2023-07-05 | Mirror image distribution method, device and system in network element cluster |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116684416A true CN116684416A (en) | 2023-09-01 |
Family
ID=87781057
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310823182.1A Pending CN116684416A (en) | 2023-07-05 | 2023-07-05 | Mirror image distribution method, device and system in network element cluster |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116684416A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117354374A (en) * | 2023-12-06 | 2024-01-05 | 广东车卫士信息科技有限公司 | Data transmission method and system based on Internet of things |
-
2023
- 2023-07-05 CN CN202310823182.1A patent/CN116684416A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117354374A (en) * | 2023-12-06 | 2024-01-05 | 广东车卫士信息科技有限公司 | Data transmission method and system based on Internet of things |
| CN117354374B (en) * | 2023-12-06 | 2024-02-13 | 广东车卫士信息科技有限公司 | Data transmission method and system based on Internet of Things |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR101585146B1 (en) | Distribution storage system of distributively storing objects based on position of plural data nodes, position-based object distributive storing method thereof, and computer-readable recording medium | |
| US8010488B2 (en) | Information distribution system, information processing device and memory medium | |
| CN111200657B (en) | Method for managing resource state information and resource downloading system | |
| CN108881354B (en) | Push information storage method and device, server and computer storage medium | |
| US9614926B2 (en) | Parallel I/O write processing for use in clustered file systems having cache storage | |
| KR20120018178A (en) | Swarm-based synchronization over a network of object stores | |
| CN111447248A (en) | File transmission method and device | |
| CN109639773B (en) | Dynamically constructed distributed data cluster control system and method thereof | |
| US20100250674A1 (en) | Content delivery apparatus, content delivery method, and content delivery program | |
| US11265182B2 (en) | Messaging to enforce operation serialization for consistency of a distributed data structure | |
| US20020156931A1 (en) | Remote file system using network multicast | |
| CN105868251A (en) | Cache data updating method and device | |
| CN116684416A (en) | Mirror image distribution method, device and system in network element cluster | |
| CN101848188A (en) | Method and device for updating data block checking information | |
| US10757171B1 (en) | Merge trees for collaboration | |
| CN113411376A (en) | Sensor data processing method and device based on block chain fragmentation storage | |
| KR20180073128A (en) | A data updating method based on data block comparison | |
| JP2004199578A (en) | Content delivery method, device, program, and storage medium | |
| CN111405021A (en) | Data transmission method, device, equipment and storage medium of peer-to-peer node | |
| CN113268540B (en) | Data synchronization method and device | |
| US8799212B2 (en) | Repository synchronization in a ranked repository cluster | |
| US8250140B2 (en) | Enabling connections for use with a network | |
| CN107465706B (en) | Distributed data object storage device based on wireless communication network | |
| US9015371B1 (en) | Method to discover multiple paths to disk devices cluster wide | |
| CN110581873B (en) | Cross-cluster redirection method and monitoring server |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |