CN111142799A - Distributed storage method and device, network node and storage medium - Google Patents
Distributed storage method and device, network node and storage medium Download PDFInfo
- Publication number
- CN111142799A CN111142799A CN201911368709.6A CN201911368709A CN111142799A CN 111142799 A CN111142799 A CN 111142799A CN 201911368709 A CN201911368709 A CN 201911368709A CN 111142799 A CN111142799 A CN 111142799A
- Authority
- CN
- China
- Prior art keywords
- time window
- storage
- network
- network node
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention provides a distributed storage method, which is applied to network nodes of a distributed storage network and comprises the following steps: receiving a storage request for specified data; establishing a time window array according to a preset storage period; determining a time window corresponding to the specified data from the time window array, and inserting the storage task of the specified data into the determined time window; and executing the storage task of the designated data according to the preset storage period and the determined time window, determining a target node from the distributed storage network, and storing the designated data to the target node. The invention also provides a distributed storage device, a network node and a storage medium. The invention can enable a single network node to execute the storage task uniformly.
Description
Technical Field
The invention relates to the technical field of computer networks, in particular to a distributed storage method and device, a network node and a computer readable storage medium.
Background
The distributed storage network is used for storing data on a plurality of independent devices in a distributed mode. Each data corresponds to a number of storage devices (i.e., network nodes), and multiple data may be stored simultaneously on a single storage device. A single network node in an existing Distributed storage network, such as a DHT (Distributed Hash Table) network, may receive a large number of storage requests in a short time. If the network node directly stores data to the storage network and then periodically stores the data again, the problems of short-time overhigh CPU load, network packet loss and the like of the network node can be caused, and the realization of the storage function of the network node is influenced.
Disclosure of Invention
In view of the above, there is a need for a distributed storage method, a distributed storage apparatus, a network node, a computer-readable storage medium, and a computer program product, which can enable a single network node to perform storage tasks evenly.
A first aspect of the present application provides a distributed storage method, which is applied to a network node of a distributed storage network, and the method includes:
receiving a storage request for specified data;
establishing a time window array according to a preset storage period;
determining a time window corresponding to the specified data from the time window array, and inserting the storage task of the specified data into the determined time window;
and executing the storage task of the designated data according to the preset storage period and the determined time window, determining a target node from the distributed storage network, and storing the designated data to the target node.
In another possible implementation manner, the determining, from the time window array, a time window corresponding to the specified data includes:
determining a starting time window from the time window array;
selecting a time window from the starting time window, and judging whether the storage task of the selected time window meets a preset condition or not;
and if the storage task of the selected time window meets the preset condition, adding the storage task of the specified data into the selected time window.
In another possible implementation manner, the determining whether the storage task of the selected time window satisfies a preset condition includes:
and judging whether the storage task number or the weighted storage task number of the selected time window is smaller than the sum of the average storage task number of the time window array and a preset constant.
In another possible implementation manner, the determining a starting time window from the time window array includes:
calculating the time difference seconds between the current time of the network node and the preset initial time;
and performing modulo operation on the length of the time window array by using the time difference seconds to obtain the position of the starting time window in the time window array.
In another possible implementation manner, the determining a target node from the distributed storage network includes:
calculating the distance between each network node and the designated data according to the node ID of each network node of the distributed storage network and the identification information of the designated data;
and determining a specified number of network nodes with the minimum distance to the specified data as the target nodes.
In another possible implementation manner, the calculating, according to the node ID of each network node of the distributed storage network and the identification information of the designated data, a distance between each network node and the designated data includes:
and carrying out XOR operation on the node ID of each network node and the identification information of the designated data to obtain the distance between each network node and the designated data.
In another possible implementation manner, the determining a target node from the distributed storage network includes:
selecting candidate nodes according to the region and the operator to which the requester of the storage request belongs and the region and the operator to which each network node belongs;
judging whether the candidate node meets a screening condition;
and if the candidate node meets the screening condition, taking the candidate node as the target node.
A second aspect of the application provides a network node comprising a memory and a processor, the memory having stored thereon a computer program executable on the processor, the computer program, when executed by the processor, implementing the distributed storage method.
A third aspect of the present application provides a distributed storage apparatus, applied to a network node of a distributed storage network, the apparatus including:
a receiving unit configured to receive a storage request of specified data;
the establishing unit is used for establishing a time window array according to a preset storage period;
the inserting unit is used for determining a time window corresponding to the specified data from the time window array and inserting the storage task of the specified data into the determined time window;
and the execution unit is used for executing the storage task of the specified data according to the preset storage period and the determined time window, determining a target node from the distributed storage network, and storing the specified data to the target node.
A fourth aspect of the present application provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the distributed storage method.
A fifth aspect of the application provides a computer program product comprising computer instructions which, when run on a network node, cause the network node to perform the distributed storage method.
The invention receives a storage request of specified data; establishing a time window array according to a preset storage period; determining a time window corresponding to the specified data from the time window array, and inserting the storage task of the specified data into the determined time window; and executing the storage task of the designated data according to the preset storage period and the determined time window, determining a target node from the distributed storage network, and storing the designated data to the target node. The invention can enable a single network node to execute the storage task in a balanced manner, avoids the problems of data loss or storage failure and the like caused by overhigh load in a short time of the network node, improves the quantity of the data stored by the network node, and ensures the realization of the storage function of the network node.
Drawings
Fig. 1 is a flowchart of a distributed storage method according to an embodiment of the present invention.
Fig. 2 is a flowchart of a distributed storage apparatus according to an embodiment of the present invention.
Fig. 3 is a schematic diagram of a network node according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The terms "first," "second," "third," "fourth," and the like in the description and in the claims of the present application and in the drawings described above, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that the embodiments described herein may be practiced otherwise than as specifically illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, apparatus, article, or device that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or device.
It should be noted that the description relating to "first", "second", etc. in the present invention is for descriptive purposes only and is not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In addition, technical solutions between various embodiments may be combined with each other, but must be realized by a person skilled in the art, and when the technical solutions are contradictory or cannot be realized, such a combination should not be considered to exist, and is not within the protection scope of the present invention.
Preferably, the distributed storage method of the present invention is applied in one or more network nodes. The network node is a device capable of automatically performing numerical calculation and/or information processing according to a preset or stored instruction, and the hardware includes, but is not limited to, a microprocessor, an Application Specific Integrated Circuit (ASIC), a Programmable Gate Array (FPGA), a Digital Signal Processor (DSP), an embedded device, and the like.
The network node may be a desktop computer, a notebook, a palm computer, a cloud server, or other computing device. The network node can carry out man-machine interaction with a user through a keyboard, a mouse, a remote controller, a touch panel or voice control equipment and the like.
The Network node may be a node constituting a CDN (Content Delivery Network) Network or a blockchain Network.
Example one
Fig. 1 is a flowchart of a distributed storage method according to an embodiment of the present invention. The distributed storage method is applied to network nodes of a distributed storage network. The distributed storage network includes a plurality of network nodes, each network node storing a plurality of data.
In this embodiment, the Distributed storage network may be a Distributed Hash Table (DHT) network. The data stored by the network node is a key value pair (key value). The key-value pair includes a key value (i.e., a key value) of the data, which may be a hash value of the data, and a value, which may be an index of the data.
In other embodiments, the distributed storage network may be other types of networks. The data stored by the network node may be different types of data, such as video data, audio data, text data, etc. In this embodiment, the network node may be a personal cloud disk, such as a guest-playing cloud.
The distributed storage method generates a storage task of specified data according to a received storage request, periodically executes the storage task, and stores the specified data to a target node in a distributed storage network. The distributed storage method can enable a single network node to execute storage tasks in a balanced manner, avoids the problems of data loss or storage failure and the like caused by overhigh load in a short time of the network node, improves the quantity of the data stored by the network node, and ensures the realization of the storage function of the network node.
Referring to fig. 1, the distributed storage method specifically includes the following steps:
101, a storage request specifying data is received.
The network node may receive storage requests sent by different requesters. The requesting party includes, but is not limited to, a desktop computer, a laptop computer, a personal digital assistant, a tablet computer, a personal cloud disk, a smart phone, an electronic book reader, an MP3(Moving Picture Experts Group Audio Layer III, motion Picture Experts compression standard Audio Layer 3) or MP4(Moving Picture Experts Group Audio Layer IV, motion Picture Experts compression standard Audio Layer 4) player, a POS terminal, a vehicle-mounted computer, and the like.
The network node may receive storage requests issued by other network nodes within the distributed storage network.
Alternatively, the network node may receive storage requests from other computing devices outside the distributed storage network.
The specified data may be a key-value pair. Alternatively, the specified data may be various types of data, and may be, for example, video data, audio data, text data, or the like.
And 102, establishing a time window array according to a preset storage period.
The time window array is used for counting the storage tasks required to be executed in each time period in one storage cycle. In one embodiment, a time window records the storage tasks that need to be performed within one second. Therefore, the number of seconds included in one memory cycle is the length of the time window array, i.e., the total number of time windows. For example, if the storage period is 10 minutes, the length of the time window array is 60 × 10 — 600.
In other embodiments, a time window may record storage tasks that need to be performed for other time periods (e.g., 100 milliseconds).
103, determining a time window corresponding to the specified data from the time window array, and inserting the storage task of the specified data into the determined time window.
In one embodiment, the determining the time window corresponding to the specified data from the time window array includes:
(1) determining a starting time window from the time window array;
(2) selecting a time window from the starting time window, and judging whether the storage task of the selected time window meets a preset condition or not;
(3) and if the storage task of the selected time window meets the preset condition, adding the storage task of the specified data into the selected time window.
And if the storage task of the selected time window does not meet the preset condition, returning to the step (2), and selecting the next time window from the time window array and judging.
In an embodiment, if the last time window of the time window array is selected and the storage task of the last time window does not meet a preset condition, the next time window is selected from the first time window of the time window array. The maximum number of times the time window is selected is equal to the length of the time window array.
In an embodiment, the determining whether the storage task of the selected time window satisfies a preset condition includes:
and judging whether the storage task number or the weighted storage task number of the selected time window is smaller than the sum of the average storage task number of the time window array and a preset constant.
And if the number of the storage tasks or the weighted storage tasks of the selected time window is less than the sum of the average number of the storage tasks and a preset constant of the time window array, the storage tasks of the selected time window meet a preset condition. Otherwise, if the number of the storage tasks or the weighted storage tasks of the selected time window is greater than or equal to the sum of the average number of the storage tasks of the time window array and a preset constant, the storage tasks of the selected time window do not meet the preset condition.
The preset constant may be 2 or 3.
The weighted storage task number of the selected time window may be calculated according to the historical execution condition, data size, task priority, etc. of the storage tasks of the selected time window. And if the historical execution speed of a storage task is higher, the data is smaller, and the task priority is lower, the weight of the storage task is smaller. Conversely, if the historical execution speed of a storage task is slower, the data is larger, and the task priority is higher, the weight of the storage task is larger.
In one embodiment, the determining a starting time window from the time window array comprises:
calculating the time difference seconds between the current time of the network node and the preset initial time;
and performing modulo operation on the length of the time window array by using the time difference seconds to obtain the position of the starting time window in the time window array.
For example, if the length of the time window array is modulo 5 by the time difference in seconds, the starting time window is the 5 th time window in the time window array.
The preset initial time may be 1970, month 1, day 0, 0 minutes 0 seconds.
In another embodiment, said determining a starting time window from said array of time windows comprises:
and randomly selecting a time window from the time window array as the starting time window.
In an embodiment, the start execution time of the storage task of the specified data is t + h + r, where t is the current time of the network node, h is the position of the start time window in the time window array (e.g., h is 5, the 5 th time window representing the time window array is the start time window), and r is a random time (e.g., 10 ms) within a range of one second.
104, executing the storage task of the designated data according to the preset storage period and the determined time window, determining a target node from the distributed storage network, and storing the designated data to the target node.
In one embodiment, said storing said specified data to said target node comprises:
dividing the specified data into data blocks;
and storing the data block to the target node.
Whether the size of the specified data is larger than a standard data block size can be judged; and if the size of the specified data is larger than the size of a standard data block, segmenting the specified data according to the size of the standard data block.
In one embodiment, the determining the target node from the distributed storage network comprises:
calculating the distance between each network node and the designated data according to the node ID of each network node of the distributed storage network and the identification information of the designated data;
and determining a specified number of network nodes with the minimum distance to the specified data as the target nodes.
In an embodiment, the calculating the distance between each network node and the designated data according to the node ID of each network node of the distributed storage network and the identification information of the designated data includes:
and carrying out XOR operation on the node ID of each network node and the identification information of the designated data to obtain the distance between each network node and the designated data.
In an embodiment, the identification information of the specified data may be a hash value of the specified data.
In an embodiment, the specified data is a key-value pair, and the identification information of the specified data is a key value (key value) of the key-value pair (key value).
The node ID of a network node may be assigned when the network node joins the distributed storage network.
In another embodiment, the determining a target node from the distributed storage network comprises:
selecting candidate nodes according to the region and the operator to which the requester of the storage request belongs and the region and the operator to which each network node belongs;
judging whether the candidate node meets a screening condition;
and if the candidate node meets the screening condition, taking the candidate node as the target node.
The storage priority policy of the network nodes can be preset, and the candidate nodes are selected from the network nodes according to the sequence of the storage priorities from top to bottom and are segmented. For example, a network node belonging to the same operator in the same area as the requester may be set to have a first storage priority (i.e., the highest storage priority), a network node belonging to the same operator in a different area from the requester to have a second storage priority, a network node belonging to a different operator in the same area as the requester to have a third storage priority, and a network node belonging to a different operator in a different area from the requester to have a fourth storage priority (i.e., the lowest storage priority). The method comprises the steps of firstly selecting candidate nodes from network nodes with a first storage priority and judging, if the network nodes with the first storage priority are selected and a sufficient number of target nodes are not obtained, selecting the candidate nodes from network nodes with a second storage priority and judging, and so on.
The area and the operator to which the requester belongs can be obtained according to the IP address of the requester.
In a particular embodiment, the regions may include six geographic regions, "east of china," north of china, "" south of china, "" north of east, "" north of west, "and" south of west. The operators can comprise five network operators of China telecom, China Unicom, China Mobile, China Internet Community and China iron Tong. It is understood that the area may also be a province, a city or a county.
In the embodiment of the invention, each operator has the corresponding IP address field, and the operator can allocate the IP address in the corresponding IP address field to the requester accessing the operator network, so that the requester can perform data interaction with the service server in the operator network by using the IP address allocated by the operator.
The IP address field of each operator may be looked up, and the operator of the address field containing the IP address of the requesting party is determined as the operator to which the requesting party belongs. For example: 111.0.0.0/10 is the IP address field of China Mobile, the IP address of the requester is 111.0.0.7, 111.0.0.0/10 includes 111.0.0.7, then the carrier to which the requester belongs is China Mobile.
Similarly, the region and operator to which the requester belongs may be obtained from the IP address of the network node.
The screening conditions can be set according to actual needs.
The screening condition may be a combination of a plurality of conditions.
The screening condition may include that the storage heat degree of the network node is a first preset heat degree or a first preset heat degree range. For example, the first condition may include a thermal storage of the network node being less than or equal to a first preset degree of heat (e.g., 20). The more times a network node stores data as a target node, the higher the storage heat of the network node. For example, each time a network node stores data as a target node, the storage heat of the network node is increased by 1.
The screening condition may further include that the network type of the network node is a preset network type. For example, the requestor and the network node may include a first network type, a second network type, a third network type, a fourth network type, a fifth network type. If the requester is the first network type, the screening condition may include that the network type of the network node is a fourth network type or a fifth network type; if the requester is the fifth network type, the screening condition may include that the network type of the network node is the first network type, the second network type, the third network type, or the fourth network type.
The screening condition may further include that the network node supports a preset protocol. For example, the screening condition may include that the network node supports the TCP protocol. Alternatively, the screening condition may include that the network node supports an external Socket.
A single network node in an existing distributed storage network may receive a large number of storage requests in a short time, and if the network node directly stores data to the storage network and then periodically stores the data again, the network node may have problems of short-time CPU overload, network packet loss and the like, which affect the implementation of the storage function of the network node. The distributed storage method receives a storage request of specified data; establishing a time window array according to a preset storage period; determining a time window corresponding to the specified data from the time window array, and inserting the storage task of the specified data into the determined time window; and executing the storage task of the designated data according to the preset storage period and the determined time window, determining a target node from the distributed storage network, and storing the designated data to the target node. According to the distributed storage method, the time window arrays are established for the single network node, and the storage tasks of the network node are inserted into different time windows, so that the network node can execute the storage tasks in a balanced manner, the problems that the short-time load of the network node is too high, the data loss or the storage failure is caused and the like are solved, the quantity of the stored data of the network node is increased, and the storage function of the network node is guaranteed to be realized.
In another embodiment, after receiving the storage request for the specified data, the method further comprises:
storing the specified data in a non-volatile storage device of the network node.
In another embodiment, the method further comprises:
if the network node is restarted, reading the specified data from the nonvolatile storage equipment;
and executing the storage task on the read specified data according to the preset storage period and the determined time window.
In another embodiment, the method further comprises:
judging whether the execution times of the storage tasks of the specified data reach preset times or not;
and if the execution times of the storage tasks of the specified data reach preset times, deleting the specified data from the nonvolatile storage equipment.
Example two
Fig. 2 is a structural diagram of a distributed storage apparatus according to an embodiment of the present invention. The distributed storage apparatus 20 is in a network node of a distributed storage network. The distributed storage network includes a plurality of network nodes, each network node storing a plurality of data.
As shown in fig. 2, the distributed storage apparatus 20 may include: a receiving unit 201, a establishing unit 202, an inserting unit 203, and an executing unit 204.
A receiving unit 201, configured to receive a storage request of specified data.
The network node may receive storage requests sent by different requesters. The requesting party includes, but is not limited to, a desktop computer, a laptop computer, a personal digital assistant, a tablet computer, a personal cloud disk, a smart phone, an electronic book reader, an MP3(Moving Picture Experts Group Audio Layer III, motion Picture Experts compression standard Audio Layer 3) or MP4(Moving Picture Experts Group Audio Layer IV, motion Picture Experts compression standard Audio Layer 4) player, a POS terminal, a vehicle-mounted computer, and the like.
The network node may receive storage requests issued by other network nodes within the distributed storage network.
Alternatively, the network node may receive storage requests from other computing devices outside the distributed storage network.
The specified data may be various types of data, and may be, for example, video data, audio data, text data, or the like.
The establishing unit 202 is configured to establish a time window array according to a preset storage period.
The time window array is used for counting the storage tasks required to be executed in each time period in one storage cycle. In one embodiment, a time window records the storage tasks that need to be performed within one second. Therefore, the number of seconds included in one memory cycle is the length of the time window array, i.e., the total number of time windows. For example, if the storage period is 10 minutes, the length of the time window array is 60 × 10 — 600.
In other embodiments, a time window may record storage tasks that need to be performed for other time periods (e.g., 100 milliseconds).
An inserting unit 203, configured to determine a time window corresponding to the specified data from the time window array, and insert the storage task of the specified data into the determined time window.
In one embodiment, the determining the time window corresponding to the specified data from the time window array includes:
(1) determining a starting time window from the time window array;
(2) selecting a time window from the starting time window, and judging whether the storage task of the selected time window meets a preset condition or not;
(3) and if the storage task of the selected time window meets the preset condition, adding the storage task of the specified data into the selected time window.
And if the storage task of the selected time window does not meet the preset condition, returning to the step (2), and selecting the next time window from the time window array and judging.
In an embodiment, if the last time window of the time window array is selected and the storage task of the last time window does not meet a preset condition, the next time window is selected from the first time window of the time window array. The maximum number of times the time window is selected is equal to the length of the time window array.
In an embodiment, the determining whether the storage task of the selected time window satisfies a preset condition includes:
and judging whether the storage task number or the weighted storage task number of the selected time window is smaller than the sum of the average storage task number of the time window array and a preset constant.
And if the number of the storage tasks or the weighted storage tasks of the selected time window is less than the sum of the average number of the storage tasks and a preset constant of the time window array, the storage tasks of the selected time window meet a preset condition. Otherwise, if the number of the storage tasks or the weighted storage tasks of the selected time window is greater than or equal to the sum of the average number of the storage tasks of the time window array and a preset constant, the storage tasks of the selected time window do not meet the preset condition.
The preset constant may be 2 or 3.
The weighted storage task number of the selected time window may be calculated according to the historical execution condition, data size, task priority, etc. of the storage tasks of the selected time window. And if the historical execution speed of a storage task is higher, the data is smaller, and the task priority is lower, the weight of the storage task is smaller. Conversely, if the historical execution speed of a storage task is slower, the data is larger, and the task priority is higher, the weight of the storage task is larger.
In one embodiment, the determining a starting time window from the time window array comprises:
calculating the time difference seconds between the current time of the network node and the preset initial time;
and performing modulo operation on the length of the time window array by using the time difference seconds to obtain the position of the starting time window in the time window array.
For example, if the length of the time window array is modulo 5 by the time difference in seconds, the starting time window is the 5 th time window in the time window array.
The preset initial time may be 1970, month 1, day 0, 0 minutes 0 seconds.
In another embodiment, said determining a starting time window from said array of time windows comprises:
and randomly selecting a time window from the time window array as the starting time window.
In an embodiment, the start execution time of the storage task of the specified data is t + h + r, where t is the current time of the network node, h is the position of the start time window in the time window array (e.g., h is 5, the 5 th time window representing the time window array is the start time window), and r is a random time (e.g., 10 ms) within a range of one second.
An executing unit 204, configured to execute a storage task of the specified data according to the preset storage period and the determined time window, determine a target node from the distributed storage network, and store the specified data in the target node.
In one embodiment, said storing said specified data to said target node comprises:
dividing the specified data into data blocks;
and storing the data block to the target node.
Whether the size of the specified data is larger than a standard data block size can be judged; and if the size of the specified data is larger than the size of a standard data block, segmenting the specified data according to the size of the standard data block.
In one embodiment, the determining the target node from the distributed storage network comprises:
calculating the distance between each network node and the designated data according to the node ID of each network node of the distributed storage network and the identification information of the designated data;
and determining a specified number of network nodes with the minimum distance to the specified data as the target nodes.
In an embodiment, the calculating the distance between each network node and the designated data according to the node ID of each network node of the distributed storage network and the identification information of the designated data includes:
and carrying out XOR operation on the node ID of each network node and the identification information of the designated data to obtain the distance between each network node and the designated data.
In an embodiment, the identification information of the specified data may be a hash value of the specified data.
In an embodiment, the specified data is a key-value pair, and the identification information of the specified data is a key value (key value) of the key-value pair (key value).
The node ID of a network node may be assigned when the network node joins the distributed storage network.
In another embodiment, the determining a target node from the distributed storage network comprises:
selecting candidate nodes according to the region and the operator to which the requester of the storage request belongs and the region and the operator to which each network node belongs;
judging whether the candidate node meets a screening condition;
and if the candidate node meets the screening condition, taking the candidate node as the target node.
The storage priority policy of the network nodes can be preset, and the candidate nodes are selected from the network nodes according to the sequence of the storage priorities from top to bottom and are segmented. For example, a network node belonging to the same operator in the same area as the requester may be set to have a first storage priority (i.e., the highest storage priority), a network node belonging to the same operator in a different area from the requester to have a second storage priority, a network node belonging to a different operator in the same area as the requester to have a third storage priority, and a network node belonging to a different operator in a different area from the requester to have a fourth storage priority (i.e., the lowest storage priority). The method comprises the steps of firstly selecting candidate nodes from network nodes with a first storage priority and judging, if the network nodes with the first storage priority are selected and a sufficient number of target nodes are not obtained, selecting the candidate nodes from network nodes with a second storage priority and judging, and so on.
The area and the operator to which the requester belongs can be obtained according to the IP address of the requester.
In a particular embodiment, the regions may include six geographic regions, "east of china," north of china, "" south of china, "" north of east, "" north of west, "and" south of west. The operators can comprise five network operators of China telecom, China Unicom, China Mobile, China Internet Community and China iron Tong. It is understood that the area may also be a province, a city or a county.
In the embodiment of the invention, each operator has the corresponding IP address field, and the operator can allocate the IP address in the corresponding IP address field to the requester accessing the operator network, so that the requester can perform data interaction with the service server in the operator network by using the IP address allocated by the operator.
The IP address field of each operator may be looked up, and the operator of the address field containing the IP address of the requesting party is determined as the operator to which the requesting party belongs. For example: 111.0.0.0/10 is the IP address field of China Mobile, the IP address of the requester is 111.0.0.7, 111.0.0.0/10 includes 111.0.0.7, then the carrier to which the requester belongs is China Mobile.
Similarly, the region and operator to which the requester belongs may be obtained from the IP address of the network node.
The screening conditions can be set according to actual needs.
The screening condition may be a combination of a plurality of conditions.
The screening condition may include that the storage heat degree of the network node is a first preset heat degree or a first preset heat degree range. For example, the first condition may include a thermal storage of the network node being less than or equal to a first preset degree of heat (e.g., 20). The more times a network node stores data as a target node, the higher the storage heat of the network node. For example, each time a network node stores data as a target node, the storage heat of the network node is increased by 1.
The screening condition may further include that the network type of the network node is a preset network type. For example, the requestor and the network node may include a first network type, a second network type, a third network type, a fourth network type, a fifth network type. If the requester is the first network type, the screening condition may include that the network type of the network node is a fourth network type or a fifth network type; if the requester is the fifth network type, the screening condition may include that the network type of the network node is the first network type, the second network type, the third network type, or the fourth network type.
The screening condition may further include that the network node supports a preset protocol. For example, the screening condition may include that the network node supports the TCP protocol. Alternatively, the screening condition may include that the network node supports an external Socket.
A single network node in an existing distributed storage network may receive a large number of storage requests in a short time, and if the network node directly stores data to the storage network and then periodically stores the data again, the network node may have problems of short-time CPU overload, network packet loss and the like, which affect the implementation of the storage function of the network node. The distributed storage method receives a storage request of specified data; establishing a time window array according to a preset storage period; determining a time window corresponding to the specified data from the time window array, and inserting the storage task of the specified data into the determined time window; and executing the storage task of the designated data according to the preset storage period and the determined time window, determining a target node from the distributed storage network, and storing the designated data to the target node. The distributed storage device 20 establishes a time window array for a single network node, and inserts the storage tasks of the network node into different time windows, so that the network node executes the storage tasks in a balanced manner, the problems of data loss or storage failure and the like caused by short-time overhigh load of the network node are avoided, the quantity of the storage data of the network node is increased, and the realization of the storage function of the network node is ensured.
In another embodiment, the distributed storage apparatus 20 further includes:
and the storage unit is used for storing the specified data into the nonvolatile storage equipment of the network node after receiving the storage request of the specified data.
In another embodiment, the execution unit 204 is further configured to:
if the network node is restarted, reading the specified data from the nonvolatile storage equipment;
and executing the storage task on the read specified data according to the preset storage period and the determined time window.
In another embodiment, the distributed storage apparatus 20 further includes:
a deleting unit, configured to determine whether the execution frequency of the storage task of the specified data reaches a preset frequency; and if the execution times of the storage tasks of the specified data reach preset times, deleting the specified data from the nonvolatile storage equipment.
EXAMPLE III
The present embodiment provides a computer-readable storage medium, which stores thereon a computer program, and when the computer program is executed by a processor, the computer program implements the steps in the above-mentioned distributed storage method embodiment, for example, 101-104 shown in fig. 1:
101, receiving a storage request of specified data;
102, establishing a time window array according to a preset storage period;
103, determining a time window corresponding to the specified data from the time window array, and inserting the storage task of the specified data into the determined time window;
104, executing the storage task of the designated data according to the preset storage period and the determined time window, determining a target node from the distributed storage network, and storing the designated data to the target node.
Alternatively, the computer program, when executed by the processor, implements the functions of the modules/units in the above-mentioned device embodiments, for example, the unit 201 and 204 in fig. 2:
a receiving unit 201 for receiving a storage request of specified data;
the establishing unit 202 is configured to establish a time window array according to a preset storage period;
the inserting unit 203 is used for determining a time window corresponding to the specified data from the time window array and inserting the storage task of the specified data into the determined time window;
an executing unit 204, configured to execute a storage task of the specified data according to the preset storage period and the determined time window, determine a target node from the distributed storage network, and store the specified data in the target node.
EXAMPLE III
Fig. 3 is a schematic diagram of a network node according to an embodiment of the present invention. The network node 3 comprises a memory 30, a processor 31, a bus 33 and a computer program 32 stored in the memory 30 and executable on the processor 31. The processor 31 implements the steps in the above-described distributed storage method embodiment when executing the computer program 32, such as 101-104 shown in fig. 1.
Alternatively, the processor 31, when executing the computer program 32, implements the functions of the modules/units in the above-mentioned device embodiments, such as the units 201 and 204 in fig. 2.
Illustratively, the computer program 32 may be partitioned into one or more modules/units that are stored in the memory 30 and executed by the processor 31 to carry out the invention. The one or more modules/units may be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution of the computer program 32 in the network node 3.
The network node 3 may be a PC (Personal Computer), or may also be a terminal device such as a smart phone, a tablet Computer, a palmtop Computer, a portable Computer, an intelligent router, an ore machine, a network storage device, and the like.
The Processor 31 may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic device, discrete hardware component, or the like. A general purpose processor may be a microprocessor or the processor 31 may be any conventional processor or the like, said processor 31 being the control center of said network node 3, the various parts of the entire network node 3 being connected by means of various interfaces and lines.
The memory 30 may be used to store the computer program 32 and/or the module/unit, and the processor 31 implements various functions of the network node 3 by running or executing the computer program and/or the module/unit stored in the memory 30 and by invoking data stored in the memory 30. The memory 30 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data created according to the use of the network node 3. In addition, the memory 30 may include a non-volatile memory such as a hard disk, a memory, a plug-in hard disk, a Smart Memory Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), at least one magnetic disk storage device, a Flash memory device.
The bus 33 may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus 33 may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown in FIG. 3, but this does not mean only one bus or one type of bus.
Further, the network node 3 may further comprise a network interface, which may optionally comprise a wired interface and/or a wireless interface (such as a WI-FI interface, a bluetooth interface, etc.), typically for establishing a communication connection between the network node 3 and other electronic devices.
Optionally, the network node 3 may further comprise a user interface, which may comprise a Display (Display), an input unit such as a Keyboard (Keyboard), and optionally a standard wired interface, a wireless interface. Alternatively, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, or the like. The display, which may also be referred to as a display screen or display unit, is suitable for displaying information processed in the network node 3 and for displaying a visual user interface, among other things.
It will be appreciated by a person skilled in the art that the schematic diagram 3 is merely an example of the network node 3 and does not constitute a limitation of the network node 3 and may comprise more or less components than those shown, or some components may be combined, or different components.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product.
The computer program product includes one or more computer instructions. When loaded and executed on a network node, causes, in whole or in part, the procedures or functions described in accordance with embodiments of the invention. The network node may be a general purpose computer, a special purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can store or a data storage device, such as a server, a data center, etc., that is integrated with one or more available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the above-described device embodiments are merely illustrative, and for example, the division of the units is only one type of logical function division, and other division manners may be available in actual implementation, for example, a plurality of units or components may be combined or integrated into another device, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a network node (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-only memory (ROM), a magnetic disk, or an optical disk.
It should be noted that the above-mentioned numbers of the embodiments of the present invention are merely for description, and do not represent the merits of the embodiments. And the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, apparatus, article, or method. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, apparatus, article, or method that includes the element.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (10)
1. A distributed storage method is applied to a network node of a distributed storage network, and is characterized by comprising the following steps:
receiving a storage request for specified data;
establishing a time window array according to a preset storage period;
determining a time window corresponding to the specified data from the time window array, and inserting the storage task of the specified data into the determined time window;
and executing the storage task of the designated data according to the preset storage period and the determined time window, determining a target node from the distributed storage network, and storing the designated data to the target node.
2. The method of claim 1, wherein the determining the time window corresponding to the specified data from the time window array comprises:
determining a starting time window from the time window array;
selecting a time window from the starting time window, and judging whether the storage task of the selected time window meets a preset condition or not;
and if the storage task of the selected time window meets the preset condition, adding the storage task of the specified data into the selected time window.
3. The method of claim 1, wherein the determining whether the storage task of the selected time window satisfies a preset condition comprises:
and judging whether the storage task number or the weighted storage task number of the selected time window is smaller than the sum of the average storage task number of the time window array and a preset constant.
4. The method of claim 1, wherein the determining a starting time window from the array of time windows comprises:
calculating the time difference seconds between the current time of the network node and the preset initial time;
and performing modulo operation on the length of the time window array by using the time difference seconds to obtain the position of the starting time window in the time window array.
5. The method of claim 1, wherein the determining a target node from the distributed storage network comprises:
calculating the distance between each network node and the designated data according to the node ID of each network node of the distributed storage network and the identification information of the designated data;
and determining a specified number of network nodes with the minimum distance to the specified data as the target nodes.
6. The method of claim 5, wherein the calculating the distance of each network node from the designated data according to the node ID of each network node of the distributed storage network and the identification information of the designated data comprises:
and carrying out XOR operation on the node ID of each network node and the identification information of the designated data to obtain the distance between each network node and the designated data.
7. The method of claim 1, wherein the determining a target node from the distributed storage network comprises:
selecting candidate nodes according to the region and the operator to which the requester of the storage request belongs and the region and the operator to which each network node belongs;
judging whether the candidate node meets a screening condition;
and if the candidate node meets the screening condition, taking the candidate node as the target node.
8. A network node, characterized in that the network node comprises a memory and a processor, the memory having stored thereon a computer program executable on the processor, the computer program, when executed by the processor, implementing the distributed storage method according to any of claims 1 to 7.
9. A distributed storage apparatus applied to a network node of a distributed storage network, the apparatus comprising:
a receiving unit configured to receive a storage request of specified data;
the establishing unit is used for establishing a time window array according to a preset storage period;
the inserting unit is used for determining a time window corresponding to the specified data from the time window array and inserting the storage task of the specified data into the determined time window;
and the execution unit is used for executing the storage task of the specified data according to the preset storage period and the determined time window, determining a target node from the distributed storage network, and storing the specified data to the target node.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the distributed storage method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911368709.6A CN111142799A (en) | 2019-12-26 | 2019-12-26 | Distributed storage method and device, network node and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911368709.6A CN111142799A (en) | 2019-12-26 | 2019-12-26 | Distributed storage method and device, network node and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111142799A true CN111142799A (en) | 2020-05-12 |
Family
ID=70520541
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911368709.6A Pending CN111142799A (en) | 2019-12-26 | 2019-12-26 | Distributed storage method and device, network node and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111142799A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113259481A (en) * | 2021-06-21 | 2021-08-13 | 湖南视觉伟业智能科技有限公司 | Distributed data storage method, system and readable storage medium |
| CN115550251A (en) * | 2022-12-01 | 2022-12-30 | 杭州蚂蚁酷爱科技有限公司 | Block chain network, node set maintenance method and device |
| CN117053619A (en) * | 2023-10-13 | 2023-11-14 | 常州中冷环保技术有限公司 | Cooling tower operation monitoring system and control method |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104486442A (en) * | 2014-12-29 | 2015-04-01 | 成都致云科技有限公司 | Method and device for transmitting data of distributed storage system |
| CN104932953A (en) * | 2015-06-04 | 2015-09-23 | 华为技术有限公司 | Data distribution method, data storage method, and relevant device and system |
| CN109062512A (en) * | 2018-07-26 | 2018-12-21 | 郑州云海信息技术有限公司 | A kind of distributed storage cluster, data read-write method, system and relevant apparatus |
| CN109471727A (en) * | 2018-10-29 | 2019-03-15 | 北京金山云网络技术有限公司 | A kind of task processing method, apparatus and system |
| CN109525410A (en) * | 2017-09-20 | 2019-03-26 | 华为技术有限公司 | The method, apparatus and distributed memory system of distributed memory system updating and management |
| CN110008181A (en) * | 2019-04-08 | 2019-07-12 | 深圳市网心科技有限公司 | File polling method and device, computer installation and storage medium |
| CN110442773A (en) * | 2019-08-13 | 2019-11-12 | 深圳市网心科技有限公司 | Distributed system interior joint caching method, system, device and computer media |
| CN110460676A (en) * | 2019-08-22 | 2019-11-15 | 腾讯科技（深圳）有限公司 | Request amount control method, device, storage medium and computer equipment |
-
2019
- 2019-12-26 CN CN201911368709.6A patent/CN111142799A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104486442A (en) * | 2014-12-29 | 2015-04-01 | 成都致云科技有限公司 | Method and device for transmitting data of distributed storage system |
| CN104932953A (en) * | 2015-06-04 | 2015-09-23 | 华为技术有限公司 | Data distribution method, data storage method, and relevant device and system |
| CN109525410A (en) * | 2017-09-20 | 2019-03-26 | 华为技术有限公司 | The method, apparatus and distributed memory system of distributed memory system updating and management |
| CN109062512A (en) * | 2018-07-26 | 2018-12-21 | 郑州云海信息技术有限公司 | A kind of distributed storage cluster, data read-write method, system and relevant apparatus |
| CN109471727A (en) * | 2018-10-29 | 2019-03-15 | 北京金山云网络技术有限公司 | A kind of task processing method, apparatus and system |
| CN110008181A (en) * | 2019-04-08 | 2019-07-12 | 深圳市网心科技有限公司 | File polling method and device, computer installation and storage medium |
| CN110442773A (en) * | 2019-08-13 | 2019-11-12 | 深圳市网心科技有限公司 | Distributed system interior joint caching method, system, device and computer media |
| CN110460676A (en) * | 2019-08-22 | 2019-11-15 | 腾讯科技（深圳）有限公司 | Request amount control method, device, storage medium and computer equipment |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113259481A (en) * | 2021-06-21 | 2021-08-13 | 湖南视觉伟业智能科技有限公司 | Distributed data storage method, system and readable storage medium |
| CN113259481B (en) * | 2021-06-21 | 2021-10-12 | 湖南视觉伟业智能科技有限公司 | Distributed data storage method, system and readable storage medium |
| CN115550251A (en) * | 2022-12-01 | 2022-12-30 | 杭州蚂蚁酷爱科技有限公司 | Block chain network, node set maintenance method and device |
| CN117053619A (en) * | 2023-10-13 | 2023-11-14 | 常州中冷环保技术有限公司 | Cooling tower operation monitoring system and control method |
| CN117053619B (en) * | 2023-10-13 | 2024-01-02 | 常州中冷环保技术有限公司 | Cooling tower operation monitoring system and control method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9760723B2 (en) | Techniques for in-app user data authorization | |
| US20180181725A1 (en) | User-generated content permissions status analysis system and method | |
| US9747201B2 (en) | Methods and systems for managing memory allocation | |
| US9544726B2 (en) | Adding location names using private frequent location data | |
| CN111800462B (en) | Micro service instance processing method, micro service instance processing device, computer equipment and storage medium | |
| CN111142799A (en) | Distributed storage method and device, network node and storage medium | |
| US20150205593A1 (en) | Information processing device, system, information processing method and computer readable non-transitory medium | |
| CN110740088B (en) | Method, device, terminal and medium for recommending and adding social resources | |
| CN112468409A (en) | Access control method, device, computer equipment and storage medium | |
| US20160188717A1 (en) | Network crawling prioritization | |
| CN110677506B (en) | Network access method, device, computer equipment and storage medium | |
| CN111193804B (en) | Distributed storage method and device, network node and storage medium | |
| US10643252B2 (en) | Banner display method of electronic device and electronic device thereof | |
| US9565164B2 (en) | Techniques to rate-adjust data usage with a virtual private network | |
| CN109639801A (en) | Back end distribution and data capture method and system | |
| CN107797758B (en) | Date storage method, data access method and device | |
| US20140282063A1 (en) | System for updating icon interface with icons of different operating systems and method thereof | |
| CN113254106B (en) | Task execution method and device based on Flink, computer equipment and storage medium | |
| WO2018049941A1 (en) | Data migration method and related product | |
| CN110245016B (en) | Data processing method, system, device and terminal equipment | |
| EP2871802B1 (en) | Techniques to rate-adjust data usage with a virtual private network | |
| WO2021114075A1 (en) | Credit score processing method, system and apparatus based on blockchain, and medium | |
| CN110336823B (en) | Task allocation method, electronic device, system and medium based on RTT | |
| CN113434069A (en) | Menu configuration method, device, equipment and storage medium | |
| CN111147556A (en) | Load balancing method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |