CN108875035B - Data storage method of distributed file system and related equipment - Google Patents
Data storage method of distributed file system and related equipment Download PDFInfo
- Publication number
- CN108875035B CN108875035B CN201810665263.2A CN201810665263A CN108875035B CN 108875035 B CN108875035 B CN 108875035B CN 201810665263 A CN201810665263 A CN 201810665263A CN 108875035 B CN108875035 B CN 108875035B
- Authority
- CN
- China
- Prior art keywords
- storage node
- storage
- weight parameter
- network delay
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000013500 data storage Methods 0.000 title claims abstract description 41
- 238000000034 method Methods 0.000 title claims abstract description 38
- 238000004891 communication Methods 0.000 description 5
- 238000012360 testing method Methods 0.000 description 5
- 230000008859 change Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 241000282414 Homo sapiens Species 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000001934 delay Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001052 transient effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the application discloses a data storage method and related equipment of a distributed file system, which are used for reducing the delay of displaying a directory and improving the efficiency of displaying the directory. The distributed file system comprises at least one storage node, and the method of the embodiment of the application comprises the following steps: receiving data to be stored; acquiring a first weight parameter of each storage node in the at least one storage node, wherein the first weight parameter is generated according to a first network delay and a first remaining storage capacity of each storage node, and the larger the first network delay is, the smaller the first weight parameter is; determining the storage node with the largest weight parameter in all the storage nodes as a first storage node according to the first weight parameter; and storing the data to be stored in the first storage node.
Description
Technical Field
The present application relates to the field of data storage, and in particular, to a data storage method and related device for a distributed file system.
Background
In the 21 st century, with the coming of the internet, especially the rapid development of the mobile internet, social networks and electronic commerce, the data generated by human beings in production and life is exponentially increased. With the increasing data, the storage capacity is required to be larger, the traditional storage technologies of single node and disk array are difficult to meet the requirement of mass data storage, and the cluster storage system is widely used with its natural scalability advantage, including distributed file system.
The distributed file system is composed of a plurality of storage nodes, and each storage node can be distributed at different places to carry out communication and data transmission among the storage nodes through a network. When data needs to be written into the distributed file system, a certain storage node needs to be selected from a plurality of storage nodes to serve as a target storage node for storing the data.
In the prior art, when data is written, a certain storage node is randomly selected from a plurality of storage nodes as a target storage node, the efficiency of writing the data into the storage node with poor network condition is low due to different network conditions of the storage nodes, and when the randomly selected storage node is the storage node with poor network condition, the time spent on storing the data is long, and the data storage efficiency is low.
Disclosure of Invention
The embodiment of the application provides a data storage method and related equipment of a distributed file system, which are used for reducing the time spent on storing data and improving the data storage efficiency.
In a first aspect, an embodiment of the present application provides a data storage method for a distributed file system, where the distributed file system includes at least one storage node, and the method includes:
receiving data to be stored;
acquiring a first weight parameter of each storage node in the at least one storage node, wherein the first weight parameter is generated according to a first network delay and a first remaining storage capacity of each storage node, and the larger the first network delay is, the smaller the first weight parameter is;
determining the storage node with the largest first weight parameter in the storage nodes as a first storage node;
and storing the data to be stored in the first storage node.
In a second aspect, an embodiment of the present application provides a server applied to a distributed file system, where the distributed file system includes at least one storage node, and the system includes:
the receiving unit is used for receiving data to be stored;
a first obtaining unit, configured to obtain a first weight parameter of each storage node in the at least one storage node, where the first weight parameter is generated according to a first network delay and a first remaining storage capacity of each storage node, and the larger the first network delay is, the smaller the first weight parameter is;
a determining unit, configured to determine a storage node with a largest first weight parameter in the storage nodes as a first storage node;
and the storage unit is used for storing the data to be stored received by the receiving unit in the first storage node.
In a third aspect, an embodiment of the present application further provides a server, where the server includes: a processor and a memory, wherein instructions for data storage of the distributed file system according to the first aspect are stored in the memory;
the processor is configured to execute instructions of the data storage of the distributed file system stored in the memory, to perform the steps of the data storage method of the distributed file system as described in the foregoing first aspect.
In a fourth aspect, the present application provides a computer-readable storage medium, where instructions of data storage of a distributed file system are stored, and when the instructions are executed on a computer, the computer is caused to execute the steps of the data storage method of the distributed file system according to the first aspect.
According to the technical scheme, the embodiment of the application has the following advantages:
and after receiving the data to be stored, acquiring the weight parameter of each node, wherein the weight parameter of each storage node is generated according to the network delay and the residual storage capacity of each storage node, and the larger the network delay is, the smaller the weight parameter of the storage node is, and the storage node with the largest weight parameter in each storage node is selected as the storage node for storing the data. In the scheme, when the storage nodes are selected, the network delay of each storage node is considered, and the larger the network delay of a certain node is, namely the node with poor network condition is, the smaller the possibility of selecting the node is, so that the node with poor network condition is avoided to be selected as much as possible, the time spent on storing data is further reduced, and the data storage efficiency is improved.
Drawings
Fig. 1 is a schematic network architecture diagram of a data storage method of a distributed file system according to an embodiment of the present application;
fig. 2 is a schematic flowchart of a data storage method of a distributed file system according to an embodiment of the present application;
fig. 3 is a schematic flowchart of another data storage method of a distributed file system according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of a server according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of a server according to an embodiment of the present application.
Detailed Description
The embodiment of the application provides a data storage method of a distributed file system, which is used for reducing the time spent on storing data and improving the data storage efficiency. The embodiment of the application also provides a corresponding server and a computer readable storage medium. The following are detailed below.
The terms "first," "second," "third," "fourth," and the like in the description and in the claims, as well as in the drawings, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein.
As shown in fig. 1, the distributed file system in the present application includes a client 10, a server 20, and a storage node 30, and although only one client, one server, and four storage nodes are shown in fig. 1, it should be understood that the illustration in fig. 1 is merely for facilitating understanding of the present solution, the client 10, the server 20, and the storage node 30 may be multiple, and the number of the specific clients 10, the servers 20, and the storage nodes 30 should be flexibly set according to actual needs, and is not limited herein.
In the embodiment of the present application, the client 10 is configured to receive data entered by a user and send the data to the server 20. The server 20 is configured to manage each storage node 30, receive data sent by the client 10, and store the data in the storage node 30 according to a certain rule. The client 10, the server 20 and the storage node 30 are all connected through network communication.
In the embodiment of the present invention, the client 10 may be a computer, a tablet computer, a PDA (Personal Digital Assistant), a mobile phone, a car computer, a TV or other devices with a communication module, and the like, which is not limited herein.
The server 20 may be a server, a server cluster composed of a plurality of servers, or a cloud computing service center, and may store and process data, which is not limited herein.
The storage node 30 may be a metadata storage node or a user data storage node, and the server 20 stores metadata to be written in the metadata storage node and stores user data to be written in the user data storage node.
The network may be a WIreless network connection or a mobile network connection, including, but not limited to, such as a WIreless-FIdelity (WiFi) connection, a bluetooth connection, or the like; the Mobile network connection includes, but is not limited to, Global System for Mobile communications (GSM), Code Division Multiple Access (CDMA), and the like.
Referring to fig. 2, a data storage method of a distributed file system according to an embodiment of the present application is described in detail below, and an embodiment of the data storage method of the distributed file system according to the present application includes:
201. the server receives data to be stored.
In this embodiment, when a user writes data to be stored through a client, a server may receive the data to be stored and determine data that needs to be written into a storage node. The data to be written into the storage node may be metadata, user data, other types of data, and the like, and is not limited herein.
Metadata is information describing attributes of user data, and is used to support functions such as indicating storage locations, searching for history data, recording user data, and the like.
202. The server acquires a first weight parameter of each storage node in the at least one storage node.
In this embodiment, the first weight parameter of each storage node is a preset parameter of the server, the first weight parameter is generated by the server according to the first network delay and the first remaining storage capacity of each storage node, and the larger the first network delay is, the smaller the first weight parameter is. And after the server receives the data to be stored, acquiring a first weight parameter of each storage node from the set data.
In this embodiment, the network delay is a total delay generated when the data to be stored is transmitted from the server to the storage node.
In this embodiment, the relationship function between the weight parameter and the network delay and the remaining storage capacity is obtained by a pre-test, and as an example, the hardware environment of the test is as follows: the processor is Intel fourth generation core i 7-4790, the memory is 16GB, the hard disk is Higase ST1000DM 003-1 ER162, the network card is Reyi RTL8168/8111/811, the operating system is Ubuntu14.04, the metadata storage system is XFS, the information of each storage node is ClientClient IP: 192.168.1.30, OSD.0OSD/MonitorIP: 192.168.1.1, osd.1 OSDIP: 192.168.1.2 and osd.2 OSDIP: 192.168.1.4. adjusting the network delay of the target storage node in the 4 storage nodes by taking 5 milliseconds as a step length and taking 0 millisecond to 50 milliseconds as a range, after the network delay of the storage node is set, the weight parameter of the target storage node is adjusted by taking 0.1 as a step length and 0 to 1 as a range, the weight parameters of the other 3 storage nodes are all 1, 2048 random files of 2MB are written into the 4 storage nodes in each weight parameter change stage, the random file has the size of 4GB totally, the time t required for storing the random file with the size of 4GB in each weight parameter change stage is calculated, by comparing the storage completion time t of the 4 storage nodes under the conditions of different network delays and different weight parameters, and fitting a relation function of the weight parameters of the 4 storage nodes and the network delay of the target storage node under the condition of no load. And adjusting the residual storage capacity of the 4 storage nodes, and repeating the test process to fit a relation function of the weight parameter of the 4 storage nodes and the network delay and the residual storage capacity of the target storage node under different residual storage capacities. It should be understood that the examples of the test hardware environment, the network delay parameter, and the weight parameter are only for convenience of understanding the present solution, and the specific setting of the relevant parameters of the test environment should be flexibly selected according to the actual situation, and is not limited herein.
203. And the server determines the storage node with the largest weight parameter in the storage nodes as the first storage node.
In this embodiment, after acquiring the first weight parameter of each storage node, the server compares the first weight parameter of each storage node to select the storage node with the largest weight parameter from the storage nodes, and determines the storage node with the largest weight parameter as the first storage node.
204. And the server stores the data to be stored in the first storage node.
In this embodiment, after receiving data to be stored, a weight parameter of each node is obtained, where the weight parameter of each storage node is generated according to a network delay and a remaining storage capacity of each storage node, and the larger the network delay is, the smaller the weight parameter of the storage node is, and a storage node with the largest weight parameter in each storage node is selected as the storage node storing the data. In the scheme, when the storage nodes are selected, the network delay of each storage node is considered, and the larger the network delay of a certain node is, namely the node with poor network condition is, the smaller the possibility of selecting the node is, so that the node with poor network condition is avoided to be selected as much as possible, the time spent on storing data is further reduced, and the data storage efficiency is improved.
Based on the foregoing embodiment described in fig. 2, another embodiment of a data storage method of a distributed file system provided in the embodiment of the present application specifically refers to fig. 3, where the method includes:
301. the server receives data to be stored.
In this embodiment, step 301 is similar to step 201 in the embodiment shown in fig. 2, and is not described herein again.
302. And the server acquires the second network delay and the second residual storage capacity of each storage node.
In this embodiment, after receiving the data to be stored, the server obtains a second network delay and a second remaining storage capacity of each storage node, where the second network delay is a current network delay of each storage node, and the second remaining storage capacity is a current remaining storage capacity of each storage node.
303. The server determines whether the second network delay of any storage node exceeds the first preset threshold, if not, the server proceeds to step 304, and if the second network delay of any storage node exceeds the first preset threshold, the server proceeds to step 305.
In this embodiment, a first preset threshold is preset in the server, and after the server obtains the second network delay of each storage node, the server traverses the second network delay of each storage node to determine whether the network delay of any storage node exceeds the first preset threshold.
In this embodiment, any storage node is any one of the storage nodes.
The first preset threshold is preset data of the server, the hardware environments of the storage nodes are different, and the setting data of the first preset threshold is also different, for example, the first preset threshold may be 5 milliseconds, 15 milliseconds, 25 milliseconds or other parameter values, and the like.
304. The server acquires the first weight parameter of each storage node.
In this embodiment, when the server determines that the second network delay without any storage node exceeds the first preset threshold, the server obtains a first weight parameter of each storage node, where the first weight parameter is generated according to the first network delay and the first remaining storage capacity of each storage node, and the larger the first network delay is, the smaller the first weight parameter is.
The first network delay may be a network delay of each storage node when the server sets a weight parameter for each storage node for the first time, may also be a network delay of each storage node when the network delay of any storage node in the last time exceeds a first preset threshold, and may also be a network delay in other cases, which is not limited herein.
305. And the server generates a second weight parameter of each storage node according to the second network delay and the second remaining storage capacity.
In this embodiment, when the server determines that the second network delay of any storage node exceeds the first preset threshold, the server generates a second weight parameter of the storage node according to the obtained second network delay and the second remaining storage capacity of each storage node, where the larger the second network delay is, the smaller the second weight parameter is.
306. And the server updates the first weight parameter of each storage node into the second weight parameter.
In this embodiment, since the weight parameter of each storage node does not change in real time, after the server generates the second weight parameter of each storage node, the first weight parameter of each storage node is updated to the second weight parameter.
307. The server judges whether a second storage node with the second network delay exceeding a second preset threshold exists, and if the second storage node with the second network delay exceeding the second preset threshold exists, the step 308 is executed; if there is no second storage node exceeding the second predetermined threshold, step 309 is entered.
In this embodiment, a second preset threshold is preset in the server, and after the server acquires the second network delay of each storage node, the server traverses the second network delay of each storage node to determine whether the network delay of any storage node exceeds the second preset threshold.
The second preset threshold is preset data of the server, and the setting of the second preset threshold is also different according to a change of a hardware environment, generally speaking, a value of the second preset threshold is greater than the first preset threshold, and a value of the second preset threshold may be not less than 50 milliseconds.
308. The server updates the weight parameter of the second storage node to 0.
In this embodiment, step 303, step 305, and step 306 are optional steps, and if step 303, step 305, and step 306 are not executed, step 307 may be entered after step 304 is executed, and when the server determines that there is a second storage node exceeding the second preset threshold, the weight parameter of each storage node is updated to 0 on the basis that the weight parameter of each storage node is the first weight parameter, so as to obtain an updated third weight parameter of each storage node.
If step 303, step 305, and step 306 are executed, when the server determines that there is a second storage node exceeding the second preset threshold, the weight parameter of each storage node is updated to 0 on the basis that the weight parameter of the second storage node is the second weight parameter, so as to obtain an updated fourth weight parameter of each storage node.
It should be understood that if step 303, step 305, and step 306 are executed, and step 303, step 305, and step 306 are not sequentially defined from step 307 to step 308, step 303, step 305, and step 306 may be executed first, and then step 307 to step 308 may be executed; step 307 to step 308 may be performed first, then step 303, step 305 and step 306 may be performed, and step 303, step 305 and step 306 may be performed simultaneously with step 307 to step 308.
309. And the server determines the storage node with the largest weight parameter in the storage nodes as the first storage node.
In this embodiment, steps 302 to 303, 305 to 306, and 307 to 308 are optional steps, and if steps 302 to 303, 305 to 306, and 307 to 308 are not executed, step 309 is similar to step 203 in the embodiment shown in fig. 2, and will not be described again here.
If step 302 to step 303 and step 305 to step 306 are executed, and step 307 to step 308 are not executed, the server determines the storage node with the largest weight parameter among the storage nodes as the first storage node according to the generated second weight parameter of each storage node. That is, after the second weight parameters of the storage nodes are acquired, the second weight parameters of the storage nodes are compared to select the storage node with the largest weight parameter from the storage nodes, and the storage node with the largest weight parameter is determined as the first storage node.
If step 302, step 307 and step 308 are executed, and step 303, step 305 and step 306 are not executed, the server determines the storage node with the largest weight parameter among the storage nodes as the first storage node according to the updated third weight parameter of each storage node. That is, after the third weight parameters of the storage nodes are acquired, the third weight parameters of the storage nodes are compared to select the storage node with the largest weight parameter from the storage nodes, and the storage node with the largest weight parameter is determined as the first storage node.
If steps 302 to 303, 305 to 306, and 307 to 308 are all executed, the server determines the storage node with the largest weight parameter in the storage nodes as the first storage node according to the updated fourth weight parameter of each storage node. That is, after the fourth weight parameters of the storage nodes are acquired, the fourth weight parameters of the storage nodes are compared to select the storage node with the largest weight parameter from the storage nodes, and the storage node with the largest weight parameter is determined as the first storage node.
310. And the server stores the data to be stored in the first storage node.
In this embodiment, step 310 is similar to step 204 in the embodiment shown in fig. 2, and is not described herein again.
In this embodiment, after receiving data to be stored, a weight parameter of each node is obtained, where the weight parameter of each storage node is generated according to a network delay and a remaining storage capacity of each storage node, and the larger the network delay is, the smaller the weight parameter of the storage node is, and a storage node with the largest weight parameter in each storage node is selected as the storage node storing the data. In the scheme, when the storage nodes are selected, the network delay of each storage node is considered, and the larger the network delay of a certain node is, namely the node with poor network condition is, the smaller the possibility of selecting the node is, so that the node with poor network condition is avoided to be selected as much as possible, the time spent on storing data is further reduced, and the data storage efficiency is improved.
Furthermore, the server judges whether the current network delay of each storage node exceeds a first preset threshold, and acquires the first weight parameter of each storage node only when the network delay of any one storage node in each storage node does not exceed the first preset threshold, so that the weight parameter of each storage node is associated with the current network delay of each storage node, and the flexibility and the adaptability to the network condition of the scheme are improved.
Further, the server judges whether the current network delay of each storage node exceeds a first preset threshold, when the network delay of any one storage node in each storage node exceeds the first preset threshold, a second weight parameter of each storage node is generated according to the current network delay and the current residual storage capacity of each storage node, and the storage node for storing data is determined according to the second weight parameter. The method and the device avoid the situation that the weight parameters of the storage nodes are updated too frequently, ensure that the weight parameters of the storage nodes can be adjusted according to the current network delay, and improve the feasibility of the scheme.
Furthermore, when the network delay of the storage node exceeds a second preset threshold, the weight parameter of the storage node is updated to 0, so that the condition that the data is stored in the storage node with an excessively poor network condition, which causes overlong time, is avoided, and the efficiency of data storage is further improved.
Based on the foregoing embodiments shown in fig. 2 and fig. 3, fig. 4 is a schematic structural diagram of a server provided in this embodiment, where the server 400 is applied to a distributed file system, where the distributed file system includes at least one storage node, and the server 400 includes:
a receiving unit 401, configured to receive data to be stored;
a first obtaining unit 402, configured to obtain a first weight parameter of each storage node in the at least one storage node, where the first weight parameter is generated according to a first network delay and a first remaining storage capacity of each storage node, and the larger the first network delay is, the smaller the first weight parameter is;
a determining unit 403, configured to determine, according to the first weight parameter, a storage node with a largest weight parameter in the storage nodes as a first storage node;
a storage unit 404, configured to store the data to be stored received by the receiving unit 401 in the first storage node.
Further, the first network delay is a network delay of each storage node when a network delay of any storage node existing last time exceeds a first preset threshold, where the any storage node is included in the at least one storage node, and the server further includes:
a second obtaining unit 405, configured to obtain a second network delay and a second remaining storage capacity of each storage node, where the second network delay is a current network delay, and the second remaining storage capacity is a current remaining storage capacity;
a determining unit 406, configured to determine whether the second network delay of any storage node exceeds the first preset threshold;
the first obtaining unit 402 is specifically configured to, when the determining unit determines that the second network delay of no storage node exceeds the first preset threshold, obtain a first weight parameter of each storage node.
Further, the server further includes:
a generating unit 407, configured to generate a second weight parameter of each storage node according to the second network delay and a second remaining storage capacity when the determining unit 405 determines that the second network delay of any storage node exceeds the first preset threshold, where the larger the second network delay is, the smaller the second weight parameter is;
an updating unit 408, configured to update the first weight parameter of each storage node to the second weight parameter;
the determining unit 403 is further configured to determine, according to the second weight parameter, a storage node with a largest second weight parameter in the storage nodes as the first storage node.
Further, the at least one storage node further comprises a second storage node,
the determining unit 406 is further configured to determine whether there is a second storage node where the second network delay exceeds a second preset threshold;
the updating unit 408 is further configured to, when the determining unit determines that there is a second storage node exceeding the second preset threshold, update the weight parameter of the second storage node to 0;
the determining unit 403 is further configured to determine, according to the updated third weight parameter, a storage node with a largest second weight parameter in the storage nodes as the first storage node.
In this embodiment, after the receiving unit 401 receives data to be stored, the first obtaining unit 402 obtains a weight parameter of each node, where the weight parameter of each storage node is generated according to a network delay and a remaining storage capacity of each storage node, and the larger the network delay is, the smaller the weight parameter of the storage node is, and the determining unit 403 selects a storage node with the largest weight parameter in each storage node as the storage node storing the data. In the scheme, when the storage nodes are selected, the network delay of each storage node is considered, and the larger the network delay of a certain node is, namely the node with poor network condition is, the smaller the possibility of selecting the node is, so that the node with poor network condition is avoided to be selected as much as possible, the time spent on storing data is further reduced, and the data storage efficiency is improved.
Also provided in embodiments of the present application is a server, see fig. 5, where the server 500 may have a large difference due to different configurations or performances, and may include one or more processors 501 and memory 502 (e.g., one or more mass storage devices). Memory 502 may be, among other things, transient or persistent storage. The program stored on the memory 502 may include one or more modules (not shown), each of which may include a sequence of instructions operating on a server. Further, the processor 501 may be configured to communicate with the memory 502 to execute a series of instruction operations in the memory 502 on the server 500.
The server 500 may also include one or more input-output units 503, one or more power supplies 504, one or more wired or wireless network interfaces 505.
In some embodiments of the invention, the processor 501, the memory 502, the input/output unit 503, the power supply 504, and the wired or wireless network interface 505 may be connected by a bus or other means, as exemplified by the bus connection in fig. 5.
The memory stores instructions for data storage of the distributed file system executed by the server described in the embodiments of fig. 2 and 3;
the processor is configured to execute instructions of the data storage of the distributed file system stored in the memory, and to perform the steps of the data storage method of the distributed file system as described in the foregoing embodiments shown in fig. 2 and 3.
Also provided in an embodiment of the present application is a computer-readable storage medium, which stores therein instructions for data storage of a distributed file system, and when the instructions are executed on a computer, causes the computer to execute the steps of the data storage method of the distributed file system as described in the foregoing embodiments shown in fig. 2 and 3.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The above embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions in the embodiments of the present application.
Claims (8)
1. A method of data storage for a distributed file system, the distributed file system comprising at least one storage node, the method comprising:
receiving data to be stored;
acquiring a first weight parameter of each storage node in the at least one storage node, wherein the first weight parameter is generated according to a first network delay and a first remaining storage capacity of each storage node, and the larger the first network delay is, the smaller the first weight parameter is;
determining the storage node with the largest weight parameter in the storage nodes as a first storage node according to the first weight parameter;
storing the data to be stored in the first storage node;
the first network delay is a network delay of each storage node when a network delay of any storage node existing last time exceeds a first preset threshold, where the any storage node is included in the at least one storage node, and the method further includes: acquiring a second network delay and a second remaining storage capacity of each storage node, wherein the second network delay is the current network delay, and the second remaining storage capacity is the current remaining storage capacity; judging whether the second network delay of any storage node exceeds the first preset threshold value or not; the obtaining of the first weight parameter of each storage node of the at least one storage node includes: if the second network delay without any storage node exceeds the first preset threshold, acquiring a first weight parameter of each storage node;
the first preset threshold is preset data of the server.
2. The method of claim 1, further comprising:
if the second network delay of any storage node exceeds the first preset threshold, generating a second weight parameter of each storage node according to the second network delay and a second remaining storage capacity, wherein the larger the second network delay is, the smaller the second weight parameter is;
updating the first weight parameter of each storage node to the second weight parameter;
the determining, as a first storage node, a storage node with a largest weight parameter among the storage nodes includes:
and determining the storage node with the largest weight parameter in the storage nodes as the first storage node according to the generated second weight parameter.
3. The method of claim 1, further comprising a second storage node in the at least one storage node, the method further comprising:
judging whether a second storage node with the second network delay exceeding a second preset threshold exists or not;
and if the second storage node exceeding the second preset threshold exists, updating the weight parameter of the second storage node to 0.
4. The method according to claim 3, characterized in that said second preset threshold is not less than 50 milliseconds.
5. The method according to any of claims 1 to 4, wherein the first preset threshold is 5 milliseconds, 15 milliseconds or 25 milliseconds.
6. A server, applied to a distributed file system including at least one storage node, the system comprising:
the receiving unit is used for receiving data to be stored;
a first obtaining unit, configured to obtain a first weight parameter of each storage node in the at least one storage node, where the first weight parameter is generated according to a first network delay and a first remaining storage capacity of each storage node, and the larger the first network delay is, the smaller the first weight parameter is;
a determining unit, configured to determine, according to the first weight parameter, a storage node with a largest weight parameter in the storage nodes as a first storage node;
the storage unit is used for storing the data to be stored received by the receiving unit in the first storage node;
the first network delay is a network delay of each storage node when a network delay of any storage node existing last time exceeds a first preset threshold, where the any storage node is included in the at least one storage node, and the server further includes: a second obtaining unit, configured to obtain a second network delay and a second remaining storage capacity of each storage node, where the second network delay is a current network delay, and the second remaining storage capacity is a current remaining storage capacity; the judging unit is used for judging whether the second network delay of any storage node exceeds the first preset threshold value or not; the first obtaining unit is specifically configured to obtain a first weight parameter of each storage node when the determining unit determines that the second network delay without any storage node exceeds the first preset threshold;
the first preset threshold is preset data of the server.
7. A server, characterized in that the server comprises: a processor and a memory, the memory having stored therein instructions for data storage of the distributed file system of any of claims 1-5;
the processor is adapted to execute instructions of the data storage of the distributed file system stored in the memory to perform the steps of the data storage method of the distributed file system according to any of the claims 1-5.
8. A computer-readable storage medium, having stored therein instructions for data storage of a distributed file system, which when run on a computer, cause the computer to perform the steps of the data storage method of the distributed file system of any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810665263.2A CN108875035B (en) | 2018-06-25 | 2018-06-25 | Data storage method of distributed file system and related equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810665263.2A CN108875035B (en) | 2018-06-25 | 2018-06-25 | Data storage method of distributed file system and related equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108875035A CN108875035A (en) | 2018-11-23 |
| CN108875035B true CN108875035B (en) | 2022-02-18 |
Family
ID=64294660
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810665263.2A Active CN108875035B (en) | 2018-06-25 | 2018-06-25 | Data storage method of distributed file system and related equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108875035B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110636122A (en) * | 2019-09-11 | 2019-12-31 | 中移（杭州）信息技术有限公司 | Distributed storage method, server, system, electronic device, and storage medium |
| CN112101836B (en) * | 2020-03-06 | 2021-07-09 | 江苏小梦科技有限公司 | Big data storage node dynamic management system and corresponding terminal |
| CN115865989B (en) * | 2023-02-21 | 2023-05-12 | 中国市政工程西南设计研究总院有限公司 | Wide area network configuration method for high-efficiency and safe interconnection of enterprise headquarter and branch office information |
| CN117082083A (en) * | 2023-08-30 | 2023-11-17 | 杭州浩联智能科技有限公司 | Data storage method, device and medium based on distributed Internet of things architecture |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102082830A (en) * | 2011-01-18 | 2011-06-01 | 浙江大学 | Unstable network-oriented distributed file storage method based on quality perception |
| CN102667772A (en) * | 2010-03-01 | 2012-09-12 | 株式会社日立制作所 | File level hierarchical storage management system, method, and apparatus |
| CN103731505A (en) * | 2014-01-17 | 2014-04-16 | 中国联合网络通信集团有限公司 | Data distributed storage method and system |
| CN104023088A (en) * | 2014-06-28 | 2014-09-03 | 山东大学 | Storage server selection method applied to distributed file system |
| CN104142871A (en) * | 2013-05-10 | 2014-11-12 | 中国电信股份有限公司 | Data backup method and device and distributed file system |
| CN104468670A (en) * | 2013-09-23 | 2015-03-25 | 深圳市腾讯计算机系统有限公司 | Method and device for processing management data, distributed disaster tolerance method and distributed disaster tolerance system |
| CN104780092A (en) * | 2014-01-13 | 2015-07-15 | 阿里巴巴集团控股有限公司 | File transmission method and device as well as server system |
| CN105025053A (en) * | 2014-04-24 | 2015-11-04 | 苏宁云商集团股份有限公司 | Distributed file upload method based on cloud storage technology and system |
| CN105471985A (en) * | 2015-11-23 | 2016-04-06 | 北京农业信息技术研究中心 | Load balance method, cloud platform computing method and cloud platform |
| CN107241418A (en) * | 2017-06-13 | 2017-10-10 | 腾讯科技（深圳）有限公司 | A kind of load-balancing method, device, equipment and computer-readable recording medium |
| CN107451138A (en) * | 2016-05-30 | 2017-12-08 | 中兴通讯股份有限公司 | A kind of distributed file system storage method and system |
| CN107766346A (en) * | 2016-08-15 | 2018-03-06 | 中国联合网络通信集团有限公司 | Distributed file system file access method and device |
| CN107888634A (en) * | 2016-09-29 | 2018-04-06 | 北京金山云网络技术有限公司 | The data request method and device of a kind of distributed memory system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR101791901B1 (en) * | 2016-03-30 | 2017-10-31 | 재단법인차세대융합기술연구원 | The apparatus and method of smart storage platfoam for efficient storage of big data |
-
2018
- 2018-06-25 CN CN201810665263.2A patent/CN108875035B/en active Active
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102667772A (en) * | 2010-03-01 | 2012-09-12 | 株式会社日立制作所 | File level hierarchical storage management system, method, and apparatus |
| CN102082830A (en) * | 2011-01-18 | 2011-06-01 | 浙江大学 | Unstable network-oriented distributed file storage method based on quality perception |
| CN104142871A (en) * | 2013-05-10 | 2014-11-12 | 中国电信股份有限公司 | Data backup method and device and distributed file system |
| CN104468670A (en) * | 2013-09-23 | 2015-03-25 | 深圳市腾讯计算机系统有限公司 | Method and device for processing management data, distributed disaster tolerance method and distributed disaster tolerance system |
| CN104780092A (en) * | 2014-01-13 | 2015-07-15 | 阿里巴巴集团控股有限公司 | File transmission method and device as well as server system |
| CN103731505A (en) * | 2014-01-17 | 2014-04-16 | 中国联合网络通信集团有限公司 | Data distributed storage method and system |
| CN105025053A (en) * | 2014-04-24 | 2015-11-04 | 苏宁云商集团股份有限公司 | Distributed file upload method based on cloud storage technology and system |
| CN104023088A (en) * | 2014-06-28 | 2014-09-03 | 山东大学 | Storage server selection method applied to distributed file system |
| CN105471985A (en) * | 2015-11-23 | 2016-04-06 | 北京农业信息技术研究中心 | Load balance method, cloud platform computing method and cloud platform |
| CN107451138A (en) * | 2016-05-30 | 2017-12-08 | 中兴通讯股份有限公司 | A kind of distributed file system storage method and system |
| CN107766346A (en) * | 2016-08-15 | 2018-03-06 | 中国联合网络通信集团有限公司 | Distributed file system file access method and device |
| CN107888634A (en) * | 2016-09-29 | 2018-04-06 | 北京金山云网络技术有限公司 | The data request method and device of a kind of distributed memory system |
| CN107241418A (en) * | 2017-06-13 | 2017-10-10 | 腾讯科技（深圳）有限公司 | A kind of load-balancing method, device, equipment and computer-readable recording medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108875035A (en) | 2018-11-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108875035B (en) | Data storage method of distributed file system and related equipment | |
| CN111522641B (en) | Task scheduling method, device, computer equipment and storage medium | |
| CN110753112A (en) | Elastic expansion method and device of cloud service | |
| CN110765023A (en) | Distributed system testing method and system based on chaos experiment | |
| CN117278434A (en) | Flow playback method and device and electronic equipment | |
| CN110933178A (en) | Method for adjusting node configuration in cluster system and server | |
| CN111541762B (en) | Data processing method, management server, device and storage medium | |
| CN113204451A (en) | Pressure testing method and system for Redis cluster, storage medium and terminal | |
| CN110581784B (en) | Node health check method, device and equipment | |
| CN113297229B (en) | Method for routing read request and feedback message, respective device and database | |
| CN104063377A (en) | Information processing method and electronic equipment using same | |
| CN111159131A (en) | Performance optimization method, device, equipment and computer readable storage medium | |
| CN110349076B (en) | Data processing method and device | |
| CN114385596A (en) | Data processing method and device | |
| CN110134547B (en) | Middleware-based repeated data deleting method and related device | |
| CN113805854A (en) | Method, system and device for realizing Hook of application layer based on Linux system and storage medium | |
| CN110798222B (en) | Data compression method and device | |
| CN108520025B (en) | Service node determination method, device, equipment and medium | |
| CN113392131A (en) | Data processing method and device and computer equipment | |
| CN108173689B (en) | Output system of load balancing data | |
| CN111435938B (en) | Data request processing method, device and equipment | |
| CN108933709B (en) | Method, apparatus and computer readable storage medium for testing server performance | |
| CN111078718B (en) | Frequency control method, device, equipment and computer storage medium | |
| CN114157917B (en) | Video editing method and device and terminal equipment | |
| CN114528183B (en) | Offline prediction method, device and equipment of LoRa equipment and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |