CN110308875B - Data read-write method, device, equipment and computer readable storage medium - Google Patents
Data read-write method, device, equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN110308875B CN110308875B CN201910572489.2A CN201910572489A CN110308875B CN 110308875 B CN110308875 B CN 110308875B CN 201910572489 A CN201910572489 A CN 201910572489A CN 110308875 B CN110308875 B CN 110308875B
- Authority
- CN
- China
- Prior art keywords
- stripe
- hard disks
- upper layer
- size
- adjusting
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
- G06F3/0689—Disk arrays, e.g. RAID, JBOD
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Digital Magnetic Recording (AREA)
Abstract
The invention discloses a data read-write method, which comprises the following steps: acquiring the number of hard disks contained in each current physical node of the distributed storage system; acquiring the size of a unit data stream requested to be read and written by a data read-write request issued by an upper layer application within a preset duration; adjusting the stripe width corresponding to the upper application according to the number of the hard disks; adjusting the stripe depth corresponding to the upper layer application according to the size of the unit data stream; and executing data read-write operation on each physical node according to the adjusted stripe width and stripe depth. The invention also discloses a data read-write device, equipment and a computer readable storage medium. The invention can realize the self-adaptive adjustment of the strip depth and the strip width, thereby improving the read-write performance of the distributed storage system.
Description
Technical Field
The present invention relates to the field of storage technologies, and in particular, to a data reading and writing method, apparatus, device, and computer readable storage medium.
Background
In the current distributed storage system, the striping technology is widely used, namely, the striping technology divides a continuous piece of data into a plurality of small parts and stores the small parts on different magnetic disks, so that a plurality of processes can access a plurality of different parts of the data simultaneously without causing magnetic disk conflict, and the I/O (Input/Output) parallel capability to the greatest extent can be obtained when the data needs to be accessed sequentially, thereby obtaining very good performance.
Two important parameters are involved in the striping technique: stripe depth and stripe width. Stripe depth, i.e., the size of the stripe, also called stripe size, refers to the size of the stripe data block written on each disk; stripe width refers to the number of stripes that can be concurrently read or written to, which is equal to the number of physical hard disks in a RAID (Redundant Arrays of Independent Drives, disk array), e.g., the stripe width of an striped array of 4 physical hard disks is 4.
Existing distributed storage systems all use a fixed stripe depth and stripe width, which has the disadvantage that: the physical configuration change of the host computer and the IO request change issued by the upper layer application cannot be adapted, so that the distributed storage system cannot achieve the optimal read-write performance.
Disclosure of Invention
The embodiment of the invention mainly aims to provide a data read-write method, a device, equipment and a computer readable storage medium, aiming at realizing the self-adaptive adjustment of stripe depth and stripe width so as to improve the read-write performance of a distributed storage system.
In order to achieve the above object, the present invention provides a data reading and writing method, including the steps of:
acquiring the number of hard disks contained in each current physical node of the distributed storage system;
acquiring the size of a unit data stream requested to be read and written by a data read-write request issued by an upper layer application within a preset duration;
adjusting the stripe width corresponding to the upper application according to the number of the hard disks;
adjusting the stripe depth corresponding to the upper layer application according to the size of the unit data stream;
and executing data read-write operation on each physical node according to the adjusted stripe width and stripe depth.
Preferably, the step of adjusting the stripe width corresponding to the upper layer application according to the number of hard disks includes:
acquiring an IO strategy applied by the upper layer;
if the IO policy applied by the upper layer is an IO localization policy, acquiring the number of hard disks comprising the physical nodes with the least number of hard disks, and taking the acquired number of hard disks as a target number of hard disks;
and adjusting the width of the stripe corresponding to the upper application to be smaller than or equal to the target hard disk number.
Preferably, the step of adjusting the stripe depth corresponding to the upper layer application according to the size of the unit data stream includes:
acquiring the size of a unit data stream with highest occurrence frequency in the preset time length;
determining a preset interval in which the size of the unit data stream with the highest occurrence frequency is located;
inquiring a preset mapping relation between a size interval and a stripe depth of a unit data stream to obtain the stripe depth corresponding to the preset interval, and taking the stripe depth corresponding to the preset interval as a target stripe depth;
and adjusting the stripe depth corresponding to the upper layer application to the target stripe depth.
Preferably, the data read-write method further includes:
if an instruction for creating the file fragment group is detected, acquiring the adjusted stripe width and stripe depth;
creating a file fragment group according to the adjusted stripe width and stripe depth;
and executing data read-write operation on each physical node according to the created file fragment group.
Preferably, the data read-write method further includes:
detecting whether the distributed storage system meets a preset idle condition or not;
if the distributed storage system meets a preset idle condition, acquiring file fragments which do not meet the adjusted stripe width and stripe depth;
classifying the acquired file fragments according to preset service types;
and according to the adjusted stripe width and stripe depth, rewriting the file fragments under the same classification into the hard disk of each physical node of the distributed storage system.
In addition, in order to achieve the above object, an embodiment of the present invention further provides a data reading and writing apparatus, including:
the first acquisition module is used for acquiring the number of hard disks contained in each current physical node of the distributed storage system;
the second acquisition module is used for acquiring the size of a unit data stream requested to be read and written by a data read-write request issued by the upper layer application within a preset duration;
the first adjusting module is used for adjusting the stripe width corresponding to the upper layer application according to the number of the hard disks;
the second adjusting module is used for adjusting the stripe depth corresponding to the upper layer application according to the size of the unit data stream;
and the read-write module is used for executing data read-write operation on each physical node according to the adjusted stripe width and stripe depth.
Preferably, the first adjustment module is further configured to:
acquiring an IO strategy applied by the upper layer;
if the IO policy applied by the upper layer is an IO localization policy, acquiring the number of hard disks comprising the physical nodes with the least number of hard disks, and taking the acquired number of hard disks as a target number of hard disks;
and adjusting the width of the stripe corresponding to the upper application to be smaller than or equal to the target hard disk number.
Preferably, the second adjustment module is further configured to:
acquiring the size of a unit data stream with highest occurrence frequency in the preset time length;
determining a preset interval in which the size of the unit data stream with the highest occurrence frequency is located;
inquiring a preset mapping relation between a size interval and a stripe depth of a unit data stream to obtain the stripe depth corresponding to the preset interval, and taking the stripe depth corresponding to the preset interval as a target stripe depth;
and adjusting the stripe depth corresponding to the upper layer application to the target stripe depth.
Preferably, the strap adjustment device further comprises:
the third acquisition module is used for acquiring the adjusted stripe width and stripe depth if the instruction for creating the file fragment group is detected;
and the creating module is used for creating a file fragment group according to the adjusted stripe width and stripe depth.
Preferably, the strap adjustment device further comprises:
the detection module is used for detecting whether the distributed storage system meets preset idle conditions or not;
a fourth obtaining module, configured to obtain a file fragment that does not conform to the adjusted stripe width and stripe depth if the distributed storage system meets a preset idle condition;
the classifying module is used for classifying the acquired file fragments according to a preset service type;
and the rewriting module is used for rewriting the file fragments under the same classification into the hard disk of each physical node of the distributed storage system according to the adjusted stripe width and stripe depth.
In addition, to achieve the above object, an embodiment of the present invention further provides a data reading and writing apparatus, including: the system comprises a memory, a processor and a data read-write program stored on the memory and capable of running on the processor, wherein the data read-write program realizes the steps of the data read-write method when being executed by the processor.
In addition, in order to achieve the above object, an embodiment of the present invention further provides a computer-readable storage medium having a data read-write program stored thereon, which when executed by a processor, implements the steps of the data read-write method described above.
The embodiment of the invention acquires the number of hard disks contained in each current physical node of the distributed storage system; acquiring the size of a unit data stream requested to be read and written by a data read-write request issued by an upper layer application within a preset duration; adjusting the stripe width corresponding to the upper application according to the number of the hard disks; adjusting the stripe depth corresponding to the upper layer application according to the size of the unit data stream; and executing data read-write operation on each physical node according to the adjusted stripe width and stripe depth. In this way, the stripe depth and stripe width of the distributed storage system can be adapted to the physical configuration change of the host, and the size change of the unit data stream requested to be read and written by the data read-write request issued by the upper layer application, so that the read-write performance of the distributed storage system is improved.
Drawings
FIG. 1 is a schematic diagram of a device architecture of a hardware operating environment according to an embodiment of the present invention;
FIG. 2 is a schematic flow chart of a data read/write method according to an embodiment of the invention;
fig. 3 is a schematic block diagram of a data read-write device according to an embodiment of the invention.
The achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.
Detailed Description
It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
Referring to fig. 1, fig. 1 is a schematic device structure of a hardware running environment according to an embodiment of the present invention.
The data read-write equipment of the embodiment of the invention can be terminal equipment such as a PC (personal computer) or a server (such as an X86 server) which is provided with a distributed storage system.
As shown in fig. 1, the data read-write apparatus may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, a communication bus 1002. Wherein the communication bus 1002 is used to enable connected communication between these components. The user interface 1003 may include a Display, an input unit such as a Keyboard (Keyboard), and the optional user interface 1003 may further include a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface). The memory 1005 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also optionally be a storage device separate from the processor 1001 described above.
It will be appreciated by those skilled in the art that the device structure shown in fig. 1 is not limiting of the device and may include more or fewer components than shown, or may combine certain components, or a different arrangement of components.
As shown in fig. 1, an operating system, a network communication module, a user interface module, and a data read-write program may be included in the memory 1005, which is a kind of computer-readable storage medium.
In the terminal shown in fig. 1, the network interface 1004 is mainly used for connecting to a background server and performing data communication with the background server; the user interface 1003 is mainly used for connecting a client (user side) and performing data communication with the client; and the processor 1001 may be configured to call a data read/write program stored in the memory 1005 and perform the following operations:
acquiring the number of hard disks contained in each current physical node of the distributed storage system;
acquiring the size of a unit data stream requested to be read and written by a data read-write request issued by an upper layer application within a preset duration;
adjusting the stripe width corresponding to the upper application according to the number of the hard disks;
adjusting the stripe depth corresponding to the upper layer application according to the size of the unit data stream;
and executing data read-write operation on each physical node according to the adjusted stripe width and stripe depth.
Further, the processor 1001 may call a data read-write program stored in the memory 1005, and further perform the following operations:
acquiring an IO strategy applied by the upper layer;
if the IO policy applied by the upper layer is an IO localization policy, acquiring the number of hard disks comprising the physical nodes with the least number of hard disks, and taking the acquired number of hard disks as a target number of hard disks;
and adjusting the width of the stripe corresponding to the upper application to be smaller than or equal to the target hard disk number.
Further, the processor 1001 may call a data read-write program stored in the memory 1005, and further perform the following operations:
acquiring the size of a unit data stream with highest occurrence frequency in the preset time length;
determining a preset interval in which the size of the unit data stream with the highest occurrence frequency is located;
inquiring a preset mapping relation between a size interval and a stripe depth of a unit data stream to obtain the stripe depth corresponding to the preset interval, and taking the stripe depth corresponding to the preset interval as a target stripe depth;
and adjusting the stripe depth corresponding to the upper layer application to the target stripe depth.
Further, the processor 1001 may call a data read-write program stored in the memory 1005, and further perform the following operations:
if an instruction for creating the file fragment group is detected, acquiring the adjusted stripe width and stripe depth;
creating a file fragment group according to the adjusted stripe width and stripe depth;
and executing data read-write operation on each physical node according to the created file fragment group.
Further, the processor 1001 may call a data read-write program stored in the memory 1005, and further perform the following operations:
detecting whether the distributed storage system meets a preset idle condition or not;
if the distributed storage system meets a preset idle condition, acquiring file fragments which do not meet the adjusted stripe width and stripe depth;
classifying the acquired file fragments according to preset service types;
and according to the adjusted stripe width and stripe depth, rewriting the file fragments under the same classification into the hard disk of each physical node of the distributed storage system.
Based on the hardware structure, the embodiment of the data reading and writing method is provided.
Referring to fig. 2, fig. 2 is a flow chart of a data read-write method according to an embodiment of the invention, where the method includes:
step S10, obtaining the number of hard disks contained in each current physical node of the distributed storage system;
the distributed storage method is applied to a distributed storage system, the distributed storage system adopts a banding technology, and each logic band can be distributed on hard disks of different hosts so as to realize IO load sharing of the system. Two important parameters are involved in the striping technique: stripe depth and stripe width, the stripe depth, i.e., the size of the stripe, also called stripe size, refers to the size of the stripe data block written on each disk; stripe width refers to the number of stripes that can be concurrently read or written to, which is equal to the number of physical disks in a RAID, e.g., the stripe width of an array of striped, 4 physical disks is 4.
Because of the heterogeneous compatibility of distributed storage, the number of hard disks of different hosts may be different, and if a fixed stripe width is used, the best read-write performance effect may not be achieved in some cases. For example, when the host is scaled, the set stripe width may be larger than the total number of discs in distributed storage, which may cause a plurality of stripes of a stripe group to be distributed on the same hard disc, so that an upper layer of a sequential read-write request becomes a random read-write request on the disc, and since the random performance of the hard disc is far lower than the sequential performance, this may cause a huge performance degradation, and when the host is scaled, for example, the set stripe width may be smaller than the total number of discs in distributed storage, which may cause the performance of a plurality of hard discs to be underutilized, so that the best performance effect is not achieved. In addition, due to the difference of the upper layer application services, if a fixed stripe depth is adopted, the best read-write performance effect cannot be achieved in some cases, for example, the size of the unit data stream requested to be read and written by the current upper layer application service is 1MB (the service pursuing the IO throughput), and the stripe depth of the lower layer is 4KB, which causes one IO request to be split into a plurality of different hard disks, so that the best throughput performance cannot be exerted. For this reason, the present embodiment proposes a data reading and writing method.
Firstly, the distributed storage system may periodically acquire the number of Hard disks included in each current physical node, where the Hard Disk may be an HDD (Hard Disk Drive), an SSD (Solid State Drive), or a hybrid Hard Disk of the HDD and the SSD, which is not limited in this embodiment.
Step S20, obtaining the size of a unit data stream requested to be read and written by a data read-write request issued by an upper layer application within a preset duration;
in this step, the distributed storage system obtains the size of the unit data stream requested to be read and written by the data read-write request issued by the upper layer application within the preset duration, i.e. the IO size (IO size), where the upper layer application refers to the software service running in the application layer, and the preset duration can be flexibly set, for example, the past 1 hour, 1 day, etc.
Step S30, adjusting the stripe depth corresponding to the upper layer application according to the size of the unit data stream;
in the step, the distributed storage system adaptively adjusts the stripe width corresponding to the upper layer application according to the obtained hard disk number.
In an embodiment, the step S30 may further include: acquiring an IO strategy applied by the upper layer; if the IO policy applied by the upper layer is an IO localization policy, acquiring the number of hard disks comprising the physical nodes with the least number of hard disks, and taking the acquired number of hard disks as a target number of hard disks; and adjusting the width of the stripe corresponding to the upper application to be smaller than or equal to the target hard disk number.
Specifically, when the stripe width is adjusted according to the number of hard disks, the current IO policy applied by the upper layer is firstly obtained, when the current IO policy applied by the upper layer is an IO localization policy, namely, only the local host responds to an IO request, the physical node with the least number of hard disks is obtained, the number of hard disks of the physical node is taken as the target number of hard disks, and then the corresponding stripe width applied by the upper layer is adjusted so that the stripe width is smaller than or equal to the target number of hard disks. Therefore, the stripes in one stripe group can be prevented from falling on the same hard disk, and the read-write performance of the distributed storage system is improved.
It should be noted that, when the current IO policy is an IO localization policy, the upper layer may further detect whether the data to be read and written corresponding to the IO request is multi-copy data, and if so, the stripe width should be adjusted to be less than or equal to the target hard disk number divided by the corresponding copy number.
And step S40, adjusting the stripe depth corresponding to the upper layer application according to the size of the unit data stream.
In the step, the distributed storage system adaptively adjusts the stripe width corresponding to the upper layer application according to the obtained hard disk number.
In an embodiment, the step S30 may further include: acquiring the size of a unit data stream with highest occurrence frequency in the preset time length; determining a preset interval in which the size of the unit data stream with the highest occurrence frequency is located; inquiring a preset mapping relation between a size interval and a stripe depth of a unit data stream to obtain the stripe depth corresponding to the preset interval, and taking the stripe depth corresponding to the preset interval as a target stripe depth; and adjusting the stripe depth corresponding to the upper layer application to the target stripe depth.
Specifically, in this embodiment, a mapping relationship between a size interval and a stripe depth of a unit data stream may be preset, where different intervals correspond to different stripe depths, and the size and number of intervals may be flexibly set. When the stripe depth is adjusted according to the size of the unit data stream, firstly, the size of the unit data stream with the highest occurrence frequency in the preset time length is obtained, for example, the size of the unit data stream is 10 times when the size of the unit data stream is 1M in the preset time length, 3 times when the size of the unit data stream is 128k, and 5 times when the size of the unit data stream is 64k, and the size of the unit data stream with the highest occurrence frequency in the preset time length is 1M; then, determining a preset interval in which the size of the unit data stream with the highest occurrence frequency is positioned; then, inquiring a preset mapping relation between a size interval and a stripe depth of the unit data stream to obtain the stripe depth corresponding to the preset interval, and taking the stripe depth corresponding to the preset interval as a target stripe depth; finally, the stripe depth corresponding to the upper layer application is adjusted to the target stripe depth. Therefore, the stripe depth can be adapted to the unit data stream size of the upper layer application, and the read-write performance of the distributed storage system is improved.
Of course, other manners of adjusting the stripe depth according to the size of the unit data stream with the highest occurrence frequency in the preset duration may be adopted, for example, the preset duration is divided into a plurality of time periods, corresponding weight values are set for each time period respectively, then the sizes of the unit data streams issued in each time period are multiplied by the corresponding weight values and summed to obtain a reference value of the sizes of the unit data streams, and then the stripe depth corresponding to the upper layer application is adjusted according to the reference value, so that the stripe depth can be flexibly set in specific implementation.
And S50, executing data read-write operation on each physical node according to the adjusted stripe width and stripe depth.
In the step, when a data read-write request issued by an upper layer application is subsequently received, the data read-write operation on each physical node is executed according to the adjusted stripe width and stripe depth, so that the data read-write operation on each physical node is executed according to the self-adaptively adjusted stripe depth and stripe width, and the read-write performance of the distributed storage system is improved.
In this embodiment, the number of hard disks contained in each current physical node of the distributed storage system is obtained; acquiring the size of a unit data stream requested to be read and written by a data read-write request issued by an upper layer application within a preset duration; adjusting the stripe width corresponding to the upper application according to the number of the hard disks; adjusting the stripe depth corresponding to the upper layer application according to the size of the unit data stream; and executing data read-write operation on each physical node according to the adjusted stripe width and stripe depth. In this way, the stripe depth and stripe width of the distributed storage system can be adapted to the physical configuration change of the host, and the size change of the unit data stream requested to be read and written by the data read-write request issued by the upper layer application, so that the read-write performance of the distributed storage system is improved.
Further, based on the first embodiment of the data reading and writing method of the present invention, a second embodiment of the data reading and writing method of the present invention is provided.
In this embodiment, the data read-write method may further include: if an instruction for creating the file fragment group is detected, acquiring the adjusted stripe width and stripe depth; and creating a file fragment group according to the adjusted stripe width and stripe depth.
In this embodiment, when a new file is created or an old file size needs to be increased, that is, an instruction for creating a file fragment is triggered, at this time, the distributed storage system obtains the adjusted stripe width and stripe depth, then determines the number of file fragments and the size of each fragment according to the adjusted stripe width and stripe depth, and creates a file fragment group according to the number of file fragments and the size of each fragment. Therefore, the newly created file fragment group is ensured to be in line with the number of hard disks of the physical nodes of the distributed storage system and the data distribution strategy of the current system, and a precondition guarantee is provided for improving the read-write performance of the file.
Further, the data read-write method may further include: detecting whether the distributed storage system meets a preset idle condition or not; if the distributed storage system meets a preset idle condition, acquiring file fragments which do not meet the adjusted stripe width and stripe depth; classifying the acquired file fragments according to preset service types; and according to the adjusted stripe width and stripe depth, rewriting the file fragments under the same classification into the hard disk of each physical node of the distributed storage system.
In this embodiment, the distributed storage system may enable a detection program to detect in real time whether the system currently meets a preset idle condition, where the idle condition includes, but is not limited to, a system CPU usage rate and a memory usage rate being less than a preset threshold, a current system time being in a preset idle period, and so on; if the distributed storage system currently meets the preset idle condition, scanning all current file fragments of the system to obtain the file fragments which do not currently accord with the adjusted stripe width and the stripe depth; then classifying the acquired file fragments according to preset service types, wherein the preset service types comprise, but are not limited to, uploading/downloading video, backup, big data analysis, reading and writing databases and the like; and then, according to the adjusted stripe width and stripe depth, rewriting the file fragments under the same classification into the hard disk of each physical node of the distributed storage system.
According to the mode, when the system is idle, the files which do not accord with the current stripe width and stripe depth are rewritten into the hard disk in a slicing mode, so that the read-write performance of the distributed storage system is further improved.
The invention further provides a distributed storage device. Referring to fig. 3, fig. 3 is a schematic block diagram of a data read-write device according to an embodiment of the invention. In this embodiment, the distributed storage device includes:
the first obtaining module 10 is configured to obtain the number of hard disks contained in each current physical node of the distributed storage system;
the second obtaining module 20 is configured to obtain a size of a unit data stream requested to be read and written by a data read-write request issued by an upper layer application within a preset duration;
a first adjustment module 30, configured to adjust a stripe width corresponding to the upper layer application according to the number of hard disks;
a second adjusting module 40, configured to adjust a stripe depth corresponding to the upper layer application according to the size of the unit data stream;
and the read-write module 50 is used for executing data read-write operation on each physical node according to the adjusted stripe width and stripe depth.
Further, the first adjusting module 30 is further configured to:
acquiring an IO strategy applied by the upper layer;
if the IO policy applied by the upper layer is an IO localization policy, acquiring the number of hard disks comprising the physical nodes with the least number of hard disks, and taking the acquired number of hard disks as a target number of hard disks;
and adjusting the width of the stripe corresponding to the upper application to be smaller than or equal to the target hard disk number.
Further, the second adjusting module 40 is further configured to:
acquiring the size of a unit data stream with highest occurrence frequency in the preset time length;
determining a preset interval in which the size of the unit data stream with the highest occurrence frequency is located;
inquiring a preset mapping relation between a size interval and a stripe depth of a unit data stream to obtain the stripe depth corresponding to the preset interval, and taking the stripe depth corresponding to the preset interval as a target stripe depth;
and adjusting the stripe depth corresponding to the upper layer application to the target stripe depth.
Further, the data read-write device further includes:
the third acquisition module is used for acquiring the adjusted stripe width and stripe depth if the instruction for creating the file fragment group is detected;
and the creating module is used for creating a file fragment group according to the adjusted stripe width and stripe depth.
Further, the data read-write device further includes:
the detection module is used for detecting whether the distributed storage system meets preset idle conditions or not;
a fourth obtaining module, configured to obtain a file fragment that does not conform to the adjusted stripe width and stripe depth if the distributed storage system meets a preset idle condition;
the classifying module is used for classifying the acquired file fragments according to a preset service type;
and the rewriting module is used for rewriting the file fragments under the same classification into the hard disk of each physical node of the distributed storage system according to the adjusted stripe width and stripe depth.
The method for implementing each functional module may refer to the embodiment of the data reading and writing method of the present invention, and will not be described herein.
In this embodiment, the first obtaining module 10 obtains the number of hard disks contained in each current physical node of the distributed storage system; the second obtaining module 20 obtains the size of the unit data stream requested to be read and written by the data read-write request issued by the upper layer application within the preset duration; the first adjusting module 30 adjusts the stripe width corresponding to the upper layer application according to the number of the hard disks; the second adjusting module 40 adjusts the stripe depth corresponding to the upper layer application according to the size of the unit data stream; the read-write module 50 executes data read-write operation on each physical node according to the adjusted stripe width and stripe depth. In this way, the stripe depth and stripe width of the distributed storage system can be adapted to the physical configuration change of the host, and the size change of the unit data stream requested to be read and written by the data read-write request issued by the upper layer application, so that the read-write performance of the distributed storage system is improved.
The invention also provides a computer readable storage medium.
The computer readable storage medium of the present invention stores a data read-write program which, when executed by a processor, implements the steps of the data read-write method as described above.
The method implemented when the data read-write program running on the processor is executed may refer to various embodiments of the data read-write method of the present invention, which are not described herein again.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.
The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.
From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a computer readable storage medium (e.g. ROM/RAM, magnetic disk, optical disk) as described above, comprising instructions for causing a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to perform the method according to the embodiments of the present invention.
The foregoing description is only of the preferred embodiments of the present invention, and is not intended to limit the scope of the invention, but rather is intended to cover any equivalents of the structures or equivalent processes disclosed herein or in the alternative, which may be employed directly or indirectly in other related arts.
Claims (10)
1. The data reading and writing method is characterized by comprising the following steps of:
acquiring the number of hard disks contained in each current physical node of the distributed storage system;
acquiring the size of a unit data stream requested to be read and written by a data read-write request issued by an upper layer application within a preset duration;
adjusting the stripe width corresponding to the upper application according to the number of the hard disks;
adjusting the stripe depth corresponding to the upper layer application according to the size of the unit data stream;
executing data read-write operation on each physical node according to the adjusted stripe width and stripe depth;
the step of adjusting the stripe width corresponding to the upper layer application according to the number of hard disks comprises the following steps:
when the upper layer applies the current IO policy as the IO localization policy, acquiring the number of hard disks comprising the physical nodes with the least number of hard disks, and taking the acquired number of hard disks as a target number of hard disks;
detecting whether the data to be read and written corresponding to the IO request is multi-copy data or not;
if yes, the stripe width is adjusted to be smaller than or equal to the target hard disk number divided by the corresponding copy number;
the step of adjusting the stripe depth corresponding to the upper layer application according to the size of the unit data stream includes:
dividing the preset time length into a plurality of time periods, and respectively setting corresponding weight values for the time periods;
multiplying the size of the unit data stream issued in each time period by the corresponding weight value, and then summing to obtain a reference value of the size of the unit data stream;
and adjusting the stripe depth corresponding to the upper layer application according to the reference value.
2. The data read-write method of claim 1, wherein the step of adjusting the stripe width corresponding to the upper layer application according to the number of hard disks comprises:
acquiring an IO strategy applied by the upper layer;
if the IO policy applied by the upper layer is an IO localization policy, acquiring the number of hard disks comprising the physical nodes with the least number of hard disks, and taking the acquired number of hard disks as a target number of hard disks;
and adjusting the width of the stripe corresponding to the upper application to be smaller than or equal to the target hard disk number.
3. The data read-write method as claimed in claim 1, wherein the step of adjusting the stripe depth corresponding to the upper layer application according to the size of the unit data stream comprises:
acquiring the size of a unit data stream with highest occurrence frequency in the preset time length;
determining a preset interval in which the size of the unit data stream with the highest occurrence frequency is located;
inquiring a preset mapping relation between a size interval and a stripe depth of a unit data stream to obtain the stripe depth corresponding to the preset interval, and taking the stripe depth corresponding to the preset interval as a target stripe depth;
and adjusting the stripe depth corresponding to the upper layer application to the target stripe depth.
4. A data reading and writing method according to any one of claims 1 to 3, further comprising:
if an instruction for creating the file fragment group is detected, acquiring the adjusted stripe width and stripe depth;
and creating a file fragment group according to the adjusted stripe width and stripe depth.
5. The data read-write method according to claim 4, wherein the data read-write method further comprises:
detecting whether the distributed storage system meets a preset idle condition or not;
if the distributed storage system meets a preset idle condition, acquiring file fragments which do not meet the adjusted stripe width and stripe depth;
classifying the acquired file fragments according to preset service types;
and according to the adjusted stripe width and stripe depth, rewriting the file fragments under the same classification into the hard disk of each physical node of the distributed storage system.
6. A data reading and writing apparatus, characterized in that the data reading and writing apparatus includes:
the first acquisition module is used for acquiring the number of hard disks contained in each current physical node of the distributed storage system;
the second acquisition module is used for acquiring the size of a unit data stream requested to be read and written by a data read-write request issued by the upper layer application within a preset duration;
the first adjusting module is used for adjusting the stripe width corresponding to the upper layer application according to the number of the hard disks;
the second adjusting module is used for adjusting the stripe depth corresponding to the upper layer application according to the size of the unit data stream;
the read-write module is used for executing data read-write operation on each physical node according to the adjusted stripe width and stripe depth;
the first adjustment module is further configured to obtain the number of hard disks including the physical node with the least number of hard disks when the upper layer applies the current IO policy as the IO localization policy, and take the obtained number of hard disks as a target number of hard disks; detecting whether the data to be read and written corresponding to the IO request is multi-copy data or not; if yes, the stripe width is adjusted to be smaller than or equal to the target hard disk number divided by the corresponding copy number;
the second adjusting module is further configured to divide the preset duration into a plurality of time periods, and set corresponding weight values for the time periods respectively; multiplying the size of the unit data stream issued in each time period by the corresponding weight value, and then summing to obtain a reference value of the size of the unit data stream; and adjusting the stripe depth corresponding to the upper layer application according to the reference value.
7. The data read-write device of claim 6, wherein the first adjustment module is further to:
acquiring an IO strategy applied by the upper layer;
if the IO policy applied by the upper layer is an IO localization policy, acquiring the number of hard disks comprising the physical nodes with the least number of hard disks, and taking the acquired number of hard disks as a target number of hard disks;
and adjusting the width of the stripe corresponding to the upper application to be smaller than or equal to the target hard disk number.
8. The data read-write device of claim 6, wherein the second adjustment module is further configured to:
acquiring the size of a unit data stream with highest occurrence frequency in the preset time length;
determining a preset interval in which the size of the unit data stream with the highest occurrence frequency is located;
inquiring a preset mapping relation between a size interval and a stripe depth of a unit data stream to obtain the stripe depth corresponding to the preset interval, and taking the stripe depth corresponding to the preset interval as a target stripe depth;
and adjusting the stripe depth corresponding to the upper layer application to the target stripe depth.
9. A data reading and writing apparatus, characterized in that the data reading and writing apparatus comprises: memory, a processor and a data read-write program stored on the memory and executable on the processor, which when executed by the processor, implements the steps of the data read-write method according to any one of claims 1 to 5.
10. A computer-readable storage medium, wherein a data read-write program is stored on the computer-readable storage medium, which when executed by a processor, implements the steps of the data read-write method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910572489.2A CN110308875B (en) | 2019-06-27 | 2019-06-27 | Data read-write method, device, equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910572489.2A CN110308875B (en) | 2019-06-27 | 2019-06-27 | Data read-write method, device, equipment and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110308875A CN110308875A (en) | 2019-10-08 |
| CN110308875B true CN110308875B (en) | 2023-07-14 |
Family
ID=68078445
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910572489.2A Active CN110308875B (en) | 2019-06-27 | 2019-06-27 | Data read-write method, device, equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110308875B (en) |
Families Citing this family (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112417350B (en) * | 2020-09-17 | 2023-03-24 | 上海哔哩哔哩科技有限公司 | Data storage adjusting method and device and computer equipment |
| CN112764684A (en) * | 2021-01-22 | 2021-05-07 | 苏州浪潮智能科技有限公司 | Hard disk performance identification method and system of storage system |
| CN112988824B (en) * | 2021-03-15 | 2023-09-26 | 平安科技（深圳）有限公司 | Data generation method, device, computer equipment and storage medium |
| CN113467720B (en) * | 2021-06-29 | 2023-05-23 | 成都佰维存储科技有限公司 | Load balancing method and device, readable storage medium and electronic equipment |
| CN113312009A (en) * | 2021-07-29 | 2021-08-27 | 南京鹏云网络科技有限公司 | Distributed storage system and read-write method thereof |
| CN114124715A (en) * | 2021-11-19 | 2022-03-01 | 中国电信集团系统集成有限责任公司 | Bandwidth determination method and device, electronic equipment and storage medium |
| CN115373609A (en) * | 2022-10-25 | 2022-11-22 | 浪潮电子信息产业股份有限公司 | Task processing method and related equipment |
| CN115712390B (en) * | 2022-11-14 | 2023-05-09 | 安超云软件有限公司 | Method and system for determining available data stripe fragmentation number |
| CN116301660B (en) * | 2023-05-10 | 2023-08-22 | 深圳创新科技术有限公司 | Video recording and writing method, device, equipment and medium based on raid |
| CN117742652A (en) * | 2023-12-27 | 2024-03-22 | 国创芯科技(江苏)有限公司 | Memory bandwidth optimization device and method based on dynamic window statistics |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6055354A (en) * | 1996-04-15 | 2000-04-25 | Discreet Logic Inc. | Data storage apparatus |
| CN104272274A (en) * | 2013-12-31 | 2015-01-07 | 华为技术有限公司 | Data processing method and device in distributed file storage system |
| CN107273048A (en) * | 2017-06-08 | 2017-10-20 | 浙江大华技术股份有限公司 | A kind of method for writing data and device |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101526886B (en) * | 2009-04-27 | 2011-09-07 | 杭州华三通信技术有限公司 | Data operating method of redundant arrays of independent disk and device |
| CN102819408B (en) * | 2012-08-24 | 2016-01-06 | 记忆科技(深圳)有限公司 | The implementation method of the RAID 0 of dynamic adjustment stripe depth and system |
| CN103778222A (en) * | 2014-01-22 | 2014-05-07 | 浪潮(北京)电子信息产业有限公司 | File storage method and system for distributed file system |
| US20160048342A1 (en) * | 2014-08-12 | 2016-02-18 | Facebook, Inc. | Reducing read/write overhead in a storage array |
| JP6477025B2 (en) * | 2015-03-03 | 2019-03-06 | 富士通株式会社 | Storage control device, control method, and control program |
| US9997233B1 (en) * | 2015-10-08 | 2018-06-12 | Rambus Inc. | Memory module with dynamic stripe width |
| CN106293511B (en) * | 2016-07-26 | 2018-12-04 | 北京理工大学 | A kind of dynamic local parallel data layout method towards continuous data storage |
-
2019
- 2019-06-27 CN CN201910572489.2A patent/CN110308875B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6055354A (en) * | 1996-04-15 | 2000-04-25 | Discreet Logic Inc. | Data storage apparatus |
| CN104272274A (en) * | 2013-12-31 | 2015-01-07 | 华为技术有限公司 | Data processing method and device in distributed file storage system |
| CN107273048A (en) * | 2017-06-08 | 2017-10-20 | 浙江大华技术股份有限公司 | A kind of method for writing data and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110308875A (en) | 2019-10-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110308875B (en) | Data read-write method, device, equipment and computer readable storage medium | |
| US11169967B2 (en) | Selective deduplication | |
| US10168946B2 (en) | Extent migration in multi-tier storage systems | |
| US7895398B2 (en) | System and method for dynamically adjusting the caching characteristics for each logical unit of a storage array | |
| US9152336B1 (en) | System and method for LUN adjustment | |
| US8296533B2 (en) | Method and system for deleting low-load allocated virtual server resources | |
| US20170269849A1 (en) | System and method for controlling automated page-based tier management in storage systems | |
| US20090094413A1 (en) | Techniques for Dynamic Volume Allocation in a Storage System | |
| US20160132523A1 (en) | Exploiting node-local deduplication in distributed storage system | |
| US9753668B2 (en) | Method and apparatus to manage tier information | |
| US8578092B2 (en) | Method and apparatus to support determining storage area unit size | |
| US20130185506A1 (en) | Controlling a Storage System | |
| US8307164B2 (en) | Automatic determination of read-ahead amount | |
| US10606510B2 (en) | Memory input/output management | |
| US10228860B2 (en) | Storage optimization based I/O pattern modeling | |
| US9823875B2 (en) | Transparent hybrid data storage | |
| Xu et al. | {SpringFS}: Bridging Agility and Performance in Elastic Distributed Storage | |
| US20210191619A1 (en) | Re-placing data within a mapped-raid environment | |
| US10592137B1 (en) | Method, apparatus and computer program product for determining response times of data storage systems | |
| US10277676B2 (en) | Storage management device, storage management method, and computer-readable recording medium | |
| US9727252B2 (en) | Methods and systems for optimal snapshot distribution within a protection schedule | |
| US11055008B2 (en) | Managing wear balancing in mapped RAID storage systems | |
| CN109947365B (en) | Distributed storage data verification method and device | |
| US9448941B1 (en) | System and method for cache management | |
| US8595438B1 (en) | System and method for improving cache performance |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |