CN110597779A - Data reading and writing method in distributed file system and related device - Google Patents
Data reading and writing method in distributed file system and related device Download PDFInfo
- Publication number
- CN110597779A CN110597779A CN201910894939.XA CN201910894939A CN110597779A CN 110597779 A CN110597779 A CN 110597779A CN 201910894939 A CN201910894939 A CN 201910894939A CN 110597779 A CN110597779 A CN 110597779A
- Authority
- CN
- China
- Prior art keywords
- data
- disk
- write
- data read
- cache
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Abstract
The data reading and writing method in the distributed file system, which is provided by the application, is applied to any storage node in the distributed file system, and comprises the following steps: judging whether a cache disk in a storage node is in failure; the storage nodes comprise cache disks and common disks; if the cache disk fails, receiving a data read-write instruction sent by a client; and executing corresponding data read-write operation by using the common disk according to the data read-write instruction and the preset rule mapping table. Therefore, when the cache disk fails, the method receives the data read-write instruction sent by the client, and then executes corresponding data read-write operation by using the common disk according to the data read-write instruction and the preset rule mapping table, so that the robustness and the usability of the distributed file system can be improved. The application also provides a data read-write device, an electronic device and a computer readable storage medium in the distributed file system, which have the beneficial effects.
Description
Technical Field
The present application relates to the field of data reading and writing in a distributed file system, and in particular, to a method and an apparatus for reading and writing data in a distributed file system, an electronic device, and a computer-readable storage medium.
Background
In order to improve performance and reduce time delay, a distributed file system is often accelerated by using a cache. Therefore, the distributed file system can be easily applied to high-performance application scenes such as database and video monitoring. If a high-performance physical disk, such as an SSD, is used fully, the cost is too high and the capacity at the same cost is small. On the other hand, if a physical disk (normal disk) with low performance is used, such as a STAT HDD, although the cost is low and the capacity is large, the performance does not meet the practical application requirements. Therefore, the two physical disks can be used in combination to exert the advantages of each other.
In the existing distributed file system, a cache disk and a normal disk usually coexist in any storage node. However, when the cache disk fails, the storage node cannot provide services continuously, which greatly reduces the robustness and usability of the distributed file system.
Therefore, how to improve the robustness and usability of the distributed file system is a technical problem that needs to be solved urgently by those skilled in the art.
Disclosure of Invention
The application aims to provide a data reading and writing method and device in a distributed file system, electronic equipment and a computer readable storage medium, which can improve the robustness and usability of the distributed file system.
In order to solve the above technical problem, the present application provides a data reading and writing method in a distributed file system, which is applied to any storage node in the distributed file system, and includes:
judging whether a cache disk in the storage node is in failure or not; wherein the storage nodes comprise the cache disk and a normal disk;
if the cache disk fails, receiving a data read-write instruction sent by a client;
and executing corresponding data read-write operation by using a common disc according to the data read-write instruction and a preset rule mapping table.
Optionally, when the data read-write instruction is a data read instruction, the performing, by using a common disc, a corresponding data read-write operation according to the data read-write instruction and a preset rule mapping table includes:
judging whether first target data corresponding to the data reading instruction are distributed on the common disc or not based on a data distribution record table prestored in the common disc;
if the first target data corresponding to the data reading instruction are distributed on the common disk, reading the first target data from the common disk according to the rule mapping table, and sending the first target data to the client;
and if the first target data corresponding to the data reading instruction are not uniformly distributed on the common disk, sending a return value to the client according to the rule mapping table.
Optionally, when the data read-write instruction is a data write instruction, the performing, by using a common disc, a corresponding data read-write operation according to the data read-write instruction and a preset rule mapping table includes:
judging whether a second target data corresponding to the data writing instruction is stored in the cache disk or the common disk or not based on a data distribution record table prestored in the common disk;
if the second target data corresponding to the data writing instruction is stored in the cache disk or the common disk, writing the second target data into the common disk according to the rule mapping table, and updating the data distribution recording table;
and if the cache disk and the common disk do not store second target data corresponding to the data writing instruction, writing the second target data into the common disk according to the rule mapping table, and updating the data distribution recording table.
Optionally, the data reading and writing method in the distributed file system further includes:
and when the cache disk and/or the common disk have faults, sending alarm information to the network management client.
The present application further provides a data read/write device in a distributed file system, which is applied to any storage node in the distributed file system, and includes:
the cache disk fault judging module is used for judging whether a cache disk in the storage node is in fault or not; wherein the storage nodes comprise the cache disk and a normal disk;
the data read-write instruction receiving module is used for receiving a data read-write instruction sent by the client if the cache disk fails;
and the data read-write operation execution module is used for executing corresponding data read-write operation by using the common disk according to the data read-write instruction and the preset rule mapping table.
Optionally, when the data read-write instruction is a data read instruction, the data read-write operation execution module includes:
a first target data distribution judging unit, configured to judge whether first target data corresponding to the data reading instruction are all distributed on the general disc based on a data distribution record table pre-stored in the general disc;
a first target data reading unit, configured to, if first target data corresponding to the data reading instruction are all distributed on the ordinary disc, read the first target data from the ordinary disc according to the rule mapping table, and send the first target data to the client;
and the return value sending unit is used for sending a return value to the client according to the rule mapping table if the first target data corresponding to the data reading instruction are not uniformly distributed on the common disk.
Optionally, when the data read-write instruction is a data write instruction, the data read-write operation execution module includes:
a second target data storage determining unit, configured to determine, based on a data distribution record table pre-stored in the general disk, whether second target data corresponding to the data write instruction has been stored in the cache disk or the general disk;
a first data distribution record table updating unit, configured to, if second target data corresponding to the data writing instruction has been stored in the cache disk or the general disk, write the second target data into the general disk according to the rule mapping table, and update the data distribution record table;
and a second data distribution record table updating unit, configured to, if no second target data corresponding to the data writing instruction has been stored in the cache disk and the normal disk, write the second target data into the normal disk according to the rule mapping table, and update the data distribution record table.
Optionally, the data reading and writing device in the distributed file system further includes:
and the warning information sending module is used for sending warning information to the network management client side when the cache disk and/or the common disk have faults.
The present application further provides an electronic device, comprising:
a memory and a processor; the memory is used for storing a computer program, and the processor is used for implementing the steps of the data reading and writing method in the distributed file system when executing the computer program.
The present application further provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the computer program implements the steps of the data reading and writing method in the distributed file system.
The data reading and writing method in the distributed file system, which is provided by the application, is applied to any storage node in the distributed file system, and comprises the following steps: judging whether a cache disk in the storage node is in failure or not; wherein the storage nodes comprise the cache disk and a normal disk; if the cache disk fails, receiving a data read-write instruction sent by a client; and executing corresponding data read-write operation by using a common disc according to the data read-write instruction and a preset rule mapping table.
According to the method, when the cache disk fails, a data read-write instruction sent by a client is received, and then corresponding data read-write operation is executed by using the common disk according to the data read-write instruction and a preset rule mapping table, so that the file system can continuously perform read-write operation, the influence of the cache disk failure on data read-write is reduced, and the robustness and usability of the distributed file system can be improved. The application also provides a data read-write device, an electronic device and a computer readable storage medium in the distributed file system, which all have the beneficial effects, and are not described herein again.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a flowchart of a data reading and writing method in a distributed file system according to an embodiment of the present application;
fig. 2 is a block diagram of a data reading/writing device in a distributed file system according to an embodiment of the present disclosure.
Detailed Description
The core of the application is to provide a data reading and writing method in a distributed file system, which can improve the robustness and usability of the distributed file system. At the other core of the application, a data reading and writing device, an electronic device and a computer readable storage medium in a distributed file system are provided.
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
In order to improve performance and reduce time delay, a distributed file system is often accelerated by using a cache. Therefore, the distributed file system can be easily applied to high-performance application scenes such as database and video monitoring. If a high-performance physical disk, such as an SSD, is used fully, the cost is too high and the capacity at the same cost is small. On the other hand, if a physical disk (normal disk) with low performance is used, such as a STAT HDD, although the cost is low and the capacity is large, the performance does not meet the practical application requirements. Therefore, the two physical disks can be used in combination to exert the advantages of each other. In the existing distributed file system, a cache disk and a normal disk usually coexist in any storage node. However, when the cache disk fails, the storage node cannot provide services continuously, which greatly reduces the robustness and usability of the distributed file system.
Referring to fig. 1 specifically, fig. 1 is a flowchart of a data reading and writing method in a distributed file system according to an embodiment of the present application, where the data reading and writing method in the distributed file system is applied to any storage node in the distributed file system, and specifically includes:
s101, judging whether a cache disk in a storage node has a fault or not; the storage nodes comprise cache disks and common disks;
s102, if the cache disk fails, receiving a data read-write instruction sent by a client;
and S103, executing corresponding data read-write operation by using the common disk according to the data read-write instruction and the preset rule mapping table.
The distributed file system performs cache acceleration in order to improve performance and reduce time delay. The cache acceleration is to use a high-performance SSD solid state disk or an Nvme as a cache to accelerate the performance, and when a system is relatively idle, data is only printed from the cache to a large-capacity device such as an HDD. Any storage node in the distributed file system is provided with a cache disk and a common disk, wherein the cache disk is a cache accelerated disk, and the common disk is a physical disk with low performance.
According to the method, when the cache disk fails, a data read-write instruction sent by a client is received, and corresponding data read-write operation is executed by using the common disk according to the data read-write instruction and a preset rule mapping table, so that the robustness and the usability of the distributed file system can be improved.
Based on the foregoing embodiment, in this embodiment, when the data read-write instruction is a data read instruction, according to the data read-write instruction and the preset rule mapping table, the performing, by using the ordinary disc, a corresponding data read-write operation may include:
judging whether first target data corresponding to the data reading instruction are distributed on the common disc or not based on a data distribution record table prestored in the common disc;
if the first target data corresponding to the data reading instruction are distributed on the common disk, reading the first target data from the common disk according to the rule mapping table, and sending the first target data to the client;
and if the first target data corresponding to the data reading instruction are not uniformly distributed on the common disk, sending a return value to the client according to the rule mapping table.
Based on the foregoing embodiment, in this embodiment, when the data read/write command is a data write command, according to the data read/write command and the preset rule mapping table, the performing, by using the ordinary disc, a corresponding data read/write operation may include:
judging whether a second target data corresponding to a data writing instruction is stored in the cache disk or the common disk or not based on a data distribution record table prestored in the common disk;
if the second target data corresponding to the data writing instruction has been stored in the cache disk or the normal disk, writing the second target data into the normal disk according to the rule mapping table, and updating the data distribution recording table;
and if the cache disk and the normal disk do not have the second target data corresponding to the stored data writing instruction, writing the second target data into the normal disk according to the rule mapping table, and updating the data distribution recording table.
Based on the foregoing embodiment, the data reading and writing method in the distributed file system in this embodiment may further include:
and when the cache disk and/or the common disk have faults, sending alarm information to the network management client.
The above is explained in detail in the following with a preferred embodiment, which is as follows:
in order to reduce the cost and obtain the expected performance, the method adopts a shuffle mode with cache acceleration, and a storage node in the shuffle mode needs at least two physical disks, and when a cache disk fails, in order to improve the robustness of the system, the availability of the system should be ensured as much as possible, and special treatment is performed on the condition:
(1) and when the data to be read is stored in the cache disk, the storage node informs the client through a response message, so that the client reads from other storage nodes.
(2) And when the data to be read is not stored in the cache disk, the storage node can normally read the data and return the data to the client.
(3) And when the object to be written is not in the cache disk or the common disk, namely, the object is a creating write operation, directly writing the data into the common disk.
(4) When the object to be written is already in the cache disk or the normal disk, that is, the object is an overwriting operation, the data is also directly written into the normal disk.
It should be noted that any storage node has both a normal disk and a cache disk, and the cache disk is not necessary, and the storage node can work normally only with the normal disk, but not with the cache disk.
The above-mentioned mechanism for implementing non-cache read-write switching based on cache acceleration may be implemented as follows:
storing a table record (namely a data distribution record table) of object data distribution in a common disk, namely, recording whether each part of an object is distributed in a cache disk or a common disk, and updating the table record after data writing succeeds when the data is written; when reading data, the table data is read first and then the data is read from the corresponding position of the ordinary disk or the cache disk.
When the cache disk fails and the storage node is read, checking the table record of the object to be read:
(1) if all the contents of the object are distributed on the common disk, reading the data from the corresponding position of the common disk, and returning the read contents to the client.
(2) If the contents of the objects are partially or completely distributed in the cache disk, the client is directly informed of the contents through a return value, and the client is enabled to read data from other storage nodes.
(3) The client also needs to modify, and when receiving a response message returned by the storage node and finding that the storage node cannot read data, the client needs to check the storage node to which other copies or fragments of the object to be read belong and initiate a read request to the storage node.
When the cache disk fails and the storage node is subjected to write operation, checking the table record of the object to be read:
(1) if the object to be written is already on the cache disk or on the normal disk, i.e. it is an overwrite operation, the data is also written directly to the normal disk and the table record is updated to indicate where the data currently exists.
(2) If the object to be written is not in the cache disk or the normal disk, that is, the object is a create write operation, the data is directly written into the normal disk, and a table record is added to indicate where the data exists currently.
After the cache disk is damaged, the network management system needs to be informed of the replacement through an alarm message, after the replacement, the alarm needs to be removed after the data in the cache disk is recovered, or the alarm can be removed after all the objects containing the data of the cache disk are overwritten.
The number of damaged cache disks cannot exceed the allowable range of the allowable fault domain, otherwise, the possibility of losing data occurs (for example, three copies, the number of damaged storage nodes cannot exceed two; 2+1 erasure, and the number of damaged storage nodes cannot exceed one), and the fault domain is a storage node as an example. Erasure coding: the erasure code technology is used for making certain redundancy on data to increase the reliability of a system, dividing the data to be stored into K blocks, then calculating M check blocks through a coding algorithm, and recovering the original data by using any K of the (K + M) data blocks. By setting the erasure correction rule, the size of storage overhead can be controlled, and the storage utilization rate is effectively improved. For example, when the erasure correction rule K + M of the erasure correction pool is 4+2, the ratio of the data block to the parity block is 4:2, the available capacity of the storage system is 4/(4+2) 2/3 of the total capacity, and data security is ensured in case of no more than 2-node failure. And (4) erasure pool: with erasure codes as the storage pool for data redundancy (all taking 2+1 as an example herein), each copy is said to be a slice. Copy pool: data is stored in multiple identical storage pools, each copy being said to be one copy. A fault domain: the largest area in which one fragment or copy exists uniquely, such as a storage node or a storage cabinet, is referred to herein as a fault domain.
When the normal disk is damaged, the storage node does not continue to provide the read-write service, and informs the network manager to repair and replace the normal disk through the alarm message.
The embodiment of the application provides a read-write mechanism for cache-accelerated switching of a distributed file system, which has the advantages that: (1) the stability and the robustness of the system are enhanced, and the file system can continue to provide services to the outside under the condition that part of cache disks are abnormal. (2) Even if the cache disk is not replaced after a failure of a part of the cache disks, the system can release the alarm to restore to normal after a large number of writing operations are performed by the system, and if the cache disk is not used without replacing the cache disk, the performance may be greatly reduced. (3) When the damaged cache disk is replaced, the system can recover the data in the original damaged disk in a new cache disk through other copies or fragments or directly recover the data in a common disk.
The key points of the application are as follows: when the cache disk is damaged, the read-write operation under various conditions is processed respectively, so that the system can normally provide services, data recovery can be performed after the cache disk is replaced, the system returns to a normal state, meanwhile, the abnormal condition can be detected, the network manager is informed of repairing as soon as possible through an alarm message, the data volume needing to be recovered after the cache disk is replaced is relatively small, and the data recovery speed is improved. The system ensures the continuous use of the system under the condition that part of the cache disk is damaged, reduces the time delay of the system and enhances the robustness and the usability of the system.
In the following, a data reading and writing device, an electronic device, and a computer readable storage medium in a distributed file system provided in an embodiment of the present application are introduced, and the data reading and writing device, the electronic device, and the computer readable storage medium in the distributed file system described below and the data reading and writing method in the distributed file system described above may be referred to correspondingly.
Referring to fig. 2, fig. 2 is a block diagram of a data reading/writing device in a distributed file system according to an embodiment of the present disclosure; the data read-write device in the distributed file system is applied to any storage node in the distributed file system and comprises the following components:
a cache disk failure determination module 201, configured to determine whether a cache disk in a storage node fails; the storage nodes comprise cache disks and common disks;
a data read-write instruction receiving module 202, configured to receive a data read-write instruction sent by a client if the cache disk fails;
and the data read-write operation execution module 203 is configured to execute corresponding data read-write operation by using the normal disc according to the data read-write instruction and the preset rule mapping table.
Based on the foregoing embodiment, in this embodiment, when the data read/write instruction is a data read instruction, the data read/write operation executing module 203 includes:
the first target data distribution judging unit is used for judging whether the first target data corresponding to the data reading instruction are distributed on the common disc or not based on a data distribution recording table prestored in the common disc;
the first target data reading unit is used for reading the first target data from the ordinary disc according to the rule mapping table if the first target data corresponding to the data reading instruction are distributed on the ordinary disc, and sending the first target data to the client;
and the return value sending unit is used for sending the return value to the client according to the rule mapping table if the first target data corresponding to the data reading instruction are not uniformly distributed on the ordinary disc.
Based on the foregoing embodiment, in this embodiment, when the data read/write instruction is a data write instruction, the data read/write operation executing module 203 includes:
the second target data storage judging unit is used for judging whether the cache disk or the normal disk stores second target data corresponding to the data writing instruction or not based on a data distribution record table prestored in the normal disk;
the first data distribution record table updating unit is used for writing second target data into the common disc according to the rule mapping table and updating the data distribution record table if the second target data corresponding to the data writing instruction has been stored in the cache disc or the common disc;
and the second data distribution record table updating unit is used for writing the second target data into the common disc according to the rule mapping table and updating the data distribution record table if the cache disc and the common disc do not have the second target data corresponding to the stored data writing instruction.
Based on the foregoing embodiment, the data reading and writing apparatus in the distributed file system in this embodiment further includes:
and the warning information sending module is used for sending warning information to the network management client side when the cache disk and/or the common disk have faults.
The present application further provides an electronic device, comprising: a memory and a processor; the memory is used for storing a computer program, and the processor is used for implementing the steps of the data reading and writing method in the distributed file system according to any of the above embodiments when executing the computer program.
The present application further provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the steps of the data reading and writing method in the distributed file system according to any of the above embodiments are implemented.
The computer-readable storage medium may include: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The embodiments are described in a progressive manner in the specification, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. For the system provided by the embodiment, the description is relatively simple because the system corresponds to the method provided by the embodiment, and the relevant points can be referred to the method part for description.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The data reading and writing method and device, the electronic device, and the computer-readable storage medium in the distributed file system provided by the present application are described in detail above. The principles and embodiments of the present application are explained herein using specific examples, which are provided only to help understand the method and the core idea of the present application. It should be noted that, for those skilled in the art, it is possible to make several improvements and modifications to the present application without departing from the principle of the present application, and such improvements and modifications also fall within the scope of the claims of the present application.
Claims (10)
1. A data read-write method in a distributed file system is characterized in that the method is applied to any storage node in the distributed file system and comprises the following steps:
judging whether a cache disk in the storage node is in failure or not; wherein the storage nodes comprise the cache disk and a normal disk;
if the cache disk fails, receiving a data read-write instruction sent by a client;
and executing corresponding data read-write operation by using the common disk according to the data read-write instruction and a preset rule mapping table.
2. The method according to claim 1, wherein when the data read/write command is a data read command, the performing a corresponding data read/write operation by using the normal disk according to the data read/write command and a preset rule mapping table includes:
judging whether first target data corresponding to the data reading instruction are distributed on the common disc or not based on a data distribution record table prestored in the common disc;
if the first target data corresponding to the data reading instruction are distributed on the common disk, reading the first target data from the common disk according to the rule mapping table, and sending the first target data to the client;
and if the first target data corresponding to the data reading instruction are not uniformly distributed on the common disk, sending a return value to the client according to the rule mapping table.
3. The method according to claim 1, wherein when the data read/write command is a data write command, the performing a corresponding data read/write operation by using the normal disk according to the data read/write command and a preset rule mapping table includes:
judging whether a second target data corresponding to the data writing instruction is stored in the cache disk or the common disk or not based on a data distribution record table prestored in the common disk;
if the second target data corresponding to the data writing instruction is stored in the cache disk or the common disk, writing the second target data into the common disk according to the rule mapping table, and updating the data distribution recording table;
and if the cache disk and the common disk do not store second target data corresponding to the data writing instruction, writing the second target data into the common disk according to the rule mapping table, and updating the data distribution recording table.
4. The method for reading and writing data in the distributed file system according to claim 1, further comprising:
and when the cache disk and/or the common disk have faults, sending alarm information to the network management client.
5. A data read-write device in a distributed file system is applied to any storage node in the distributed file system, and comprises:
the cache disk fault judging module is used for judging whether a cache disk in the storage node is in fault or not; wherein the storage nodes comprise the cache disk and a normal disk;
the data read-write instruction receiving module is used for receiving a data read-write instruction sent by the client if the cache disk fails;
and the data read-write operation execution module is used for executing corresponding data read-write operation by utilizing the common disk according to the data read-write instruction and a preset rule mapping table.
6. The apparatus according to claim 5, wherein when the data read/write command is a data read command, the data read/write operation execution module includes:
a first target data distribution judging unit, configured to judge whether first target data corresponding to the data reading instruction are all distributed on the general disc based on a data distribution record table pre-stored in the general disc;
a first target data reading unit, configured to, if first target data corresponding to the data reading instruction are all distributed on the ordinary disc, read the first target data from the ordinary disc according to the rule mapping table, and send the first target data to the client;
and the return value sending unit is used for sending a return value to the client according to the rule mapping table if the first target data corresponding to the data reading instruction are not uniformly distributed on the common disk.
7. The apparatus according to claim 5, wherein when the data read/write command is a data write command, the data read/write operation execution module includes:
a second target data storage determining unit, configured to determine, based on a data distribution record table pre-stored in the general disk, whether second target data corresponding to the data write instruction has been stored in the cache disk or the general disk;
a first data distribution record table updating unit, configured to, if second target data corresponding to the data writing instruction has been stored in the cache disk or the general disk, write the second target data into the general disk according to the rule mapping table, and update the data distribution record table;
and a second data distribution record table updating unit, configured to, if no second target data corresponding to the data writing instruction has been stored in the cache disk and the normal disk, write the second target data into the normal disk according to the rule mapping table, and update the data distribution record table.
8. The apparatus for reading from and writing to data in a distributed file system according to claim 5, further comprising:
and the warning information sending module is used for sending warning information to the network management client side when the cache disk and/or the common disk have faults.
9. An electronic device, comprising:
a memory and a processor; wherein the memory is used for storing a computer program, and the processor is used for implementing the steps of the data reading and writing method in the distributed file system according to any one of claims 1 to 4 when executing the computer program.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program which, when executed by a processor, implements the steps of a method for reading and writing data in a distributed file system according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910894939.XA CN110597779A (en) | 2019-09-20 | 2019-09-20 | Data reading and writing method in distributed file system and related device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910894939.XA CN110597779A (en) | 2019-09-20 | 2019-09-20 | Data reading and writing method in distributed file system and related device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110597779A true CN110597779A (en) | 2019-12-20 |
Family
ID=68861956
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910894939.XA Pending CN110597779A (en) | 2019-09-20 | 2019-09-20 | Data reading and writing method in distributed file system and related device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110597779A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111176898A (en) * | 2019-12-29 | 2020-05-19 | 浪潮电子信息产业股份有限公司 | Distributed file system MDS (maintenance description Server) fault switching method, device, equipment and medium |
| CN114115697A (en) * | 2020-08-26 | 2022-03-01 | 浙江宇视科技有限公司 | Cloud storage data processing method and device, electronic equipment and storage medium |
| CN114442956A (en) * | 2022-01-27 | 2022-05-06 | 苏州浪潮智能科技有限公司 | Data write-in processing method, device, equipment and medium for double storage pools |
| CN116312725A (en) * | 2023-05-16 | 2023-06-23 | 苏州浪潮智能科技有限公司 | Data storage method and device, electronic equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100228903A1 (en) * | 2009-03-03 | 2010-09-09 | Vmware, Inc. | Block Map Based I/O Optimization for Storage Virtual Appliances |
| CN105242881A (en) * | 2015-10-12 | 2016-01-13 | 创新科软件技术（深圳）有限公司 | Distributed storage system and data read-write method for same |
| CN108197159A (en) * | 2017-12-11 | 2018-06-22 | 厦门集微科技有限公司 | Digital independent, wiring method and device based on distributed file system |
| CN109656480A (en) * | 2018-12-12 | 2019-04-19 | 广东浪潮大数据研究有限公司 | A kind of management method, system and the device of SSD cache pool |
-
2019
- 2019-09-20 CN CN201910894939.XA patent/CN110597779A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100228903A1 (en) * | 2009-03-03 | 2010-09-09 | Vmware, Inc. | Block Map Based I/O Optimization for Storage Virtual Appliances |
| CN105242881A (en) * | 2015-10-12 | 2016-01-13 | 创新科软件技术（深圳）有限公司 | Distributed storage system and data read-write method for same |
| CN108197159A (en) * | 2017-12-11 | 2018-06-22 | 厦门集微科技有限公司 | Digital independent, wiring method and device based on distributed file system |
| CN109656480A (en) * | 2018-12-12 | 2019-04-19 | 广东浪潮大数据研究有限公司 | A kind of management method, system and the device of SSD cache pool |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111176898A (en) * | 2019-12-29 | 2020-05-19 | 浪潮电子信息产业股份有限公司 | Distributed file system MDS (maintenance description Server) fault switching method, device, equipment and medium |
| CN114115697A (en) * | 2020-08-26 | 2022-03-01 | 浙江宇视科技有限公司 | Cloud storage data processing method and device, electronic equipment and storage medium |
| CN114115697B (en) * | 2020-08-26 | 2024-03-22 | 浙江宇视科技有限公司 | Cloud storage data processing method and device, electronic equipment and storage medium |
| CN114442956A (en) * | 2022-01-27 | 2022-05-06 | 苏州浪潮智能科技有限公司 | Data write-in processing method, device, equipment and medium for double storage pools |
| CN114442956B (en) * | 2022-01-27 | 2023-08-25 | 苏州浪潮智能科技有限公司 | Double-storage-pool data writing processing method, device, equipment and medium |
| CN116312725A (en) * | 2023-05-16 | 2023-06-23 | 苏州浪潮智能科技有限公司 | Data storage method and device, electronic equipment and storage medium |
| CN116312725B (en) * | 2023-05-16 | 2023-08-15 | 苏州浪潮智能科技有限公司 | Data storage method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110597779A (en) | Data reading and writing method in distributed file system and related device | |
| US7167880B2 (en) | Method and apparatus for avoiding journal overflow on backup and recovery system using storage based journaling | |
| US7103811B2 (en) | Mechanisms for detecting silent errors in streaming media devices | |
| US20070208918A1 (en) | Method and apparatus for providing virtual machine backup | |
| US6618794B1 (en) | System for generating a point-in-time copy of data in a data storage system | |
| US7490103B2 (en) | Method and system for backing up data | |
| US11068414B2 (en) | Process for maintaining data write ordering through a cache | |
| EP2425344B1 (en) | Method and system for system recovery using change tracking | |
| EP2908254A1 (en) | Data redundancy implementation method and device | |
| US6701455B1 (en) | Remote copy system with data integrity | |
| CN101887351B (en) | Fault-tolerance method and system for redundant array of independent disk | |
| JP2005071119A (en) | Method for switching system, method for creating replica, and disk device | |
| US10901646B2 (en) | Update of RAID array parity | |
| US6363457B1 (en) | Method and system for non-disruptive addition and deletion of logical devices | |
| CN111176900A (en) | Distributed storage system and data recovery method, device and medium thereof | |
| CN112506710A (en) | Distributed file system data repair method, device, equipment and storage medium | |
| CN109992445B (en) | Processing method and device for modifying write operation, electronic equipment and storage medium | |
| US20180217897A1 (en) | Online volume repair | |
| CN115599607A (en) | Data recovery method of RAID array and related device | |
| US20100169572A1 (en) | Data storage method, apparatus and system for interrupted write recovery | |
| US20220374310A1 (en) | Write request completion notification in response to partial hardening of write data | |
| CN113485872A (en) | Fault processing method and device and distributed storage system | |
| CN111104256A (en) | Data reading method, device, equipment and storage medium | |
| CN105893172A (en) | Hard disk data recovery method and system | |
| CN114443368B (en) | redundant data processing method, device, system and medium of raid system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191220 |
|
| RJ01 | Rejection of invention patent application after publication |