CN115756317A - Storage method, storage device, electronic equipment and storage medium of shingled magnetic disk - Google Patents

Abstract

The application discloses a storage method of a shingled disk, a shingled disk storage device, electronic equipment and a computer storage medium. The storage method of the shingled magnetic disk comprises the following steps: initializing a shingled disk to divide the shingled disk into a first storage area and a second storage area; and responding to the data reading and writing task, reading and writing data of the shingled magnetic disk, recording the data by using the first storage area in the reading and writing process, and recording metadata information, log information and cache data of the data by using the second storage area. By the method, the read-write performance and the safety performance of the SMR disk can be improved.

Description

Storage method, storage device, electronic equipment and storage medium of shingled magnetic disk

Technical Field

The present application relates to the field of data storage technologies, and in particular, to a storage method for a shingled disk, a shingled disk storage apparatus, an electronic device, and a computer storage medium.

Background

The data volume under the big data era is exponentially increased, and the hard disk is used as an economic and efficient storage medium and still plays the role of a medium-current column in each current technical field. Currently, most hard disks are still using Conventional Magnetic Recording (CMR) technology, but the technology is about to reach its storage density limit, and it is highly desirable to introduce new technology to overcome the limit so as to continue to increase the storage capacity.

The birth of Shingled Magnetic Recording (SMR) discs holds more tracks in the same area, increasing storage density and further reducing the cost per unit storage capacity of the disc.

Although SMR disks control cost, SMR disks have read-write limitations: dividing the storage space of the whole SMR disk by taking 256M as a storage unit zone, wherein each zone has a Write Pointer (WP), data must be written backwards from the position of the current WP in sequence, the data after the WP cannot be read, and the data cannot be read and written across the zones. When writing data in a zone, the current start writing position must be guaranteed to be an integer multiple of 4K. There is a certain risk of the read-write performance of SMR disks due to the above-described sequential writing characteristic of SMR disks.

Disclosure of Invention

The technical problem that this application mainly solved is how to improve the read-write performance and the security performance of promoting SMR dish.

In order to solve the above technical problem, the present application provides a storage method for a shingled disk. The storage method of the shingled magnetic disk comprises the following steps: initializing a shingled disk to divide the shingled disk into a first storage area and a second storage area; and responding to the data reading and writing task, reading and writing data of the shingled magnetic disk, recording the data by using the first storage area in the reading and writing process, and recording metadata information, log information and cache data of the data by using the second storage area.

To solve the above technical problem, the present application provides a shingled magnetic disk storage device. The shingled disk storage unit comprises: a shingled disk for storing data; and the file system is connected with the shingled magnetic disk and used for realizing the reading and writing of the data on the shingled magnetic disk by adopting the storage method.

In order to solve the technical problem, the application provides an electronic device. The electronic device includes: the storage device comprises a processor and a memory, wherein program data are stored in the memory, and the processor is used for executing the program data to realize the storage method.

In order to solve the above technical problem, the present application provides a computer-readable storage medium storing program data, which can be executed by a processor to implement the above storage method.

The beneficial effect of this application is: the storage method of the shingled magnetic disk initializes the shingled magnetic disk to divide the shingled magnetic disk into a first storage area and a second storage area; when data is read and written, the first storage area is used for recording the data, and the second storage area is used for recording metadata information, log information and cache data of the data. By the method, metadata and real data can be managed in batch writing through the log type storage mode of the second storage area, complexity of management data of the first partition can be reduced, SMR disk data writing performance is improved, reliability of a file system is improved, and integrity of cluster data can be guaranteed.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

FIG. 1 is a schematic flow chart diagram illustrating an embodiment of a shingled disk storage method according to the present application;

FIG. 2 is a schematic structural diagram of an embodiment of partitions of a shingled disk according to the present application;

FIG. 3 is a flowchart illustrating an embodiment of a method for initializing a shingled disk in the storage method for a shingled disk according to the present application;

FIG. 4 is a schematic structural diagram of another embodiment of partitions of a shingled disk of the present application;

FIG. 5 is a schematic flowchart of an embodiment of a method for writing data to a shingled disk in the storage method for shingled disks according to the present application;

FIG. 6 is a detailed flowchart of step S52 in the embodiment of FIG. 5;

FIG. 7 is a schematic structural diagram of an embodiment of shingled disk data unloading according to the present application;

FIG. 8 is a schematic structural diagram of another embodiment of the shingled disk data unloading of the present application;

FIG. 9 is a schematic structural diagram of another embodiment of the shingled disk data unloading of the present application;

FIG. 10 is a schematic flowchart of another embodiment of a method for writing data to a shingled disk in the storage method for shingled disks according to the present application;

FIG. 11 is a schematic flowchart of an embodiment of a method for reading data from a shingled disk in the storage method for a shingled disk according to the present application;

FIG. 12 is a schematic structural diagram of an embodiment of a shingled disk storage apparatus of the present application;

FIG. 13 is a schematic flow chart illustrating operation of the shingled disk storage apparatus of the embodiment of FIG. 12;

FIG. 14 is a schematic diagram of an embodiment of a distributed storage system of the present application;

FIG. 15 is a schematic structural diagram of an embodiment of an electronic device of the present application;

FIG. 16 is a schematic structural diagram of an embodiment of a computer-readable storage medium according to the present application.

Detailed Description

In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system structures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

It is also to be understood that the terminology used in the description of the present application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in the specification of the present application and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

It should be further understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

As used in this specification and the appended claims, the term "if" may be interpreted contextually as "when", "upon" or "in response to a determination" or "in response to a detection". Similarly, the phrase "if it is determined" or "if a [ described condition or event ] is detected" may be interpreted contextually to mean "upon determining" or "in response to determining" or "upon detecting [ described condition or event ]" or "in response to detecting [ described condition or event ]".

The technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only some embodiments of the present application, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

In a plurality of technical fields, such as the technical field of video monitoring, along with the expansion of the monitoring service scale, the data volume also shows exponential growth, and the distributed storage cluster is difficult to avoid the situation of insufficient capacity. In order to meet the increasing storage requirements and reduce the capacity expansion cost to a certain extent, the adoption of SMR disk storage data is a technical problem to be solved urgently in the field of current video monitoring.

The storage method of the shingled disk, the shingled disk storage device, the electronic equipment and the computer storage medium are provided based on the sequential writing characteristic of the SMR disk, so that the data writing performance of the SMR disk is improved, the reliability of a file system is improved, and the integrity of cluster data can be guaranteed.

The technical scheme of the application can be used for a distributed storage cluster system, based on the SMR disk, the cluster signaling is enriched by optimizing the Data and metadata storage mode of Data nodes (Data nodes, DN) and the storage format of Data block information, the fault tolerance reliability of the Data nodes is improved in a log type Data caching mode, the complexity of the Data management of the storage area of the SMR disk under an abnormal scene can be reduced, and therefore the availability and the reliability of a file system can be effectively improved.

The present application first proposes a storage method for a shingled disk, as shown in fig. 1, where fig. 1 is a schematic flow chart of an embodiment of the storage method for a shingled disk, and the storage method of the embodiment specifically includes the following steps:

step S11: the shingled disk is initialized to divide the shingled disk into a first storage area and a second storage area.

The first storage area is an SMR partition, the SMR partition takes a storage unit zone as a management unit, and each zone in the SMR partition supports out-of-order writing and reading operations but only supports sequential writing operations, namely, data can be written from front to back sequentially according to the sequence; and each zone supports ZAC/ZBC commands to acquire, read, write and clear the SMR partition state.

The second storage area is a CMR (Conventional Magnetic Recording disk) partition, the space size of the second storage area is smaller than that of the first storage area, and the second storage area supports out-of-order write operation and random read and write.

Before DN starts, file system carries out format operation to SMR disk, divides SMR disk in DN into designated partition: a second storage area, i.e., the CMR partition, and a first storage area, i.e., the SMR partition, as shown in FIG. 2; the CMR partition is used to record metadata information, log data, cache data, and the like of real data, and the SMR partition is used to record the real data.

Alternatively, the present embodiment may implement initialization of the shingled magnetic disk by the method shown in fig. 3, where the method of the present embodiment includes steps S31 to S35.

Step S31: the shingled disk is divided into a first storage area and a second storage area.

In the initialization method of the present embodiment, after the SMR disk is divided into the SMR partition and the CMR partition, the SMR partition and the CMR partition are further subdivided in steps S32 to S35.

Step S32: the first storage area is divided into a plurality of storage units.

The SMR partition is divided into a plurality of zones, the SMR partition has a zone as a management unit, and a space size of each zone is generally 256MB.

Step S33: the second storage area is divided into a metadata description area, a journal cache area, a data cache area and a full metadata area.

The execution order of step S32 and step S33 may not be limited.

The CMR partition is divided into a metadata description area, a journal cache area, a data cache area, and a full metadata area, as shown in FIG. 4.

The metadata description area is mainly responsible for describing the disk type, the total capacity of the disk, the size of the CMR partition, the offset position of the log area and the size of the log area; information such as a start position of the full metadata area, a size of the full metadata area, a start position of the SMR partition, a size of the SMR partition, a total number of zones in the SMR partition, a size of the zones, and the like.

A log area: the log area is divided into a log cache area and a data cache area. The log cache region is mainly responsible for recording metadata in a log storage mode, serializing the metadata periodically and storing the metadata to the full metadata region; the data cache region is mainly responsible for caching real data corresponding to the metadata written into the log cache region, the DN can start a monitor, and when the data in the data cache region reaches integral multiple of the size of the zone, the cached real data is transferred to the SMR partition. In practical application, a threshold value can be set according to the size of the data cache region, the threshold value is controlled by the configuration file, and the configuration file is loaded when DN is started to complete the initialization process.

The full metadata area is mainly responsible for storing periodically serialized metadata information and a transferred log file in the log cache area, and the metadata information records related information such as offset position, data size and creation time of an SMR partition where real data are located.

Optionally, as shown in fig. 4, the second storage area further includes: the device comprises a main Boot sector (MBR) and a plurality of reserved areas R, wherein the sum of the capacity of the main Boot sector and the capacity of a metadata description area is equal to the capacity of a storage unit, and the capacities of a log cache area, a data cache area and a full metadata area are integral multiples of the capacity of the storage unit, so that file system management is facilitated, read and write data are kept aligned at 4K, and the file system is convenient to position and address.

And reserved areas R are arranged between the main guide sector and the log cache area, between the log cache area and the data cache area and between the data cache area and the full metadata area. Because the division of each partition in the CMR partition can not exactly meet the integral multiple of the zone capacity and requires 4K alignment, the filling can be carried out through the reserved area R, and the starting position of each partition in the CMR partition can be ensured to be the integral multiple of the zone capacity and 4K alignment.

Step S34: and writing the disk information, the information of the storage unit, the information of the metadata description area, the information of the log cache area, the information of the data cache area and the information of the full metadata area of the shingled disk into the metadata description area.

In the initialization stage of the SMR disk, the SMR disk is partitioned according to the above scheme by the file system, and the disk information, the zone information, and the type, size, and address of each partition are written in the metadata description area. The MBR + metadata description area occupies one zone, and the rest of the partitions are integer multiples of the zone, so that file system management is facilitated, read-write data are kept aligned at 4K, and the file system is convenient to position and address.

Step S35: and loading the information written into the metadata description area into a memory, and reporting to a metadata service node.

The DN collects basic information of each SMR disk, then loads information of a metadata description area of the SMR disk into a memory, and reports the information to a metadata Service (MDS) (as shown in fig. 14), when a subsequent Service Data, that is, a real Data is written, a user applies for a space to the MDS through a client SDK, writes Data according to the fed-back DN address information, and the capacity management module provides a write-in space for the written Data.

Step S12: and responding to the data reading and writing task, reading and writing data of the shingled magnetic disk, recording the data by using the first storage area in the reading and writing process, and recording metadata information, log information and cache data of the data by using the second storage area.

Optionally, the data read-write task includes a data write task, and in response to the data write task, metadata information is constructed for the data in the temporary memory, and the constructed metadata is cached in the log cache region, and the data is cached in the data cache region.

In another embodiment, the method for writing data to the shingled magnetic disk in step S12 can also be implemented by the method shown in fig. 5, and the method of this embodiment includes steps S51 to S53.

Step S51: and responding to the data writing task, constructing metadata information for the data in the temporary memory, caching the constructed metadata in the log cache region, and caching the data in the data cache region.

The file system divides two memory spaces, namely a temporary memory and a global memory, for the metadata storage. When the DN receives a data writing task, the file system acquires the position of the log area from the memory, constructs metadata information for real data in the temporary memory, caches the constructed metadata in the log cache area, and caches the real data in the data cache area. The metadata in the temporary memory records attributes of real data in the data cache region, such as creation time, data size, position offset, and the like.

Step S52: and transferring the data cached in the data cache region to the first storage region in response to the data volume or the data cache duration of the data cache region being greater than the threshold value.

Furthermore, the CMR partition of the application is divided into a log area used for gathering the cache data, and when the cache data reaches a threshold value, the cache data is stored in batch, so that the data stored in the SMR partition is integral multiple of the zone. The method has great advantages when small IO is faced, and by carrying out aggregation and unloading on a small amount of data written for many times, the addressing frequency of the SMR partition is reduced, meanwhile, the generation of space fragments is reduced, and the space utilization rate of the SMR partition is improved.

Alternatively, the present embodiment may implement step S52 by a method as shown in fig. 6, where the method of the present embodiment includes step S61 to step S64.

Step S61: and calculating the remainder of the data quantity in the data cache region and the capacity of the storage unit.

Step S62: and in response to the fact that the data amount of the data cache region is larger than the capacity of the storage unit and the remainder is larger than or equal to the duration threshold, the data of the data cache region is stored to the first storage region after being filled to be integral multiple of the capacity of the storage unit.

Step S63: and in response to that the data amount of the data cache region is larger than the capacity of the storage unit and the remainder is zero, directly transferring the data of the data cache region to the first storage region.

Step S64: responding to the fact that the data volume of the data cache region is larger than the capacity of the storage unit and the remainder is smaller than the duration threshold value, reserving a first part of data in the data cache region, and transferring another part of data in the data cache region to the first storage region; the data quantity of the first part of data is a remainder.

The duration threshold is a preset percentage of the capacity of the storage unit.

Next, step S61 to step S64 will be described together.

For the threshold, the present embodiment may be set from two dimensions of time and data volume:

1) Data volume dimension: the dimensionality mainly aims at a scene with large service volume, and when the data volume of the data cache region reaches an integral multiple of a threshold value TCache, the data cache region can be transferred in time, so that the cache region is prevented from being written through.

2) The time dimension is as follows: the dimension is mainly used for transferring the data when the data caching duration reaches a duration threshold TTime aiming at a scene with small service volume.

In order to ensure that the transferred data is an integral multiple of the zone and reduce the waste of zone space in the SMR partition, the method can be divided into the following three cases, wherein Vcache represents the amount of data cached in the data cache region; vzone represents zone capacity; vremia stands for Vcache% Vzone; the duration threshold Tremain represents the percentage of zone capacity.

a) When Vcache is greater than Vzone, vremia is greater than threshold Tremain (example: vzone x 80%), the cached data is flushed to the SMR partition after being filled to an integer multiple of zone, as shown in fig. 7.

b) When Vcache is greater than Vzone, vremia is equal to 0, which indicates that the amount of buffered data is an integer multiple of zone, and the data can be directly transferred, as shown in fig. 8.

c) When Vcache is greater than Vzone, and vremia is smaller than threshold duration Tremain, then the data of vremia is retained in the buffer area, and the data except vremia in Vcache is transferred to the SMR partition, as shown in fig. 9. The reserved data is transferred with the data written into the cache in the next cycle.

Step S53: and constructing metadata of the data transferred to the first storage area in the global memory, and updating the data in the global memory to the full-amount metadata area in a log storage mode so as to generate a metadata log of the transferred data in the full-amount metadata area.

When the data volume or the data storage duration of the data cache region reaches a set threshold value, the file system transfers the cached monitoring data to the SMR partition, the file system constructs metadata of the transferred data in the global memory, and then updates the data of the global memory to the full-volume metadata region in a log storage mode.

The execution order of step S51, step S52 and step S53 is not limited.

Two service scenarios are considered in this embodiment: a scenario with a small amount of traffic and a scenario with a large amount of traffic. Under the scene of small traffic, the backlog of the cache data is long, and in order to guarantee the safety of the data, a threshold value is set to write the cache data into a local disk. And for the scene with large service volume, selectively unloading the data according to whether the cached data volume exceeds a threshold value or not, and avoiding the data cache region from being written through. And the data is persisted according to the measures set in the two scenes, so that the integrity of the data is ensured.

In another embodiment, the method for writing data to the shingled magnetic disk in step S12 can also be implemented by the method shown in fig. 10, and the method of this embodiment includes steps S101 to S105.

Step S101: and responding to the data writing task, constructing metadata information for the data in the temporary memory, caching the constructed metadata in the log cache region, and caching the data in the data cache region.

Detailed description may refer to step S51.

Step S102: and transferring the data cached in the data cache region to the first storage region in response to the data volume or the data cache duration of the data cache region being greater than the threshold value.

The detailed description may refer to step S52.

Step S103: and constructing metadata of the data transferred to the first storage area in the global memory, and updating the data in the global memory to the full-amount metadata area in a log storage mode so as to generate a metadata log of the transferred data in the full-amount metadata area.

The detailed description may refer to step S53.

Step S104: checkpoints the metadata log.

Step S105: and deleting the corresponding metadata from the temporary memory and the data which is transferred out from the log cache region based on the check point.

In this embodiment, after the data is completely transferred, a Check Point (CP) is made on the metadata log corresponding to the transferred data in the full metadata area through step S104 and step S105, and a corresponding log file (equivalent to a snapshot function) is generated; the log file is also stored in the full metadata area; the corresponding metadata in the temporary memory and the data transferred and stored in the log cache region can be cleared based on the check point, and the vacated space is used for recording the newly written data.

The method of the present embodiment may further include step S106 to step S108.

Step S106: when an exception occurs, the metadata log is looked up for the most recent checkpoint.

Step S107: and loading the full amount of metadata information before the latest checkpoint into the global memory.

Step S108: and deleting the data which is transferred after the latest checkpoint from the first storage area.

When abnormity such as power failure occurs, the log file is read, the latest CP point is searched, the previous full metadata information of the CP point is loaded into the global memory and is recovered to the state before the abnormity, the data in the SMR area does not need to be scanned, and the quick recovery capability of the file system is improved. And the data transferred after the CP point is subjected to corresponding deletion operation, such as deletion from the first storage area.

The embodiment sets a checkpoint on the metadata log, which can ensure consistency of the monitoring data and the metadata. The metadata in the journal cache records attributes of the real data in the data cache, and the full metadata area records attributes of the real data in the SMR partition. Therefore, in the unloading stage, when the real data is unloaded from the cache region to the SMR region, the attribute of the real data may be changed, and in order to ensure that the metadata in the memory is consistent with the attribute of the real data, the embodiment writes the metadata and the real data into the disk in a transactional manner. And if the exception occurs in the unloading process, all the real data written in the unloading process are regarded as invalid.

And the checkpoint setting of the embodiment can also shorten the data recovery time when data damage occurs. Writing data and a log file are two main IO operations, in order to ensure the writing performance, before reporting the written data, ensuring that the logs are written into the log file, storing real data and metadata in a cache, and then transferring and writing the real data and the metadata into a designated partition in batches, namely, the log writing and reporting operations are synchronous, and the data writing and reporting operations are asynchronous. When the service is crashed or the server is abnormally restarted, in order to ensure that all data in the cache and the memory are written into the full-amount metadata area and the SMR partition, the CP can be used for restoring the service to a normal state before the crash according to the log, only the cache data which is not transferred to the designated partition needs to be restored, and all the data does not need to be restored.

The embodiment periodically sets a CP for the persistent data, records the current correct persistent data, and the CP is recorded in the log file. When an abnormal power failure scene occurs, the log file is loaded to find the latest CP, the log file is scanned and loaded to quickly construct metadata information of real data in the memory, and compared with a method of constructing metadata by scanning real data of an SMR partition, the storage node can quickly respond to services, and the fault tolerance of a cluster is improved.

In another embodiment, the method for reading data from the shingled disk in step S12 can be implemented by the method shown in fig. 11, and the method of this embodiment includes step S111 to step S114.

Step S111: and responding to the data reading task, and searching the position of the metadata of the data in the memory according to the data index.

Step S112: and reading data corresponding to the metadata in the data cache region in response to the fact that the metadata is only in the temporary memory.

Step S113: and responding to the fact that the metadata is only in the global memory, and reading data corresponding to the metadata in the first storage area.

Step S114: and respectively reading corresponding first data from the data cache region and corresponding second data from the first storage region in response to the fact that the metadata is partially in the temporary memory and partially in the global memory, and merging the first data and the second data.

Step S111 to step S114 will be described together.

For example, in the field of video monitoring, data from a front-end camera is mostly read in a non-real-time manner, the data is generally stored in an SMR disk from hot to cold, the data is usually read only in an emergency, for the type of data, less space is used for storing more data, or a small number of clusters are used for managing more data, so that video recording is conveniently checked at a later stage, and the data query efficiency is improved.

When the file system receives a data reading task, the file system searches the position of the real data in the metadata information recorded in the memory according to the data index. In this embodiment, the temporary memory is mainly used to record the newly written "hot data", and the global memory is used to record the "cold data" after the dump. The distribution of data can be divided into three cases: (1) all in the data buffer area; (2) all in SMR partitions; and (3) respectively existing in the data cache region and the SMR region. When data is read, target metadata can be searched in a temporary memory and a global memory, and the following three conditions exist in the process of reading the data:

and the metadata is only stored in the temporary memory, and corresponding data in the data cache region is read.

And reading corresponding data in the SMR partition only in the global memory by using the metadata.

And the metadata is partially in the temporary memory and partially in the global memory, and corresponding data are respectively read from the data cache region and the SMR partition and are combined into complete data.

The execution subject of the above embodiment is a file system.

The present application further provides a shingled magnetic disk storage apparatus, as shown in fig. 12, fig. 12 is a schematic structural diagram of an embodiment of the shingled magnetic disk storage apparatus of the present application, where the shingled magnetic disk storage apparatus of the present embodiment includes a shingled magnetic disk 121 and a file system 122 connected to the shingled magnetic disk 121, the shingled magnetic disk 121 is used to store data, and the file system 122 is used to implement reading and writing of data on the shingled magnetic disk 121 by using the storage method of the above embodiment.

In an application scenario, as shown in fig. 13, the file system 122 first performs a disk scanning operation on the shingled magnetic disk 121, if the SMR disk is a new disk, the method of the above embodiment is used to perform an initialization post-processing task on the new disk, and if the SMR disk is an old disk, the method scans the log file and loads the metadata post-processing task; in the process of processing tasks, if data writing tasks are executed, a disk is found through load balancing, data are cached in a data cache region, metadata are cached in a log cache region, when the cache data capacity reaches a threshold value or the data caching duration reaches the threshold value, cache data are transferred to an SMR partition, metadata information is updated, and the metadata are transferred to a full metadata region; and if the data reading task is executed, summarizing and searching the mapping position in the memory according to the data index, locking an SMR partition or a data cache region, finding corresponding data according to the address offset, and finishing the data reading.

The present application further provides an electronic device, as shown in fig. 15, fig. 15 is a schematic structural diagram of an embodiment of the electronic device of the present application. The electronic device 100 of the present embodiment includes a processor 101, a memory 102 coupled to the processor 101, an input/output device 103, and a bus 104.

The processor 101, the memory 102, and the input/output device 103 are respectively connected to the bus 104, the memory 102 stores program data, and the processor 101 is configured to execute the program data to implement the storage method.

In the present embodiment, the processor 101 may also be referred to as a CPU (Central Processing Unit). The processor 101 may be an integrated circuit chip having signal processing capabilities. The processor 101 may also be a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components. A general purpose processor may be a microprocessor or the processor 101 may be any conventional processor or the like.

The present application further provides a computer-readable storage medium, as shown in fig. 16, fig. 16 is a schematic structural diagram of an embodiment of the computer-readable storage medium of the present application. The computer-readable storage medium 131 has stored thereon program data 132, the program data 132 implementing the above-described storage method when executed by a processor (not shown).

The computer-readable storage medium 131 of this embodiment may be, but is not limited to, a usb disk, an SD card, a PD optical drive, a removable hard disk, a high-capacity floppy drive, a flash memory, a multimedia memory card, a server, etc.

The storage method of the shingled magnetic disk initializes the shingled magnetic disk to divide the shingled magnetic disk into a first storage area and a second storage area; when data is read and written, the first storage area is used for recording the data, and the second storage area is used for recording metadata information, log information and cache data of the data. By the method, metadata and real data can be managed in batch writing through the log type storage mode of the second storage area, complexity of management data of the first partition can be reduced, SMR disk data writing performance is improved, reliability of a file system is improved, and integrity of cluster data can be guaranteed.

Furthermore, the log cache region is arranged according to the sequential write characteristic of the SMR disk, and the log cache region has the main functions of converging and merging written IO and merging data written m times into n times of writing (m > n), so that IO scheduling of the SMR partition is reduced, and IO efficiency of the SMR partition is improved.

The present application considers at least two service scenarios:

scenario with small traffic: in order to prevent the backlog time of the cache data in the log area from being too long, a threshold value is set for the cache data, and the cache is performed when the time length of the cache data exceeds the threshold value, so that the safety of the data is ensured. A scene with large traffic: and setting a threshold value (integral multiple of zone) of the cache data amount due to the limited size of the data cache area, and when the cache data amount reaches the threshold value, unloading the cache data into the SMR partition in batches. The method prevents the journal area from being written through, reduces the space fragments of the SMR partition and increases the space utilization rate.

According to the method and the device, the persistent data are recorded in a log storage mode, a snapshot is periodically set for the persistent data, the complete data in the current state are recorded, and when abnormal power failure or flying disk and other scenes occur, the storage nodes quickly construct full metadata information in the memory through snapshot information, so that the recovery capability of the storage nodes is improved, and the data security is guaranteed.

In addition, if the above functions are implemented in the form of software functions and sold or used as a standalone product, the functions may be stored in a storage medium readable by a mobile terminal, that is, the present application also provides a storage device storing program data, which can be executed to implement the method of the above embodiments, the storage device may be, for example, a usb disk, an optical disk, a server, etc. That is, the present application may be embodied as a software product, which includes several instructions for causing an intelligent terminal to perform all or part of the steps of the methods described in the embodiments.

In the description of the present application, reference to the description of the terms "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present application. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.

Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present application, "plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.

Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and the scope of the preferred embodiments of the present application includes other implementations in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present application.

The logic and/or steps represented in the flowcharts or otherwise described herein, such as an ordered listing of executable instructions that can be viewed as implementing logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device (e.g., a personal computer, server, network device, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions). For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.

If the technical scheme of the application relates to personal information, a product applying the technical scheme of the application clearly informs personal information processing rules before processing the personal information, and obtains personal independent consent. If the technical scheme of the application relates to sensitive personal information, a product applying the technical scheme of the application obtains individual consent before processing the sensitive personal information, and simultaneously meets the requirement of 'express consent'. For example, at a personal information collection device such as a camera, a clear and significant identifier is set to inform that the personal information collection range is entered, the personal information is collected, and if the person voluntarily enters the collection range, the person is regarded as agreeing to collect the personal information; or on the device for processing the personal information, under the condition of informing the personal information processing rule by using obvious identification/information, obtaining personal authorization in the modes of pop-up window information or asking the person to upload personal information thereof and the like; the personal information processing rule may include information such as a personal information processor, a personal information processing purpose, a processing method, and a type of personal information to be processed.

The above description is only an example of the present application and is not intended to limit the scope of the present application, and all modifications of equivalent structures and equivalent processes, which are made by the contents of the specification and the drawings, or which are directly or indirectly applied to other related technical fields, are intended to be included within the scope of the present application.

Claims (12)

1. A method of storing shingled disks, comprising:

initializing a shingled disk to divide the shingled disk into a first storage area and a second storage area;

and responding to a data reading and writing task, reading and writing data of the shingled magnetic disk, recording the data by using the first storage area in the reading and writing process, and recording metadata information, log information and cache data of the data by using the second storage area.

2. The storage method according to claim 1, wherein initializing the shingled disk further comprises:

dividing the first storage area into a plurality of storage units;

dividing the second storage area into a metadata description area, a log cache area, a data cache area and a full metadata area;

writing the disk information of the shingled disk, the information of the storage unit, the information of the metadata description area, the information of the log cache area, the information of the data cache area and the information of the full metadata area into the metadata description area;

and loading the information written into the metadata description area into a memory, and reporting the information to a metadata service node.

3. The storage method according to claim 2, wherein the second storage area further comprises: the device comprises a main boot sector and a plurality of reserved areas, wherein the sum of the capacity of the main boot sector and the capacity of the metadata description area is equal to the capacity of the storage unit, the capacities of the journal cache area, the data cache area and the full metadata area are integral multiples of the capacity of the storage unit, and the reserved areas are arranged between the main boot sector and the journal cache area, between the journal cache area and the data cache area and between the data cache area and the full metadata area.

4. The storage method according to claim 2, wherein the data read-write task includes a data write task, and the reading and writing of data to the shingled magnetic disk in response to the data read-write task includes:

and responding to a data writing task, constructing metadata information for the data in a temporary memory, caching the constructed metadata in the log cache region, and caching the data in the data cache region.

5. The storage method according to claim 4, wherein the reading and writing of data to and from the shingled disk in response to a data reading and writing task further comprises:

transferring the data cached in the data cache region to the first storage region based on the data volume and the data caching duration of the data cache region;

and constructing metadata of the data transferred to the first storage area in a global memory, and updating the data in the global memory to the full-amount metadata area in a log storage mode so as to generate a metadata log of the transferred data in the full-amount metadata area.

6. The storage method according to claim 5, wherein the reading and writing of data to and from the shingled disk in response to a data reading and writing task further comprises:

checkpointing the metadata log;

and deleting corresponding metadata from the temporary memory and deleting the data which is transferred out from the log cache region based on the check point.

7. The storage method according to claim 6, further comprising:

when an exception occurs, searching the latest check point of the metadata log;

loading the full amount of metadata information before the latest check point into a global memory;

deleting the data that was transferred after the last checkpoint.

8. The storage method according to claim 5, wherein the unloading the data cached in the data cache region to the first storage region based on the data amount and the data caching duration of the data cache region comprises:

calculating the remainder of the data amount in the data cache region and the capacity of the storage unit;

responding to the fact that the data amount of the data cache region is larger than the capacity of the storage unit and the remainder is larger than or equal to a duration threshold, and then, after the data of the data cache region is filled to be integral multiple of the capacity of the storage unit, transferring the data to the first storage region;

in response to that the data amount of the data cache region is larger than the capacity of the storage unit and the remainder is zero, directly transferring the data of the data cache region to the first storage region;

responding to the fact that the data amount of the data cache region is larger than the capacity of the storage unit and the remainder is smaller than the duration threshold, reserving a first part of data in the data cache region, and transferring another part of data in the data cache region to the first storage region; wherein the data amount of the first part of data is the remainder;

the duration threshold is a preset percentage of the capacity of the storage unit.

9. The storage method according to claim 2, wherein the data read/write task includes a data read task, and the reading/writing of data to the shingled disk in response to the data read/write task includes:

responding to the data reading task, and searching the position of metadata of the data in a memory according to a data index;

responding to the fact that the metadata are only in the temporary memory, and reading data corresponding to the metadata in the data cache region;

responding to the fact that the metadata are only in the global memory, and reading data corresponding to the metadata in the first storage area;

and responding to the situation that the metadata is partially in the temporary memory and partially in the global memory, respectively reading corresponding first data from the data cache region and corresponding second data from the first storage region, and merging the first data and the second data.

10. A shingled disk storage unit comprising:

a shingled disk for storing data;

a file system connected to the shingled disk, for implementing reading and writing of the data on the shingled disk by using the storage method of any one of claims 1 to 9.

11. An electronic device, comprising: a processor and a memory, the memory having stored therein program data, the processor being configured to execute the program data to implement the storage method of any one of claims 1-9.

12. A computer-readable storage medium, characterized in that the computer-readable storage medium stores program data which, when executed by a processor, implements the storage method according to any one of claims 1 to 9.

Images