WO2020024933A1 - Data writing method and server - Google Patents

Data writing method and server Download PDF

Info

Publication number
WO2020024933A1
WO2020024933A1 PCT/CN2019/098338 CN2019098338W WO2020024933A1 WO 2020024933 A1 WO2020024933 A1 WO 2020024933A1 CN 2019098338 W CN2019098338 W CN 2019098338W WO 2020024933 A1 WO2020024933 A1 WO 2020024933A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
file system
area
stored
writing position
Prior art date
Application number
PCT/CN2019/098338
Other languages
French (fr)
Chinese (zh)
Inventor
陈伟
汪渭春
Original Assignee
杭州海康威视系统技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 杭州海康威视系统技术有限公司 filed Critical 杭州海康威视系统技术有限公司
Publication of WO2020024933A1 publication Critical patent/WO2020024933A1/en

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/08Error detection or correction by redundancy in data representation, e.g. by using checking codes
    • G06F11/10Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F12/00Accessing, addressing or allocating within memory systems or architectures
    • G06F12/02Addressing or allocation; Relocation
    • G06F12/08Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/11File system administration, e.g. details of archiving or snapshots
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/17Details of further file system functions
    • G06F16/172Caching, prefetching or hoarding of files
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0602Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0628Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638Organizing or formatting or addressing of data
    • G06F3/0644Management of space entities, e.g. partitions, extents, pools
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0628Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0662Virtualisation aspects
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0668Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Definitions

  • the present application relates to the field of computer technology, and in particular, to a data writing method and a server.
  • a block device is a device that can read a certain length of data from any location on the device, including hard disks, disks, USB flash drives, and flash memory.
  • the operating system uses a part of the system memory as a cache area. When receiving the data to be stored, the operating system can first save the data to be stored in the cache area, and then write some data in the cache area according to the existing elimination algorithm. Block device saved.
  • the software organization responsible for managing and storing file information in the operating system is called a file system, and the data in the file system includes general data and metadata.
  • general data refers to the actual data in ordinary files
  • metadata is system data used to describe the characteristics of a file, such as access rights, file owner, and distribution information of file data blocks (such as Inode).
  • storing metadata and some popular general data in the cache area can speed up the read and write operations of block devices.
  • the block device is generally used to mount the file system, when the block device has been formatted as a file system, there is no file attribute information after the data reaches the block device, and the operating system cannot accurately distinguish the received data.
  • Data is also metadata, which may cause important metadata to be eliminated from the cache.
  • the present application provides a data writing method and a server, which are used to distinguish metadata from general data, and write the distinguished metadata into a cache area to optimize the read and write performance of a block device.
  • a data writing method is provided.
  • the method is applied to a server and includes: receiving first to-be-stored data sent by a client and the first to-be-stored data in a block device.
  • each of the file system's area location information is generated according to the storage space size of the block device, and each of the file system's area location information includes the file system's metadata area and general data Location information of the area; when it is determined that the writing position of the first to-be-stored data is located in the metadata area of the first file system, writing the first to-be-stored data into a cache area of the server.
  • the second aspect of the embodiments of the present application provides a server, which has a function of implementing the method provided in the first aspect.
  • the functions may be implemented by hardware, and may also be implemented by hardware executing corresponding software.
  • the hardware or software includes one or more modules or units corresponding to the functions described above.
  • the server may include: a receiving unit, configured to receive the first to-be-stored data and the writing position of the first to-be-stored data sent by the client in a block device; a comparison unit, Comparing the writing position of the first to-be-stored data with the area location information of the first file system that has been generated in the server; the first file system is a plurality of files that have been generated by the server One of the file systems, the area location information of each of the file systems is generated according to the storage space size of the block device, and the area location information of each of the file systems includes the location of a metadata area and a general data area of the file system Information; a write data unit, configured to write the first data to be stored when the comparison unit determines that a write position of the first data to be stored is located in a metadata area of the first file system A cache area of the server.
  • a receiving unit configured to receive the first to-be-stored data and the writing position of the first to-be-stored data sent by the client in a block
  • the server includes a processor, a communication interface, a memory, and a bus, where the processor, the communication interface, and the memory are connected to each other through a bus; the memory stores machine-readable instructions, and the processor The method provided by the first aspect of the embodiments of the present application is executed by calling the machine-readable instructions.
  • metadata is distinguished from general data, and the distinguished metadata is stored in the buffer area for a long time.
  • the distinguished general data is written into the block device according to the method, which can avoid the mixed writing of metadata and general data, which leads to the magnetic head. Frequent movement between general data storage locations and metadata storage locations of block devices improves data read and write performance.
  • FIG. 1 is a schematic diagram of an architecture provided by an embodiment of the present application.
  • FIG. 3 is a schematic diagram of partitioning a block device in an EXT2 system according to an embodiment of the present application
  • FIG. 4 is a schematic diagram of a logical structure of a block group in an EXT2 system according to an embodiment of the present application
  • FIG. 5a, FIG. 5b, FIG. 5c, and FIG. 5d are schematic diagrams of an anti-check operation provided by an embodiment of the present application.
  • FIG. 6 is a module block diagram of a server provided in an embodiment of the present application.
  • FIG. 7 is a hardware structural diagram of a server provided in an embodiment of the present application.
  • Metadata includes the data structure that manages the entire file system, the data structure that manages the allocation of space, the attributes and pointers of files, the attributes and contents of directories.
  • EXT2 second extended filesystem
  • superblocks, block group descriptors, data block bitmaps, inode bitmaps, inode tables, and directory entry pages all belong to the metadata of the file system.
  • General data refers to the content of the file. Taking the EXT2 file system as an example, the general data of the file system is the file content stored in the data area.
  • Metadata area refers to a contiguous space of cache addresses allocated in the cache. It is used to cache superblocks, block group descriptors, inode bitmaps, data block bitmaps, and inode information created during copy directories and compressed file decompression operations. Cache refers to a part of space divided from system memory.
  • General data area refers to a contiguous space of cache addresses allocated in the cache. It is used to cache data information created during the copy directory and compressed file decompression operation.
  • FIG. 1 is an overall architecture diagram of an embodiment of the present application.
  • the embodiment of the present application includes a client and a server.
  • the server is connected to multiple block devices, and the server is connected between the client and the block device.
  • the connection between the server and the client and between the server and the block device can be It is a network connection or an electrical connection.
  • the server can set up a cache area for each of these block devices, and create a virtual device corresponding to the block device on the client.
  • the process of the client saving data to the virtual device is equivalent to sending a write instruction to the server, and the server writes the data carried by the write instruction to the buffer area or block device.
  • the process of the client extracting data from the virtual device is equivalent to Send a read instruction to the server, and the server extracts the corresponding data from the block device or the buffer area according to the access address of the read instruction and returns it to the client.
  • this application mainly improves the process of writing data, which is manifested in the server's separation and identification of the data to be stored, distinguishing between metadata and general data, and making the distinguished metadata permanent. Save to buffer area.
  • FIG. 2 it is a schematic flow chart of the operation of the data writing method provided by the present application.
  • the server receives the data to be stored from the client for the first time, perform the following steps:
  • Step 201 The server receives the first to-be-stored data and the writing position of the first to-be-stored data sent by the client in the block device.
  • the writing position of the first to-be-stored data in the block device is indicated by the following information: the starting writing position of the first to-be-stored data in the block device and the data size of the first to-be-stored data.
  • the start writing position and the end writing position of the first data to be stored in the block device may be determined.
  • Step 202 The server compares the writing position of the first to-be-stored data with the regional location information of the first file system generated in the server.
  • the first file system is a plurality of file systems generated in the server.
  • the area location information of each file system is generated according to the storage space size of the block device.
  • the area location information of each file system includes the location information of the metadata area and the general data area of the file system.
  • the client Before step 201, the client applies for a block device from the server in advance.
  • the size of the storage space of the block device that can be used by the client can be requested by the client or allocated uniformly by the server.
  • the server accepts the client's application for the use of the block device, it determines the block device for use by the client among the multiple block devices connected to the server, and creates a virtual device on the client. Creating a virtual device on the client is equivalent to mounting a block device on the client. This virtual device can be regarded as a mapping of the block device.
  • the client formatting the virtual device is equivalent to sending formatting instructions to the server.
  • the server formats the block device allocated to the client; the client saves the data to the virtual device, which is equivalent to sending a write instruction to the server, and the server writes the data carried by the write instruction to the cache area or Block device.
  • the client wants to use the block device as a file, it needs to format the block device into a file system.
  • the design idea of this application is to judge the type of data by comparing the location information of the block device formatted with the write position of the data. If the data write location is in the metadata area of the formatted block device, the data is considered as metadata; if the data write location is in the general data area of the formatted block device, the data to be considered is General data.
  • the server is not sure which file system the client uses to format the block device it uses. Therefore, in the embodiment of the present application, the server can set up a file system template library.
  • File system structure such as EXT2 (second extended filesystem), EXT3 (third extended filesystem), XFS, FAT (File Allocation Table), NTFS (New Technology File System) and so on.
  • the server can use the storage space size as an input to the file system template library, and use the file system template library to establish a set that matches the size of the storage space.
  • the regional location information of each file system the size of the input storage space is different, and the regional location information of each file system is also different.
  • the server can establish a set of area location information of the file system corresponding to the storage space size of the block device used by the client.
  • the block device when a block device is formatted as an EXT2 file system, the block device is divided into a boot block and multiple block groups; the boot block refers to the boot sector of the file system. Can be used to install the boot program; each block group has its own metadata area and general data area, and the internal logical results of each block group are shown in Figure 4, including the general data area and the metadata area.
  • the logical structure of the metadata area and the general data area in the block group can be seen from FIG. 4.
  • the structure of the data area is relatively single and consists of many blocks of the same size.
  • the metadata area includes multiple sub-metadata areas.
  • the metadata area of the EXT2 file system consists of a Super Block, a GDT (group descriptor), an inode table, an inode bitmap, and data.
  • a block bitmap is composed of five sub-metadata areas.
  • Step 203 When the server determines that the write position of the first data to be stored is located in the metadata area of the first file system, the server writes the first data to be stored into the cache area of the server.
  • the server compares the writing position of the first to-be-stored data with the area position information of multiple file systems that have been generated in the server in sequence.
  • the first to-be-stored data is found.
  • the start writing position and the ending writing position of the data are located in the same sub-metadata area of the first file system, and it is determined that the writing position of the first to-be-stored data is located in the metadata area of the first file system.
  • the server determines that the first file system is a file system type that the block device used by the client may adopt, and determines that the first data to be stored may be metadata. After the first file system is found, the server can stop the comparison and directly write the first data to be stored into the cache area.
  • the server can determine the first The stored data is general data.
  • the server can process the first to-be-stored data according to a set general data processing strategy. For example, the first to-be-stored data can be written directly to the block device, or the first to-be-stored data can be written to the cache area first. When the available space in the cache area is less than a certain value, the general data in the cache area is eliminated.
  • a specific implementation method is as follows: When it is determined that the write position of the first data to be stored is located in multiple generated file systems In the general data area of each file system, determine the first to-be-stored data as general data, and determine whether the available space in the cache area of this server is larger than the occupied space of the first to-be-stored data; if so, The stored data is written into the cache area of this server; if not, according to the preset elimination algorithm, such as FIFO (First In, First Out) algorithm, LRU (Least, Recently Used) algorithm, LFU (Least Frequently Used, least frequently used) algorithm, etc., determine the obsolete data in the general data stored in the cache area of this server; according to the obsolete data, In position, selecting at least one data having a writing position adjacent from the buffer; writes a block of the present server apparatus together with at least one data out of the data having adjacent writing position.
  • FIFO First In, First Out
  • LRU Least, Recently Used
  • LFU Least Frequently Used, least frequently used
  • the aggregation method can also be used to write the metadata with adjacent write positions to the block device together.
  • the written data after the data in the cache area is written to the block device, the written data also needs to be deleted from the cache area; when the data in the block device is written to the cache area, there is no need to When the delete operation is performed, the corresponding data remains in the block device.
  • the server receives the data to be stored from the client for the first time. After that, when the server receives the data to be stored from the client again, the server can perform steps 201-203 again. Or, in another embodiment, in order to further improve the read and write performance of the block device, when the server performs steps 201-203 once and has determined the type of file system that the block device used by the client may adopt, the server When receiving the data to be stored subsequently sent by the same client, such as the second data to be stored, the following process may be performed:
  • the server receives the second to-be-stored data and the writing position of the second to-be-stored data sent by the client in the block device; the writing position of the second to-be-stored data in the block device includes the second to-be-stored data in the block device Start writing position and ending writing position.
  • the server compares the writing position of the second data to be stored with the area position information of the first file system.
  • the first file system is the file system type that the block device used by the client determined in step 203 may adopt.
  • the write position of the second to-be-stored data is directly compared with the area position information of the first file system. Compare the write position of the second to-be-stored data with the regional position information of the multiple file systems corresponding to the client that has been generated.
  • the server still judges the first file system as the type of file system that the block device used by the client may use, and determines that the second data to be stored may be metadata, and directly writes the second data to be stored in the cache. Area.
  • the server still determines that the first file system is a block used by the client.
  • the file system type that the device may use, and determines that the second data to be stored may be general data.
  • the server can process the second data to be stored according to a set general data processing policy, and the specific processing process is not described in detail.
  • the start writing position and the end writing position of the second data to be stored are in different sub-metadata areas of the first file system, or the start writing position of the second data to be stored is located in the first file
  • the metadata area of the system and the end write location of the second to-be-stored data are outside the metadata area of the first file system, indicating that the first file system is not the file system type used by the block device used by the client.
  • the server needs to re-compare the writing position of the second to-be-stored data with the regional position information of other file systems (except the first file system) in the multiple file systems that have been generated.
  • the start writing position and the end writing position of the second data to be stored are located in a metadata area of the file system, it is determined that the second data to be stored is normal data.
  • the second file system may be determined to be a block used by the client.
  • the file system type that the device may adopt, and determines that the second data to be stored may be metadata.
  • the server can stop the comparison and directly write the second data to be stored into the cache area of the server.
  • the server may also perform a back-check operation to transfer the data previously misjudged as metadata to the block device, and Data that was previously misjudged as general data is rewritten into the cache area, as follows:
  • the server can save the start in the cache area of the server
  • the data at the write position and / or the end write position are in [x1, x2]
  • the data at the start write position and / or the end write position are in [y2, y1] are written to the block device of this server.
  • the server can set Data stored in the start and / or end write locations of the server's buffer area in [y2, y1] is written to the block device of the server, and the start of write is stored in the block device of the server.
  • the data at the input position and the end write position are both in [x2, x1] and written into the cache area of this server.
  • the server can set Data stored in the start and / or end write locations of the server's cache area in [x1, x2] is written to the block device of the server, and the start of write is stored in the block device of the server The data at the entry position and the end writing position are located in [y1, y2] and written into the cache area of this server.
  • the server can save the start of the block device in the server
  • the data where the write position and the end write position are both in [x2, x1], and the data where the start write position and the end write position are both in [y1, y2] are written into the cache area of this server.
  • the block device After the block device is formatted as a file system, general data and metadata are usually stored separately in the block device.
  • the metadata is distinguished from the general data, and the distinguished metadata is stored in the cache area for a long time.
  • the magnetic head by aggregating a plurality of data having adjacent write positions into a large data and writing it to the block device at one time, compared with writing a plurality of data into the block device in batches, the magnetic head is more reduced.
  • FIG. 6 is a functional module diagram of a server provided in an embodiment of the present application. As shown in Figure 6, the server includes:
  • the receiving unit 601 is configured to receive first data to be stored and a writing position of the first data to be stored in a block device sent by a client;
  • a comparison unit 602 is configured to compare the writing position of the first to-be-stored data with the regional position information of the first file system generated in the server;
  • the first file system is the service One of a plurality of file systems that have been generated in the client, each of the file system's area location information is generated according to the storage space size of the block device, and each of the file system's area location information includes metadata of the file system Location information of the data area and general data area;
  • the data writing unit 603 is configured to write the first data to be stored when the comparison unit 602 determines that the writing position of the first data to be stored is located in a metadata area of the first file system.
  • the cache area of this server is configured to write the first data to be stored when the comparison unit 602 determines that the writing position of the first data to be stored is located in a metadata area of the first file system. The cache area of this server.
  • the metadata area of each file system includes multiple sub-metadata areas
  • the writing position of the first data to be stored in the block device includes the first data to be stored in the block device.
  • the starting writing position and the ending writing position; the comparison unit 602 is configured to determine that the starting writing position and the ending writing position of the first to-be-stored data are located in the same sub-unit of the first file system When in the data area, it is determined that the writing position of the first data to be stored is located in the metadata area of the first file system.
  • the receiving unit 601 is further configured to receive the second to-be-stored data and the writing position of the second to-be-stored data in the block device sent by the client;
  • the second The writing position of the data to be stored in the block device includes a start writing position and an end writing position of the second data to be stored in the block device;
  • the comparison unit 602 is further configured to compare the writing position of the second data to be stored with the area position information of the first file system; if the starting writing position of the second data to be stored is And the end writing position are located in different sub-metadata areas of the first file system, or the start writing position of the second to-be-stored data is located in the metadata area of the first file system and the second The end writing position of the data to be stored is outside the metadata area of the first file system, and then the writing position of the second data to be stored and the area position of the second file system in the plurality of file systems Information comparison
  • the writing data unit 603 is further configured to: when the comparing unit 602 determines that the start writing position and the ending writing position of the second to-be-stored data are located in a same sub-metadata area of the second file system , Writing the second to-be-stored data into a cache area of the server.
  • the write data unit 603 is further configured to: if the metadata area range [x1, y1] of the first file system includes the metadata area range [x2, y2], the data stored in the start write position and / or the end write position in [x1, x2] stored in the cache area of this server, and the start write position and / Or the end write data located in [y2, y1] is written to the block device of this server.
  • the write data unit 603 is further configured to: if the range of the metadata area [x1, y1] of the first file system and the range of the metadata area [x2, y2] if there is an intersection and x1> x2, y1> y2, then write the data stored in the cache area of the server and / or the end write location in [y2, y1] to the block of the server
  • the device, and the data stored in the start writing position and the end writing position stored in the block device of the server are both in [x2, x1] and written to the buffer area of the server.
  • the write data unit 603 is further configured to: if the range of the metadata area [x1, y1] of the first file system and the range of the metadata area [x2, y2] if there is an intersection and x1 ⁇ x2, y1 ⁇ y2, the data stored in the start and / or end write locations in the cache area of this server is written to the block of this server
  • the device, as well as the data stored in the block device of the server at the beginning and end of writing, are located in the buffer area of the server.
  • the write data unit 603 is further configured to: if the metadata area range [x2, y2] of the second file system includes the metadata area range [x1, y1], the data stored in the start block and end write positions of the server block device are both located in [x2, x1], and the start write position and end are stored in the block device of this server end The data whose write positions are located in [y1, y2] are written into the cache area of this server.
  • the write data unit 603 is further configured to, when it is determined that the write position of the first data to be stored is located in a general data area of each file system of the multiple file systems, Determine that the first data to be stored is general data, and determine whether the available space in the cache area of the server is larger than the occupied space of the first data to be stored; if so, write the first data to be stored in the service The cache area on the server side; if not, the elimination data is determined from the general data stored in the cache area of the server side according to a preset elimination algorithm; according to the write position of the elimination data, an adjacent area is selected from the cache area on the server side Write at least one data at a location; write the obsolete data together with the at least one data with an adjacent write location into a block device on the server side.
  • the division of the units in the embodiments of the present application is schematic, and is only a logical function division. There may be another division manner in actual implementation.
  • the functional units in the embodiments of the present application may be integrated into one processing unit, or each of the units may exist separately physically, or two or more units may be integrated into one unit.
  • the above integrated unit may be implemented in the form of hardware or in the form of software functional unit.
  • FIG. 7 is a schematic diagram of a hardware structure of a server provided in an embodiment of the present application.
  • the server includes a processor 701, a communication interface 702, a memory 703, and a bus 704. Among them, the processor 701, the communication interface 702, and the memory 703 Connected to each other via a bus.
  • the processor 701 executes the method shown in FIG. 2 by calling the machine-readable instructions.
  • An embodiment of the present application further provides a machine-readable storage medium that stores machine-readable instructions.
  • the machine-executable instructions When the machine-readable instructions are called and executed by a processor, the machine-executable instructions cause The processor implements the method shown in FIG. 2.

Abstract

The present application provides a data writing method and a server. In the present application, metadata is distinguished from general data, the distinguished metadata is permanently stored in a cache area, and the distinguished general data is written into a block device, so that a case where mixed flashing of the metadata and the general data causes frequent movement of a magnetic head between a general data storage position and a metadata storage position of the block device can be avoided, thereby improving the read-write performance of data.

Description

一种数据写入方法和服务端Data writing method and server 技术领域Technical field
本申请涉及计算机技术领域,尤其涉及一种数据写入方法和服务端。The present application relates to the field of computer technology, and in particular, to a data writing method and a server.
背景技术Background technique
块设备是指可以从设备的任意位置读取一定长度数据的设备,包括硬盘、磁盘、U盘和闪存等。操作系统利用一部分系统内存作为缓存区,在收到待存储的数据时,操作系统可以先将待存储的数据保存至缓存区,之后再根据已有的淘汰算法将缓存区中的部分数据写入块设备保存。A block device is a device that can read a certain length of data from any location on the device, including hard disks, disks, USB flash drives, and flash memory. The operating system uses a part of the system memory as a cache area. When receiving the data to be stored, the operating system can first save the data to be stored in the cache area, and then write some data in the cache area according to the existing elimination algorithm. Block device saved.
操作系统中负责管理和存储文件信息的软件机构称为文件系统,文件系统中的数据包括一般数据和元数据。其中,一般数据是指普通文件中的实际数据,而元数据是用来描述一个文件的特征的系统数据,如访问权限、文件拥有者以及文件数据块的分布信息(如Inode)等。理想情况下,将元数据和部分热门的一般数据保存在缓存区能加速块设备的读写操作。The software organization responsible for managing and storing file information in the operating system is called a file system, and the data in the file system includes general data and metadata. Among them, general data refers to the actual data in ordinary files, and metadata is system data used to describe the characteristics of a file, such as access rights, file owner, and distribution information of file data blocks (such as Inode). Ideally, storing metadata and some popular general data in the cache area can speed up the read and write operations of block devices.
但由于块设备一般是以安装文件系统的方式使用的,当块设备已经格式化为文件系统时,数据到达块设备后是没有任何文件属性信息的,操作系统无法准确区分收到的数据是一般数据还是元数据,这样可能导致将重要的元数据从缓存区淘汰。However, since the block device is generally used to mount the file system, when the block device has been formatted as a file system, there is no file attribute information after the data reaches the block device, and the operating system cannot accurately distinguish the received data. Data is also metadata, which may cause important metadata to be eliminated from the cache.
发明内容Summary of the invention
有鉴于此,本申请提供一种数据写入方法和服务端,用以将元数据与一般数据进行区分,并将区分出的元数据写入缓存区,以优化块设备的读写性能。In view of this, the present application provides a data writing method and a server, which are used to distinguish metadata from general data, and write the distinguished metadata into a cache area to optimize the read and write performance of a block device.
本申请实施例第一方面,提供了一种数据写入方法,所述方法应用于服务端,包括:接收客户端发送的第一待存储数据和所述第一待存储数据在块设备中的写入位置;将所述第一待存储数据的写入位置与所述服务端中已生成的第一文件系统的区域位置信息进行比对;所述第一文件系统为所述服务端已生成的多个文件系统之一,每个所述文件系统的区域位置信息根据所述块设备的存储空间大小生成,每个所述文件系统的区域位置信息包括该文件系统的元数据区和一般数据区的位置信息;在确定所述第一待存储数据的写入位置位于所述第一文件系统的元数据区内时,将所述第一待存储数据写入所述 服务端的缓存区。According to a first aspect of the embodiments of the present application, a data writing method is provided. The method is applied to a server and includes: receiving first to-be-stored data sent by a client and the first to-be-stored data in a block device. Write location; comparing the write location of the first to-be-stored data with the area location information of the first file system that has been generated in the server; the first file system has been generated for the server One of a plurality of file systems, each of the file system's area location information is generated according to the storage space size of the block device, and each of the file system's area location information includes the file system's metadata area and general data Location information of the area; when it is determined that the writing position of the first to-be-stored data is located in the metadata area of the first file system, writing the first to-be-stored data into a cache area of the server.
本申请实施例第二方面,提供了一种服务端,具有实现上述第一方面提供的方法的功能。所述功能可以通过硬件实现,也可以通过硬件执行相应的软件实现。所述硬件或软件包括一个或多个与上述功能相对应的模块或单元。The second aspect of the embodiments of the present application provides a server, which has a function of implementing the method provided in the first aspect. The functions may be implemented by hardware, and may also be implemented by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the functions described above.
一种实现方式中,所述服务端可以包括:接收单元,用于接收客户端发送的第一待存储数据和所述第一待存储数据在块设备中的写入位置;比对单元,用于将所述第一待存储数据的写入位置与所述服务端中已生成的第一文件系统的区域位置信息进行比对;所述第一文件系统为所述服务端已生成的多个文件系统之一,每个所述文件系统的区域位置信息根据所述块设备的存储空间大小生成,每个所述文件系统的区域位置信息包括该文件系统的元数据区和一般数据区的位置信息;写数据单元,用于在所述比对单元确定所述第一待存储数据的写入位置位于所述第一文件系统的元数据区内时,将所述第一待存储数据写入所述服务端的缓存区。In an implementation manner, the server may include: a receiving unit, configured to receive the first to-be-stored data and the writing position of the first to-be-stored data sent by the client in a block device; a comparison unit, Comparing the writing position of the first to-be-stored data with the area location information of the first file system that has been generated in the server; the first file system is a plurality of files that have been generated by the server One of the file systems, the area location information of each of the file systems is generated according to the storage space size of the block device, and the area location information of each of the file systems includes the location of a metadata area and a general data area of the file system Information; a write data unit, configured to write the first data to be stored when the comparison unit determines that a write position of the first data to be stored is located in a metadata area of the first file system A cache area of the server.
另一种实现方式中,所述服务端包括处理器、通信接口、存储器和总线,其中,处理器、通信接口、存储器通过总线相互连接;所述存储器中存储机器可读指令,所述处理器通过调用所述机器可读指令执行本申请实施例第一方面提供的方法。In another implementation manner, the server includes a processor, a communication interface, a memory, and a bus, where the processor, the communication interface, and the memory are connected to each other through a bus; the memory stores machine-readable instructions, and the processor The method provided by the first aspect of the embodiments of the present application is executed by calling the machine-readable instructions.
本申请实施例将元数据与一般数据进行区分,将区分出的元数据长久保存至缓存区,将区分出的一般数据按写入块设备,可以避免元数据和一般数据混合刷写,导致磁头在块设备的一般数据存储位置和元数据存储位置之间的频繁移动,提高数据的读写性能。In the embodiment of the present application, metadata is distinguished from general data, and the distinguished metadata is stored in the buffer area for a long time. The distinguished general data is written into the block device according to the method, which can avoid the mixed writing of metadata and general data, which leads to the magnetic head. Frequent movement between general data storage locations and metadata storage locations of block devices improves data read and write performance.
附图说明BRIEF DESCRIPTION OF THE DRAWINGS
图1是本申请实施例提供的架构示意图;FIG. 1 is a schematic diagram of an architecture provided by an embodiment of the present application;
图2是本申请实施例提供的方法流程图;2 is a flowchart of a method provided by an embodiment of the present application;
图3是本申请实施例提供的EXT2系统中块设备的分区示意图;3 is a schematic diagram of partitioning a block device in an EXT2 system according to an embodiment of the present application;
图4是本申请实施例提供的EXT2系统中块组的逻辑结构示意图;4 is a schematic diagram of a logical structure of a block group in an EXT2 system according to an embodiment of the present application;
图5a、图5b、图5c和图5d是本申请实施例提供的反查操作的示意图;FIG. 5a, FIG. 5b, FIG. 5c, and FIG. 5d are schematic diagrams of an anti-check operation provided by an embodiment of the present application;
图6是本申请实施例提供的服务端的模块框图;FIG. 6 is a module block diagram of a server provided in an embodiment of the present application; FIG.
图7是本申请实施例提供的服务端的硬件结构图。FIG. 7 is a hardware structural diagram of a server provided in an embodiment of the present application.
具体实施方式detailed description
以下,首先对本申请实施例中涉及的部分用语进行解释说明。In the following, some terms used in the examples of the present application will be explained first.
元数据:包括管理整个文件系统的数据结构、管理空间分配的数据结构、文件的属性和指针、目录的属性及内容。以EXT2(second extended filesystem)文件系统为例,超级块、块组描述符、数据块位图、Inode位图、Inode表、目录项页面都属于文件系统的元数据。Metadata: includes the data structure that manages the entire file system, the data structure that manages the allocation of space, the attributes and pointers of files, the attributes and contents of directories. Taking the EXT2 (second extended filesystem) file system as an example, superblocks, block group descriptors, data block bitmaps, inode bitmaps, inode tables, and directory entry pages all belong to the metadata of the file system.
一般数据:指的是文件的内容。以EXT2文件系统为例,文件系统的一般数据是存放在数据区的文件内容。General data: refers to the content of the file. Taking the EXT2 file system as an example, the general data of the file system is the file content stored in the data area.
元数据区(matadata area):指的是在缓存中分配的一段缓存地址连续的空间。其用于缓存超级块、块组描述符、Inode位图、数据块位图以及拷贝目录和压缩文件解压操作过程中创建的Inode信息。缓存是指从系统内存中划分出的一部分空间。Metadata area (matadata area): Refers to a contiguous space of cache addresses allocated in the cache. It is used to cache superblocks, block group descriptors, inode bitmaps, data block bitmaps, and inode information created during copy directories and compressed file decompression operations. Cache refers to a part of space divided from system memory.
一般数据区(data area):指的是在缓存中分配的一段缓存地址连续的空间。其用于缓存拷贝目录和压缩文件解压操作过程中创建的数据信息。General data area (data area): Refers to a contiguous space of cache addresses allocated in the cache. It is used to cache data information created during the copy directory and compressed file decompression operation.
下面结合说明书附图和各实施例对本申请技术方案进行说明。The technical solution of the present application will be described below with reference to the accompanying drawings and embodiments of the specification.
请参考图1所示,其为本申请实施例的总体架构图。本申请实施例中包括客户端和服务端,服务端连接有多个块设备,并且服务端连接于客户端和块设备之间;服务端与客户端以及服务端与块设备之间的连接可以是网络连接或者电连接。服务端可以为这多个块设备分别设置一个缓存区,并在客户端创建与块设备对应的虚拟设备。客户端把数据保存到虚拟设备的过程,相当于向服务端发送写指令,由服务端将该写指令携带的数据写入缓存区或块设备;客户端从虚拟设备提取数据的过程,相当于向服务端发送读指令,由服务端根据读指令的访问地址向块设备或缓存区提取相应的数据并返回给客户端。为改善块设备的读写性能,本申请主要对写数据的过程作了改进,表现于服务端对待存储的数据进行分离和识别,区分出元数据和一般数据,并将区分出的元数据长久保存至缓存区。Please refer to FIG. 1, which is an overall architecture diagram of an embodiment of the present application. The embodiment of the present application includes a client and a server. The server is connected to multiple block devices, and the server is connected between the client and the block device. The connection between the server and the client and between the server and the block device can be It is a network connection or an electrical connection. The server can set up a cache area for each of these block devices, and create a virtual device corresponding to the block device on the client. The process of the client saving data to the virtual device is equivalent to sending a write instruction to the server, and the server writes the data carried by the write instruction to the buffer area or block device. The process of the client extracting data from the virtual device is equivalent to Send a read instruction to the server, and the server extracts the corresponding data from the block device or the buffer area according to the access address of the read instruction and returns it to the client. In order to improve the read and write performance of block devices, this application mainly improves the process of writing data, which is manifested in the server's separation and identification of the data to be stored, distinguishing between metadata and general data, and making the distinguished metadata permanent. Save to buffer area.
参见图2所示,其为本申请提供的数据写入方法的运作流程示意图。针对每个客户端,当服务端第一次从该客户端收到待存储的数据时,执行以下步骤:Referring to FIG. 2, it is a schematic flow chart of the operation of the data writing method provided by the present application. For each client, when the server receives the data to be stored from the client for the first time, perform the following steps:
步骤201:服务端接收客户端发送的第一待存储数据和第一待存储数据在块设备中的写入位置。Step 201: The server receives the first to-be-stored data and the writing position of the first to-be-stored data sent by the client in the block device.
本申请实施例中,第一待存储数据在块设备中的写入位置通过以下信息表示:第一待存储数据在块设备中的开始写入位置和第一待存储数据的数据大小。结合第一待存储数据的开始写入位置和数据大小,可以确定第一待存储数据在块设备中的开始写入位置和结束写入位置。In the embodiment of the present application, the writing position of the first to-be-stored data in the block device is indicated by the following information: the starting writing position of the first to-be-stored data in the block device and the data size of the first to-be-stored data. By combining the start writing position and the data size of the first data to be stored, the start writing position and the end writing position of the first data to be stored in the block device may be determined.
步骤202:服务端将第一待存储数据的写入位置与该服务端中已生成的第一文件系统的区域位置信息进行比对;第一文件系统为服务端中已生成的多个文件系统之一,每个文件系统的区域位置信息根据块设备的存储空间大小生成,每个文件系统的区域位置信息包括该文件系统的元数据区和一般数据区的位置信息。Step 202: The server compares the writing position of the first to-be-stored data with the regional location information of the first file system generated in the server. The first file system is a plurality of file systems generated in the server. First, the area location information of each file system is generated according to the storage space size of the block device. The area location information of each file system includes the location information of the metadata area and the general data area of the file system.
在步骤201之前,客户端预先向服务端申请使用块设备,该客户端能够使用的块设备存储空间大小可以由客户端主动申请,或者也可以由服务端统一分配。服务端接受客户端的块设备使用申请后,在自身连接的多个块设备中确定出供该客户端使用的块设备,以及在客户端上创建虚拟设备。在客户端上创建虚拟设备,相当于将块设备挂载到了客户端,该虚拟设备可以视为块设备的映射,例如客户端对虚拟设备进行格式化,相当于向服务端发送格式化指令,由服务端对分配给该客户端使用的块设备进行格式化;客户端将数据保存到虚拟设备,相当于向服务端发送写指令,由服务端将该写指令携带的数据写入缓存区或块设备。Before step 201, the client applies for a block device from the server in advance. The size of the storage space of the block device that can be used by the client can be requested by the client or allocated uniformly by the server. After the server accepts the client's application for the use of the block device, it determines the block device for use by the client among the multiple block devices connected to the server, and creates a virtual device on the client. Creating a virtual device on the client is equivalent to mounting a block device on the client. This virtual device can be regarded as a mapping of the block device. For example, the client formatting the virtual device is equivalent to sending formatting instructions to the server. The server formats the block device allocated to the client; the client saves the data to the virtual device, which is equivalent to sending a write instruction to the server, and the server writes the data carried by the write instruction to the cache area or Block device.
客户端如果想将块设备当成文件使用,则需要将块设备格式化成文件系统。本申请的设计思路是,通过比对块设备格式化后的区域位置信息和数据的写入位置,来判断数据的类型。如果数据的写入位置位于格式化后的块设备的元数据区,则认为该数据为元数据;如果数据的写入位置位于格式化后的块设备的一般数据区,则认为该待数据为一般数据。If the client wants to use the block device as a file, it needs to format the block device into a file system. The design idea of this application is to judge the type of data by comparing the location information of the block device formatted with the write position of the data. If the data write location is in the metadata area of the formatted block device, the data is considered as metadata; if the data write location is in the general data area of the formatted block device, the data to be considered is General data.
但在一般情况下,服务端并不确定客户端将其使用的块设备格式化成了哪个文件系统,因此本申请实施例中,服务端可以建立一个文件系统模板库,模板库中包括市面上通用的文件系统结构,如EXT2(second extended filesystem)、EXT3(third extended filesystem)、XFS、FAT(File Allocation Table)、NTFS(New Technology File System)等。当客户端确定了其所要使用的块设备的存储空间大小后,服务端可以将该存储空间大小作为入参传入文件系统模板库,利用文件系统模板库建立一套与该存储空间大小相匹配的各个文件系统的区域位置信息;输入的存储空间大小不同,生成的各个文件系统的区域位置信息也不同。针对每个客户端,服务端可以建立一套与该客户端使用的块设备的存储空间大小对应的文件系统的区域位置信息。However, in general, the server is not sure which file system the client uses to format the block device it uses. Therefore, in the embodiment of the present application, the server can set up a file system template library. File system structure, such as EXT2 (second extended filesystem), EXT3 (third extended filesystem), XFS, FAT (File Allocation Table), NTFS (New Technology File System) and so on. After the client determines the storage space size of the block device to be used, the server can use the storage space size as an input to the file system template library, and use the file system template library to establish a set that matches the size of the storage space. The regional location information of each file system; the size of the input storage space is different, and the regional location information of each file system is also different. For each client, the server can establish a set of area location information of the file system corresponding to the storage space size of the block device used by the client.
以下以EXT2文件系统为例,说明文件系统的区域位置信息。参见图3,在将块设备格式化为EXT2文件系统时会将块设备划分为一个引导块(boot block)和多个块组(block group);其中的引导块是指文件系统的启动扇区,可以用于安装引导程序;其中的每个块组均有各自的元数据区和一般数据区,每个块组的内部逻辑结果参见图4所示,包括一般数据区和元数据区。The following uses the EXT2 file system as an example to describe the area location information of the file system. Referring to FIG. 3, when a block device is formatted as an EXT2 file system, the block device is divided into a boot block and multiple block groups; the boot block refers to the boot sector of the file system. Can be used to install the boot program; each block group has its own metadata area and general data area, and the internal logical results of each block group are shown in Figure 4, including the general data area and the metadata area.
从图4可以看到块组中元数据区和一般数据区的逻辑结构。一般数据区结构比较单一,由许多大小相同的数据块(block)组成。元数据区则包括多个子元数据区,例如EXT2文件系统的元数据区由超级块(Super Block)、GDT(group descriptor table)、inode表(inode table)、inode位图(inode bitmap)和数据块位图(block bitmap)五个子元数据区组成。The logical structure of the metadata area and the general data area in the block group can be seen from FIG. 4. Generally, the structure of the data area is relatively single and consists of many blocks of the same size. The metadata area includes multiple sub-metadata areas.For example, the metadata area of the EXT2 file system consists of a Super Block, a GDT (group descriptor), an inode table, an inode bitmap, and data. A block bitmap is composed of five sub-metadata areas.
步骤203:服务端在确定第一待存储数据的写入位置位于第一文件系统的元数据区内时,将第一待存储数据写入本服务端的缓存区。Step 203: When the server determines that the write position of the first data to be stored is located in the metadata area of the first file system, the server writes the first data to be stored into the cache area of the server.
本实施例中,服务端将第一待存储数据的写入位置与服务端中已生成的多个文件系统的区域位置信息进行依次比对,当比较到第一文件系统,发现第一待存储数据的开始写入位置和结束写入位置位于第一文件系统的同一个子元数据区内,则确定第一待存储数据的写入位置位于第一文件系统的元数据区内。此时,服务端将第一文件系统判定是该客户端使用的块设备可能采用的文件系统类型,并判定第一待存储数据可能是元数据。找到第一文件系统后,服务端可以停止比对,直接将第一待存储数据写入缓存区。In this embodiment, the server compares the writing position of the first to-be-stored data with the area position information of multiple file systems that have been generated in the server in sequence. When compared to the first file system, the first to-be-stored data is found. The start writing position and the ending writing position of the data are located in the same sub-metadata area of the first file system, and it is determined that the writing position of the first to-be-stored data is located in the metadata area of the first file system. At this time, the server determines that the first file system is a file system type that the block device used by the client may adopt, and determines that the first data to be stored may be metadata. After the first file system is found, the server can stop the comparison and directly write the first data to be stored into the cache area.
在另一个例子中,如果直至比较到最后一个文件系统,未发现任何一个文件系统满足:第一待存储数据的写入位置位于该文件系统的元数据区内,则服务端可以确定第一待存储数据是一般数据。接下来,服务端可以按照设定的一般数据处理策略处理该第一待存储数据,例如可以直接将该第一待存储数据写入块设备,或者可以先将第一待存储数据写入缓存区,当缓存区的可用空间小于一定值后,再对缓存区中的一般数据进行淘汰,一种具体的实现方式如下:在确定第一待存储数据的写入位置位于已生成的多个文件系统中每个文件系统的一般数据区内时,确定第一待存储数据为一般数据,判断本服务端缓存区的可用空间是否大于第一待存储数据的占用空间;如果是,则将第一待存储数据写入本服务端的缓存区;如果否,则根据预设的淘汰算法,如FIFO(First In First Out,先进先出)算法、LRU(Least Recently Used,最近最少使用)算法、LFU(Least Frequently Used,最不经常使用)算法等,在本服务端缓存区保存的一般数据中确定淘汰数据;根据该淘汰数据的写入位置,从缓存区中选择具有相邻写入位置的至少一个数据;将淘汰 数据与该具有相邻写入位置的至少一个数据一起写入本服务端的块设备。In another example, if no file system is found until the last file system is compared: the write position of the first to-be-stored data is located in the metadata area of the file system, the server can determine the first The stored data is general data. Next, the server can process the first to-be-stored data according to a set general data processing strategy. For example, the first to-be-stored data can be written directly to the block device, or the first to-be-stored data can be written to the cache area first. When the available space in the cache area is less than a certain value, the general data in the cache area is eliminated. A specific implementation method is as follows: When it is determined that the write position of the first data to be stored is located in multiple generated file systems In the general data area of each file system, determine the first to-be-stored data as general data, and determine whether the available space in the cache area of this server is larger than the occupied space of the first to-be-stored data; if so, The stored data is written into the cache area of this server; if not, according to the preset elimination algorithm, such as FIFO (First In, First Out) algorithm, LRU (Least, Recently Used) algorithm, LFU (Least Frequently Used, least frequently used) algorithm, etc., determine the obsolete data in the general data stored in the cache area of this server; according to the obsolete data, In position, selecting at least one data having a writing position adjacent from the buffer; writes a block of the present server apparatus together with at least one data out of the data having adjacent writing position.
这里,通过将具有相邻写入位置的多个数据聚合成一个大的数据一次性写入块设备,相比将多个数据分批次写入块设备,可以减少磁头在磁碟上的往复摆动,以及可以减少将数据写入块设备消耗的时间。同样的,当需要将缓存区中的被认为是元数据的数据写入块设备时,也可以采用聚合的方式,将具有相邻写入位置的元数据一起写入块设备。Here, by aggregating a plurality of data with adjacent writing positions into a large data written to the block device at one time, compared with writing a plurality of data into the block device in batches, it is possible to reduce the back and forth of the magnetic head on the magnetic disk. Swing, as well as reducing the time it takes to write data to a block device. Similarly, when it is necessary to write the data in the buffer area that is considered as metadata into the block device, the aggregation method can also be used to write the metadata with adjacent write positions to the block device together.
需要注意的是,本申请实施例中将缓存区中的数据写入块设备后,还需要将该写入的数据从缓存区删除;当将块设备中的数据写入缓存区后,则无需执行删除操作,块设备中还保留相应的数据。It should be noted that, in the embodiment of the present application, after the data in the cache area is written to the block device, the written data also needs to be deleted from the cache area; when the data in the block device is written to the cache area, there is no need to When the delete operation is performed, the corresponding data remains in the block device.
以上描述了针对每个客户端,当服务端第一次从该客户端收到待存储的数据时所执行的数据写入流程。之后,当服务端再次从该客户端收到待存储的数据时,服务端可以再次执行步骤201-203。或者,在另一个实施例中,为进一步改善块设备的读写性能,在服务端执行过一次步骤201-203已经确定了客户端使用的块设备可能采用的文件系统类型的情况下,服务端在收到同一客户端后续发送的待存储数据,例如第二待存储数据时,可以执行以下过程:The above describes the data writing process performed for each client when the server receives the data to be stored from the client for the first time. After that, when the server receives the data to be stored from the client again, the server can perform steps 201-203 again. Or, in another embodiment, in order to further improve the read and write performance of the block device, when the server performs steps 201-203 once and has determined the type of file system that the block device used by the client may adopt, the server When receiving the data to be stored subsequently sent by the same client, such as the second data to be stored, the following process may be performed:
服务端接收客户端发送的第二待存储数据和第二待存储数据在块设备中的写入位置;第二待存储数据在块设备中的写入位置包括第二待存储数据在块设备中的开始写入位置和结束写入位置。The server receives the second to-be-stored data and the writing position of the second to-be-stored data sent by the client in the block device; the writing position of the second to-be-stored data in the block device includes the second to-be-stored data in the block device Start writing position and ending writing position.
服务端将第二待存储数据的写入位置与第一文件系统的区域位置信息进行比对。第一文件系统即步骤203中确定出的客户端使用的块设备可能采用的文件系统类型,这里直接将第二待存储数据的写入位置与第一文件系统的区域位置信息进行比对,无需将第二待存储数据的写入位置与已生成的与该客户端对应的多个文件系统的区域位置信息进行一一比对。The server compares the writing position of the second data to be stored with the area position information of the first file system. The first file system is the file system type that the block device used by the client determined in step 203 may adopt. Here, the write position of the second to-be-stored data is directly compared with the area position information of the first file system. Compare the write position of the second to-be-stored data with the regional position information of the multiple file systems corresponding to the client that has been generated.
一种情况下,如果第二待存储数据的开始写入位置和结束写入位置位于第一文件系统的同一个子元数据区内,则确定第二待存储数据的写入位置位于第一文件系统的元数据区内。此时,服务端仍将第一文件系统判定是该客户端使用的块设备可能采用的文件系统类型,并判定第二待存储数据可能是元数据,从而直接将第二待存储数据写入缓存区。In one case, if the start writing position and the ending writing position of the second data to be stored are located in the same sub-metadata area of the first file system, it is determined that the writing position of the second data to be stored is located in the first file system. In the metadata area. At this time, the server still judges the first file system as the type of file system that the block device used by the client may use, and determines that the second data to be stored may be metadata, and directly writes the second data to be stored in the cache. Area.
另一种情况下,如果第二待存储数据的开始写入位置和结束写入位置位于第一文件系统的一般数据区内,则服务端仍将第一文件系统判定是该客户端使用的块设备可能采 用的文件系统类型,并判定第二待存储数据可能是一般数据,接下来服务端可以按照设定的一般数据处理策略处理第二待存储数据,具体处理过程不作赘述。In another case, if the start writing position and the end writing position of the second data to be stored are in the general data area of the first file system, the server still determines that the first file system is a block used by the client. The file system type that the device may use, and determines that the second data to be stored may be general data. Next, the server can process the second data to be stored according to a set general data processing policy, and the specific processing process is not described in detail.
再一种情况下,如果第二待存储数据的开始写入位置和结束写入位置位于第一文件系统的不同子元数据区内,或者第二待存储数据的开始写入位置位于第一文件系统的元数据区内且第二待存储数据的结束写入位置位于第一文件系统的元数据区外,则表明第一文件系统不是该客户端使用的块设备所采用的文件系统类型,从而服务端需要重新将第二待存储数据的写入位置与已生成的多个文件系统中的(除第一文件系统外的)其它文件系统的区域位置信息进行比对。In another case, if the start writing position and the end writing position of the second data to be stored are in different sub-metadata areas of the first file system, or the start writing position of the second data to be stored is located in the first file The metadata area of the system and the end write location of the second to-be-stored data are outside the metadata area of the first file system, indicating that the first file system is not the file system type used by the block device used by the client. The server needs to re-compare the writing position of the second to-be-stored data with the regional position information of other file systems (except the first file system) in the multiple file systems that have been generated.
作为一种实施方式,如果未发现任何一个文件系统满足:第二待存储数据的开始写入位置和结束写入位置位于该文件系统的元数据区内,则确定该第二待存储数据是一般数据。As an implementation manner, if no file system is found to satisfy: the start writing position and the end writing position of the second data to be stored are located in a metadata area of the file system, it is determined that the second data to be stored is normal data.
反之,如果确定第二待存储数据的开始写入位置和结束写入位置位于第二文件系统的同一个子元数据区内,则此时,可以将第二文件系统判定是该客户端使用的块设备可能采用的文件系统类型,并判定第二待存储数据可能是元数据。找到第二文件系统后,服务端可以停止比对,直接将第二待存储数据写入本服务端的缓存区。Conversely, if it is determined that the start writing position and the ending writing position of the second to-be-stored data are located in the same sub-metadata area of the second file system, at this time, the second file system may be determined to be a block used by the client. The file system type that the device may adopt, and determines that the second data to be stored may be metadata. After the second file system is found, the server can stop the comparison and directly write the second data to be stored into the cache area of the server.
进一步的,在判定第二文件系统是客户端使用的块设备可能采用的文件系统类型之后,服务端还可以执行反查操作,将之前被误判为元数据的数据转移至块设备,以及将之前被误判为一般数据的数据重新写入缓存区,具体如下:Further, after determining that the second file system is a file system type that the client may use as the block device, the server may also perform a back-check operation to transfer the data previously misjudged as metadata to the block device, and Data that was previously misjudged as general data is rewritten into the cache area, as follows:
参见图5a,如果第一文件系统的元数据区范围[x1,y1]包含第二文件系统的元数据区范围[x2,y2],则服务端可以将保存在本服务端缓存区中的开始写入位置和/或结束写入位置位于[x1,x2]内的数据、以及开始写入位置和/或结束写入位置位于[y2,y1]内的数据写入本服务端的块设备。Referring to FIG. 5a, if the range [x1, y1] of the metadata area of the first file system includes the range [x2, y2] of the metadata area of the second file system, the server can save the start in the cache area of the server The data at the write position and / or the end write position are in [x1, x2], and the data at the start write position and / or the end write position are in [y2, y1] are written to the block device of this server.
参见图5b,如果第一文件系统的元数据区范围[x1,y1]与第二文件系统的元数据区范围[x2,y2]存在交集且x1>x2,y1>y2,则服务端可以将保存在本服务端缓存区中的开始写入位置和/或结束写入位置位于[y2,y1]内的数据写入本服务端的块设备,以及将保存在本服务端块设备中的开始写入位置和结束写入位置均位于[x2,x1]内的数据写入本服务端的缓存区。Referring to FIG. 5b, if the metadata area range [x1, y1] of the first file system and the metadata area range [x2, y2] of the second file system intersect and x1> x2, y1> y2, the server can set Data stored in the start and / or end write locations of the server's buffer area in [y2, y1] is written to the block device of the server, and the start of write is stored in the block device of the server The data at the input position and the end write position are both in [x2, x1] and written into the cache area of this server.
参见图5c,如果第一文件系统的元数据区范围[x1,y1]与第二文件系统的元数据区范围[x2,y2]存在交集且x1<x2,y1<y2,则服务端可以将保存在本服务端缓存区中的开 始写入位置和/或结束写入位置位于[x1,x2]内的数据写入本服务端的块设备,以及将保存在本服务端块设备中的开始写入位置和结束写入位置均位于[y1,y2]内的数据写入本服务端的缓存区。Referring to FIG. 5c, if the metadata area range [x1, y1] of the first file system and the metadata area range [x2, y2] of the second file system intersect and x1 <x2, y1 <y2, the server can set Data stored in the start and / or end write locations of the server's cache area in [x1, x2] is written to the block device of the server, and the start of write is stored in the block device of the server The data at the entry position and the end writing position are located in [y1, y2] and written into the cache area of this server.
参见图5d,如果第二文件系统的元数据区范围[x2,y2]包含第一文件系统的元数据区范围[x1,y1],则服务端可以将保存在本服务端块设备中的开始写入位置和结束写入位置均位于[x2,x1]内的数据、以及开始写入位置和结束写入位置均位于[y1,y2]内的数据写入本服务端的缓存区。Referring to FIG. 5d, if the metadata area range [x2, y2] of the second file system includes the metadata area range [x1, y1] of the first file system, the server can save the start of the block device in the server The data where the write position and the end write position are both in [x2, x1], and the data where the start write position and the end write position are both in [y1, y2] are written into the cache area of this server.
至此,完成对本申请提供的方法的描述。So far, the description of the method provided in this application is completed.
由于块设备被格式化为文件系统后,一般数据和元数据在块设备中通常是分别存储的,本申请实施例将元数据与一般数据进行区分,将区分出的元数据长久保存至缓存区,将区分出的一般数据按写入块设备,可以避免元数据和一般数据混合刷写,导致磁头在块设备的一般数据存储位置和元数据存储位置之间的频繁移动,提高数据的读写性能。After the block device is formatted as a file system, general data and metadata are usually stored separately in the block device. In the embodiment of the present application, the metadata is distinguished from the general data, and the distinguished metadata is stored in the cache area for a long time. By writing the distinguished general data to the block device, it is possible to avoid the mixed writing of metadata and general data, which causes frequent movement of the magnetic head between the general data storage location and the metadata storage location of the block device, and improves data read and write. performance.
进一步地,本申请实施例通过将具有相邻写入位置的多个数据聚合成一个大的数据一次性写入块设备,相比将多个数据分批次写入块设备,更加减少了磁头的往复摆动,以及减少了将数据写入块设备消耗的时间。Further, in the embodiment of the present application, by aggregating a plurality of data having adjacent write positions into a large data and writing it to the block device at one time, compared with writing a plurality of data into the block device in batches, the magnetic head is more reduced. The back-and-forth wobble and reduce the time it takes to write data to the block device.
下面对本申请实施例提供的装置进行描述。The device provided in the embodiments of the present application is described below.
参见图6,图6为本申请实施例提供的服务端的功能模块图。如图6所示,该服务端包括:Referring to FIG. 6, FIG. 6 is a functional module diagram of a server provided in an embodiment of the present application. As shown in Figure 6, the server includes:
接收单元601,用于接收客户端发送的第一待存储数据和所述第一待存储数据在块设备中的写入位置;The receiving unit 601 is configured to receive first data to be stored and a writing position of the first data to be stored in a block device sent by a client;
比对单元602,用于将所述第一待存储数据的写入位置与所述服务端中已生成的第一文件系统的区域位置信息进行比对;所述第一文件系统为所述服务端中已生成的多个文件系统之一,每个所述文件系统的区域位置信息根据所述块设备的存储空间大小生成,每个所述文件系统的区域位置信息包括该文件系统的元数据区和一般数据区的位置信息;A comparison unit 602 is configured to compare the writing position of the first to-be-stored data with the regional position information of the first file system generated in the server; the first file system is the service One of a plurality of file systems that have been generated in the client, each of the file system's area location information is generated according to the storage space size of the block device, and each of the file system's area location information includes metadata of the file system Location information of the data area and general data area;
写数据单元603,用于在所述比对单元602确定所述第一待存储数据的写入位置位于所述第一文件系统的元数据区内时,将所述第一待存储数据写入本服务端的缓存区。The data writing unit 603 is configured to write the first data to be stored when the comparison unit 602 determines that the writing position of the first data to be stored is located in a metadata area of the first file system. The cache area of this server.
在其中一种实施方式中,每个文件系统的元数据区包括多个子元数据区,所述第一 待存储数据在块设备中的写入位置包括所述第一待存储数据在块设备中的开始写入位置和结束写入位置;所述比对单元602,用于在确定所述第一待存储数据的开始写入位置和结束写入位置位于所述第一文件系统的同一个子元数据区内时,确定所述第一待存储数据的写入位置位于第一文件系统的元数据区内。In one embodiment, the metadata area of each file system includes multiple sub-metadata areas, and the writing position of the first data to be stored in the block device includes the first data to be stored in the block device. The starting writing position and the ending writing position; the comparison unit 602 is configured to determine that the starting writing position and the ending writing position of the first to-be-stored data are located in the same sub-unit of the first file system When in the data area, it is determined that the writing position of the first data to be stored is located in the metadata area of the first file system.
在其中一种实施方式中,所述接收单元601,还用于接收所述客户端发送的第二待存储数据和所述第二待存储数据在块设备中的写入位置;所述第二待存储数据在块设备中的写入位置包括所述第二待存储数据在块设备中的开始写入位置和结束写入位置;In one implementation manner, the receiving unit 601 is further configured to receive the second to-be-stored data and the writing position of the second to-be-stored data in the block device sent by the client; the second The writing position of the data to be stored in the block device includes a start writing position and an end writing position of the second data to be stored in the block device;
所述比对单元602,还用于将所述第二待存储数据的写入位置与所述第一文件系统的区域位置信息进行比对;如果所述第二待存储数据的开始写入位置和结束写入位置位于所述第一文件系统的不同子元数据区内,或者所述第二待存储数据的开始写入位置位于所述第一文件系统的元数据区内且所述第二待存储数据的结束写入位置位于所述第一文件系统的元数据区外,则将所述第二待存储数据的写入位置与所述多个文件系统中的第二文件系统的区域位置信息进行比对;The comparison unit 602 is further configured to compare the writing position of the second data to be stored with the area position information of the first file system; if the starting writing position of the second data to be stored is And the end writing position are located in different sub-metadata areas of the first file system, or the start writing position of the second to-be-stored data is located in the metadata area of the first file system and the second The end writing position of the data to be stored is outside the metadata area of the first file system, and then the writing position of the second data to be stored and the area position of the second file system in the plurality of file systems Information comparison
所述写数据单元603,还用于在所述比对单元602确定所述第二待存储数据的开始写入位置和结束写入位置位于所述第二文件系统的同一个子元数据区内时,将所述第二待存储数据写入本服务端的缓存区。The writing data unit 603 is further configured to: when the comparing unit 602 determines that the start writing position and the ending writing position of the second to-be-stored data are located in a same sub-metadata area of the second file system , Writing the second to-be-stored data into a cache area of the server.
在其中一种实施方式中,所述写数据单元603,还用于如果所述第一文件系统的元数据区范围[x1,y1]包含所述第二文件系统的元数据区范围[x2,y2],则将保存在本服务端缓存区中的开始写入位置和/或结束写入位置位于[x1,x2]内的数据、以及保存在本服务端的缓存区中的开始写入位置和/或结束写入位置位于[y2,y1]内的数据写入本服务端的块设备。In one embodiment, the write data unit 603 is further configured to: if the metadata area range [x1, y1] of the first file system includes the metadata area range [x2, y2], the data stored in the start write position and / or the end write position in [x1, x2] stored in the cache area of this server, and the start write position and / Or the end write data located in [y2, y1] is written to the block device of this server.
在其中一种实施方式中,所述写数据单元603,还用于如果所述第一文件系统的元数据区范围[x1,y1]与所述第二文件系统的元数据区范围[x2,y2]存在交集且x1>x2,y1>y2,则将保存在本服务端缓存区中的开始写入位置和/或结束写入位置位于[y2,y1]内的数据写入本服务端的块设备,以及将保存在本服务端块设备中的开始写入位置和结束写入位置均位于[x2,x1]内的数据写入本服务端的缓存区。In one implementation manner, the write data unit 603 is further configured to: if the range of the metadata area [x1, y1] of the first file system and the range of the metadata area [x2, y2] if there is an intersection and x1> x2, y1> y2, then write the data stored in the cache area of the server and / or the end write location in [y2, y1] to the block of the server The device, and the data stored in the start writing position and the end writing position stored in the block device of the server are both in [x2, x1] and written to the buffer area of the server.
在其中一种实施方式中,所述写数据单元603,还用于如果所述第一文件系统的元数据区范围[x1,y1]与所述第二文件系统的元数据区范围[x2,y2]存在交集且x1<x2,y1<y2,则将保存在本服务端缓存区中的开始写入位置和/或结束写入位置位于[x1,x2] 内的数据写入本服务端的块设备,以及将保存在本服务端块设备中的开始写入位置和结束写入位置均位于[y1,y2]内的数据写入本服务端的缓存区。In one implementation manner, the write data unit 603 is further configured to: if the range of the metadata area [x1, y1] of the first file system and the range of the metadata area [x2, y2] if there is an intersection and x1 <x2, y1 <y2, the data stored in the start and / or end write locations in the cache area of this server is written to the block of this server The device, as well as the data stored in the block device of the server at the beginning and end of writing, are located in the buffer area of the server.
在其中一种实施方式中,所述写数据单元603,还用于如果所述第二文件系统的元数据区范围[x2,y2]包含所述第一文件系统的元数据区范围[x1,y1],则将保存在本服务端块设备中的开始写入位置和结束写入位置均位于[x2,x1]内的数据、以及保存在本服务端块设备中的开始写入位置和结束写入位置均位于[y1,y2]内的数据写入本服务端的缓存区。In one embodiment, the write data unit 603 is further configured to: if the metadata area range [x2, y2] of the second file system includes the metadata area range [x1, y1], the data stored in the start block and end write positions of the server block device are both located in [x2, x1], and the start write position and end are stored in the block device of this server end The data whose write positions are located in [y1, y2] are written into the cache area of this server.
在其中一种实施方式中,所述写数据单元603,还用于在确定所述第一待存储数据的写入位置位于所述多个文件系统中每个文件系统的一般数据区内时,确定所述第一待存储数据为一般数据,判断本服务端缓存区的可用空间是否大于所述第一待存储数据的占用空间;如果是,则将所述第一待存储数据写入本服务端的缓存区;如果否,则根据预设的淘汰算法在本服务端缓存区保存的一般数据中确定淘汰数据;根据所述淘汰数据的写入位置,从本服务端缓存区中选择具有相邻写入位置的至少一个数据;将所述淘汰数据与所述具有相邻写入位置的至少一个数据一起写入本服务端的块设备。In one implementation manner, the write data unit 603 is further configured to, when it is determined that the write position of the first data to be stored is located in a general data area of each file system of the multiple file systems, Determine that the first data to be stored is general data, and determine whether the available space in the cache area of the server is larger than the occupied space of the first data to be stored; if so, write the first data to be stored in the service The cache area on the server side; if not, the elimination data is determined from the general data stored in the cache area of the server side according to a preset elimination algorithm; according to the write position of the elimination data, an adjacent area is selected from the cache area on the server side Write at least one data at a location; write the obsolete data together with the at least one data with an adjacent write location into a block device on the server side.
需要说明的是,本申请实施例中对单元的划分是示意性的,仅仅为一种逻辑功能划分,实际实现时可以有另外的划分方式。在本申请的实施例中的各功能单元可以集成在一个处理单元中,也可以是各个单元单独物理存在,也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现,也可以采用软件功能单元的形式实现。It should be noted that the division of the units in the embodiments of the present application is schematic, and is only a logical function division. There may be another division manner in actual implementation. The functional units in the embodiments of the present application may be integrated into one processing unit, or each of the units may exist separately physically, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional unit.
上述装置中各个单元的功能和作用的实现过程具体详见上述方法中对应步骤的实现过程,在此不再赘述。For details about the implementation process of the functions and functions of the units in the foregoing device, see the implementation process of the corresponding steps in the foregoing method for details, and details are not described herein again.
至此,完成图6所示装置的描述。So far, the description of the device shown in FIG. 6 is completed.
对应地,本申请实施例还提供了服务端的硬件结构。参见图7,图7为本申请实施例提供的服务端的硬件结构示意图,该服务端包括:处理器701、通信接口702、存储器703和总线704,其中,处理器701、通信接口702、存储器703通过总线相互连接。Correspondingly, the embodiment of the present application also provides a hardware structure of the server. Referring to FIG. 7, FIG. 7 is a schematic diagram of a hardware structure of a server provided in an embodiment of the present application. The server includes a processor 701, a communication interface 702, a memory 703, and a bus 704. Among them, the processor 701, the communication interface 702, and the memory 703 Connected to each other via a bus.
通信接口702,用于服务端与客户端的通信;A communication interface 702, configured to communicate between the server and the client;
存储器703,用于存储机器可读指令;A memory 703, configured to store machine-readable instructions;
处理器701,通过调用所述机器可读指令,执行图2所示的方法。The processor 701 executes the method shown in FIG. 2 by calling the machine-readable instructions.
本申请实施例还提供一种机器可读存储介质,所述机器可读存储介质存储有机器可读指令,所述机器可读指令在被处理器调用和执行时,所述机器可执行指令促使所述处理器实现图2所示的方法。An embodiment of the present application further provides a machine-readable storage medium that stores machine-readable instructions. When the machine-readable instructions are called and executed by a processor, the machine-executable instructions cause The processor implements the method shown in FIG. 2.
以上所述仅为本申请的较佳实施例而已,并不用以限制本申请,凡在本申请的精神和原则之内,所做的任何修改、等同替换、改进等,均应包含在本申请保护的范围之内。The above are only preferred embodiments of this application, and are not intended to limit this application. Any modification, equivalent replacement, or improvement made within the spirit and principles of this application shall be included in this application Within the scope of protection.

Claims (11)

  1. 一种数据写入方法,所述方法应用于服务端,包括:A data writing method applied to a server includes:
    接收客户端发送的第一待存储数据和所述第一待存储数据在块设备中的写入位置;Receiving the first to-be-stored data and the writing position of the first to-be-stored data in the block device sent by the client;
    将所述第一待存储数据的写入位置与所述服务端中已生成的第一文件系统的区域位置信息进行比对;所述第一文件系统为所述服务端中已生成的多个文件系统之一,每个所述文件系统的区域位置信息根据所述块设备的存储空间大小生成,每个所述文件系统的区域位置信息包括该文件系统的元数据区和一般数据区的位置信息;Comparing the writing position of the first to-be-stored data with the regional position information of the first file system that has been generated in the server; the first file system is a plurality of files that have been generated in the server One of the file systems, the area location information of each of the file systems is generated according to the storage space size of the block device, and the area location information of each of the file systems includes the location of a metadata area and a general data area of the file system information;
    在确定所述第一待存储数据的写入位置位于所述第一文件系统的元数据区内时,将所述第一待存储数据写入所述服务端的缓存区。When it is determined that the writing position of the first to-be-stored data is located in a metadata area of the first file system, the first to-be-stored data is written into a cache area of the server.
  2. 如权利要求1所述的方法,其特征在于,The method according to claim 1, wherein:
    每个所述文件系统的元数据区包括多个子元数据区,The metadata area of each of the file systems includes a plurality of sub-metadata areas,
    所述第一待存储数据在所述块设备中的写入位置包括所述第一待存储数据在所述块设备中的开始写入位置和结束写入位置;The writing position of the first data to be stored in the block device includes a start writing position and an end writing position of the first to be stored data in the block device;
    确定所述第一待存储数据的写入位置位于所述第一文件系统的元数据区内,包括:Determining that the writing position of the first to-be-stored data is located in a metadata area of the first file system includes:
    如果所述第一待存储数据的开始写入位置和结束写入位置位于所述第一文件系统的同一个子元数据区内,则确定所述第一待存储数据的写入位置位于所述第一文件系统的元数据区内。If the start writing position and the ending writing position of the first to-be-stored data are located in the same sub-metadata area of the first file system, determine that the writing position of the first to-be-stored data is located in the first A file system's metadata area.
  3. 如权利要求2所述的方法,还包括:The method of claim 2, further comprising:
    接收所述客户端发送的第二待存储数据和所述第二待存储数据在所述块设备中的写入位置;所述第二待存储数据在所述块设备中的写入位置包括所述第二待存储数据在所述块设备中的开始写入位置和结束写入位置;Receiving the second to-be-stored data and the writing position of the second to-be-stored data in the block device sent by the client; the writing position of the second to-be-stored data in the block device includes the The start writing position and the end writing position of the second to-be-stored data in the block device;
    将所述第二待存储数据的写入位置与所述第一文件系统的区域位置信息进行比对;Comparing the writing position of the second data to be stored with the area position information of the first file system;
    如果所述第二待存储数据的开始写入位置和结束写入位置位于所述第一文件系统的不同子元数据区内,或者所述第二待存储数据的开始写入位置位于所述第一文件系统的元数据区内且所述第二待存储数据的结束写入位置位于所述第一文件系统的元数据区外,则将所述第二待存储数据的写入位置与所述多个文件系统中的第二文件系统的区域位置信息进行比对;If the start writing position and the ending writing position of the second to-be-stored data are located in different sub-metadata areas of the first file system, or the start writing position of the second to-be-stored data is located in the first A file system's metadata area and the end writing position of the second to-be-stored data is outside the metadata area of the first file system, then writing the second to-be-stored data's writing position with the Comparing the location information of the second file system in the multiple file systems;
    在确定所述第二待存储数据的开始写入位置和结束写入位置位于所述第二文件系统的同一个子元数据区内时,将所述第二待存储数据写入所述服务端的缓存区。When it is determined that the start writing position and the ending writing position of the second to-be-stored data are located in the same sub-metadata area of the second file system, write the second to-be-stored data into a cache of the server Area.
  4. 如权利要求3所述的方法,其特征在于,在确定所述第二待存储数据的开始写入位置和结束写入位置位于所述第二文件系统的同一个子元数据区内之后,所述方法还 包括:The method according to claim 3, wherein after determining that the start writing position and the ending writing position of the second to-be-stored data are located in the same sub-metadata area of the second file system, the method The method also includes:
    如果所述第一文件系统的元数据区范围[x1,y1]包含所述第二文件系统的元数据区范围[x2,y2],则将保存在所述服务端的缓存区中的开始写入位置和/或结束写入位置位于[x1,x2]内的数据、以及保存在所述服务端的缓存区中的开始写入位置和/或结束写入位置位于[y2,y1]内的数据写入所述块设备。If the range [x1, y1] of the metadata area of the first file system includes the range [x2, y2] of the metadata area of the second file system, start writing to the cache area of the server Data whose position and / or end write position is in [x1, x2], and data write start and / or end write positions stored in the cache area of the server are written in [y2, y1] Into the block device.
  5. 如权利要求3所述的方法,其特征在于,在确定所述第二待存储数据的开始写入位置和结束写入位置位于所述第二文件系统的同一个子元数据区内之后,所述方法还包括:The method according to claim 3, wherein after determining that the start writing position and the ending writing position of the second to-be-stored data are located in the same sub-metadata area of the second file system, the method The method also includes:
    如果所述第一文件系统的元数据区范围[x1,y1]与所述第二文件系统的元数据区范围[x2,y2]存在交集且x1>x2,y1>y2,则If the metadata area range [x1, y1] of the first file system and the metadata area range [x2, y2] of the second file system intersect and x1> x2, y1> y2, then
    将保存在所述服务端的缓存区中的开始写入位置和/或结束写入位置位于[y2,y1]内的数据写入所述块设备,以及Writing data in the start writing position and / or ending writing position in [y2, y1] stored in the cache area of the server to the block device, and
    将保存在所述块设备中的开始写入位置和结束写入位置均位于[x2,x1]内的数据写入所述服务端的缓存区。Write the data stored in the block device at the start writing position and the end writing position in [x2, x1] into the cache area of the server.
  6. 如权利要求3所述的方法,其特征在于,在确定所述第二待存储数据的开始写入位置和结束写入位置位于所述第二文件系统的同一个子元数据区内之后,所述方法还包括:The method according to claim 3, wherein after determining that the start writing position and the ending writing position of the second to-be-stored data are located in the same sub-metadata area of the second file system, the method The method also includes:
    如果所述第一文件系统的元数据区范围[x1,y1]与所述第二文件系统的元数据区范围[x2,y2]存在交集且x1<x2,y1<y2,则If the metadata area range [x1, y1] of the first file system and the metadata area range [x2, y2] of the second file system intersect and x1 <x2, y1 <y2, then
    将保存在所述服务端的缓存区中的开始写入位置和/或结束写入位置位于[x1,x2]内的数据写入所述块设备,以及Writing data in the start writing position and / or the end writing position in [x1, x2] stored in the cache area of the server to the block device, and
    将保存在所述块设备中的开始写入位置和结束写入位置均位于[y1,y2]内的数据写入所述服务端的缓存区。Write the data stored in the block device at the start writing position and the end writing position in [y1, y2] into the cache area of the server.
  7. 如权利要求3所述的方法,其特征在于,在确定所述第二待存储数据的开始写入位置和结束写入位置位于所述第二文件系统的同一个子元数据区内之后,所述方法还包括:The method according to claim 3, wherein after determining that the start writing position and the ending writing position of the second to-be-stored data are located in the same sub-metadata area of the second file system, the method The method also includes:
    如果所述第二文件系统的元数据区范围[x2,y2]包含所述第一文件系统的元数据区范围[x1,y1],则将保存在所述块设备中的开始写入位置和结束写入位置均位于[x2,x1]内的数据、以及保存在所述块设备中的开始写入位置和结束写入位置均位于[y1,y2]内的数据写入所述服务端的缓存区。If the range [x2, y2] of the metadata area of the second file system includes the range [x1, y1] of the metadata area of the first file system, the start writing position and The data whose end writing position is located in [x2, x1], and the data whose start writing position and end writing position are stored in [y1, y2] stored in the block device are written into the cache of the server Area.
  8. 如权利要求1所述的方法,还包括:The method of claim 1, further comprising:
    在确定所述第一待存储数据的写入位置位于所述多个文件系统中每个所述文件系统的一般数据区内时,When it is determined that the writing position of the first to-be-stored data is located in a general data area of each of the file systems in the plurality of file systems,
    确定所述第一待存储数据为一般数据,Determining that the first data to be stored is general data,
    判断所述服务端的缓存区的可用空间是否大于所述第一待存储数据的占用空间;Determining whether the available space in the cache area of the server is larger than the occupied space of the first data to be stored;
    如果所述缓存区的可用空间大于所述第一待存储数据的占用空间,则将所述第一待存储数据写入所述服务端的缓存区;If the available space of the cache area is larger than the occupied space of the first data to be stored, writing the first data to be stored into the cache area of the server;
    如果所述缓存区的可用空间小于或等于所述第一待存储数据的占用空间,则If the available space of the buffer area is less than or equal to the occupied space of the first data to be stored, then
    根据预设的淘汰算法在所述服务端的缓存区保存的一般数据中确定淘汰数据;Determining elimination data from general data stored in a cache area of the server according to a preset elimination algorithm;
    根据所述淘汰数据的写入位置,从所述服务端的缓存区中选择具有相邻写入位置的至少一个数据;并Selecting at least one data with an adjacent write position from a cache area of the server according to the write position of the obsolete data; and
    将所述淘汰数据与所述至少一个数据一起写入所述块设备。Write the obsolete data together with the at least one data into the block device.
  9. 如权利要求1所述的方法,还包括:The method of claim 1, further comprising:
    建立文件系统模板库;Establish a file system template library;
    将所述块设备的存储空间大小作为输入参数传入所述文件系统模板库;Sending the size of the storage space of the block device as an input parameter into the file system template library;
    利用所述文件系统模板库生成与所述块设备的存储空间大小相匹配的所述多个文件系统的区域位置信息。Use the file system template library to generate region location information of the multiple file systems that match the storage space size of the block device.
  10. 一种服务端,包括:A server, including:
    接收单元,用于接收客户端发送的第一待存储数据和所述第一待存储数据在块设备中的写入位置;A receiving unit, configured to receive the first to-be-stored data sent by the client and a writing position of the first to-be-stored data in the block device;
    比对单元,用于将所述第一待存储数据的写入位置与所述服务端中已生成的第一文件系统的区域位置信息进行比对;所述第一文件系统为所述服务端中已生成的多个文件系统之一,每个所述文件系统的区域位置信息根据所述块设备的存储空间大小生成,每个所述文件系统的区域位置信息包括该文件系统的元数据区和一般数据区的位置信息;A comparison unit, configured to compare the writing position of the first to-be-stored data with the area position information of the first file system that has been generated in the server; the first file system is the server One of a plurality of file systems that have been generated in the system, each of the file system's area location information is generated according to the storage space size of the block device, and each of the file system's area location information includes a metadata area of the file system And the location information of the general data area;
    写数据单元,用于在所述比对单元确定所述第一待存储数据的写入位置位于所述第一文件系统的元数据区内时,将所述第一待存储数据写入所述服务端的缓存区。A data writing unit, configured to write the first data to be stored when the comparison unit determines that a writing position of the first data to be stored is located in a metadata area of the first file system; The cache area of the server.
  11. 一种服务端,包括处理器、通信接口、存储器和总线,其中,所述处理器、所述通信接口、所述存储器通过所述总线相互连接;A server includes a processor, a communication interface, a memory, and a bus, wherein the processor, the communication interface, and the memory are connected to each other through the bus;
    所述存储器中存储有机器可读指令,所述处理器通过调用所述机器可读指令执行如权利要求1至9任一项所述的方法。Machine-readable instructions are stored in the memory, and the processor executes the method according to any one of claims 1 to 9 by calling the machine-readable instructions.
PCT/CN2019/098338 2018-07-31 2019-07-30 Data writing method and server WO2020024933A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810857497.7A CN110795386B (en) 2018-07-31 2018-07-31 Data writing method and server
CN201810857497.7 2018-07-31

Publications (1)

Publication Number Publication Date
WO2020024933A1 true WO2020024933A1 (en) 2020-02-06

Family

ID=69230981

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/098338 WO2020024933A1 (en) 2018-07-31 2019-07-30 Data writing method and server

Country Status (2)

Country Link
CN (1) CN110795386B (en)
WO (1) WO2020024933A1 (en)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113296700A (en) * 2021-04-20 2021-08-24 新华三信息技术有限公司 Storage space management method and device and server
CN113986838B (en) * 2021-12-28 2022-03-11 成都云祺科技有限公司 Mass small file processing method and system based on file system and storage medium
CN114356241B (en) * 2021-12-28 2023-11-03 天翼云科技有限公司 Small object data storage method, small object data storage device, electronic equipment and readable medium

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103399823A (en) * 2011-12-31 2013-11-20 华为数字技术(成都)有限公司 Method, equipment and system for storing service data
CN105117351A (en) * 2015-09-08 2015-12-02 华为技术有限公司 Method and apparatus for writing data into cache
WO2015183240A1 (en) * 2014-05-27 2015-12-03 Hewlett-Packard Development Company, L.P. Storing compressed data
CN107153508A (en) * 2016-03-03 2017-09-12 杭州海康威视数字技术股份有限公司 A kind of date storage method and device
CN108021513A (en) * 2016-11-02 2018-05-11 杭州海康威视数字技术股份有限公司 A kind of date storage method and device

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7752391B2 (en) * 2006-01-20 2010-07-06 Apple Inc. Variable caching policy system and method
CN102103880B (en) * 2009-12-22 2014-02-19 鸿富锦精密工业(深圳)有限公司 Universal serial bus (USB) flash disk and memory capacity display method thereof
CN103593262B (en) * 2013-11-15 2016-05-25 上海爱数信息技术股份有限公司 A kind of virtual machine backup method based on classification
CN107451138A (en) * 2016-05-30 2017-12-08 中兴通讯股份有限公司 A kind of distributed file system storage method and system
CN107491523B (en) * 2017-08-17 2020-05-05 三星(中国)半导体有限公司 Method and device for storing data object

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103399823A (en) * 2011-12-31 2013-11-20 华为数字技术(成都)有限公司 Method, equipment and system for storing service data
WO2015183240A1 (en) * 2014-05-27 2015-12-03 Hewlett-Packard Development Company, L.P. Storing compressed data
CN105117351A (en) * 2015-09-08 2015-12-02 华为技术有限公司 Method and apparatus for writing data into cache
CN107153508A (en) * 2016-03-03 2017-09-12 杭州海康威视数字技术股份有限公司 A kind of date storage method and device
CN108021513A (en) * 2016-11-02 2018-05-11 杭州海康威视数字技术股份有限公司 A kind of date storage method and device

Also Published As

Publication number Publication date
CN110795386B (en) 2022-07-01
CN110795386A (en) 2020-02-14

Similar Documents

Publication Publication Date Title
US10635310B2 (en) Storage device that compresses data received from a host before writing therein
US7788460B2 (en) Defragmenting objects in a storage medium
US7120767B2 (en) Snapshot creating method and apparatus
US8521985B2 (en) Storage subsystem
US6598129B2 (en) Storage device and method for data sharing
US9311002B1 (en) Systems, methods, and computer readable media for compressing data at a virtually provisioned storage entity
US10635329B2 (en) Method and apparatus for performing transparent mass storage backups and snapshots
US8694563B1 (en) Space recovery for thin-provisioned storage volumes
US9778860B2 (en) Re-TRIM of free space within VHDX
US6205529B1 (en) Method and apparatus for defragmenting a storage device using a copy function in the device control logic
WO2020024933A1 (en) Data writing method and server
US10585594B1 (en) Content-based caching using digests
US9922039B1 (en) Techniques for mitigating effects of small unaligned writes
US11403021B2 (en) File merging method and controller
US9398073B1 (en) Methods and apparatus for providing content
US9367256B2 (en) Storage system having defragmentation processing function
US9430492B1 (en) Efficient scavenging of data and metadata file system blocks
US20050108296A1 (en) File system preventing file fragmentation
US11132145B2 (en) Techniques for reducing write amplification on solid state storage devices (SSDs)
US8255611B1 (en) Methods and apparatus for accessing content
US10180901B2 (en) Apparatus, system and method for managing space in a storage device
WO2015058641A1 (en) Hybrid hard disk implementation method and device
US6601135B1 (en) No-integrity logical volume management method and system
US10089125B2 (en) Virtual machines accessing file data, object data, and block data
US8489559B2 (en) Methods and apparatus for conversion of content

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19843206

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19843206

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 19843206

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 11.08.2021)

122 Ep: pct application non-entry in european phase

Ref document number: 19843206

Country of ref document: EP

Kind code of ref document: A1