CN109358818B

CN109358818B - Block device IO request processing method of data center

Info

Publication number: CN109358818B
Application number: CN201811276709.9A
Authority: CN
Inventors: 杨少白
Original assignee: Shenzhen Runxun Data Communication Co ltd
Current assignee: Shenzhen Runxun Data Communication Co ltd
Priority date: 2018-10-30
Filing date: 2018-10-30
Publication date: 2021-08-03
Anticipated expiration: 2038-10-30
Also published as: CN109358818A

Abstract

The invention relates to the technical field of data centers, provides a block device IO request processing method of a data center, and comprises the following steps: s1, loading a block device driver by a user; s2, generating a block device IO request by the file system according to the access request of the user to the block device to access the block device; s3, the block device responds and executes the block device IO request. In the invention, the kernel virtual address space obtained by applying the RAMDISK to the Linux system is aligned according to 2MB as the boundary, and the mapping relation which is not used for the longest time is searched and disconnected to establish the dynamic mapping relation between the physical address space of the block device and the kernel virtual address space, thereby effectively reducing the number of virtual address spaces applied by the RAMDISK to the Linux kernel and breaking through the capacity limit of the RAMDISK.

Description

Block device IO request processing method of data center

Technical Field

The invention relates to the technical field of data centers, in particular to a block device IO request processing method of a data center.

Background

The storage system is a core infrastructure of a data center IT environment and is a final bearer for data access of the data center. The storage system is greatly changed after related technologies such as cloud computing, virtualization and big data enter, and reading of multiple data types is supported by block storage, file storage and object storage. Therefore, the safety of the storage system has fundamental decisive significance for the normal operation of the data center.

At present, the storage data type of a data center, especially a large data center, a cloud data center or an enterprise-level data center, is mainly storage and application of structured data, and is a user-defined data type which includes a series of attributes, each attribute having a data type, stored in a relational database. A general business system has a large amount of structured data, and the structured data is generally stored in a relational database such as Oracle or MySQL and the like; in an enterprise-level data center, the data is generally stored in a centralized storage architecture or becomes a main storage system, and a storage device in a physical form of a solid state disk or a mechanical hard disk is mainly used for accessing a block device.

RAMDISK is used by virtualizing memory into a hard disk. Typically, an Operating System (OS) divides a portion of a physical address space into a memory for use as a hard disk. Because the IO request speed between the operating system and the memory is far higher than that of the traditional mechanical hard disk, the data transmission speed of the RAMDISK is far higher than that of the mechanical hard disk and is close to that between the CPU and the memory.

The existing computer system (taking X86 as an example) does not support DMA operation between the memory and the memory, which makes the operating system need to frequently apply for precious kernel virtual address space from the operating system and establish a mapping relationship between the physical address space of the memory and the kernel virtual address space when processing large-scale data transmission in the memory.

In a 32-bit Linux system, the kernel virtual address space is set to 1G and includes three ZONEs ZONE _ DMA, ZONE _ NORMAL, and ZONE _ HIGHMEM. The ZONE _ DMA is positioned in a first section area of the physical memory, is mainly used for ISA equipment and has the size of 16 MB; ZONE _ NORMAL is located after ZONE _ DMA; ZONE _ HIGHMEM refers to the physical memory left in the system, which cannot be directly mapped by the Linux kernel.

When the RAMDISK applies for the virtual address space from the Linux kernel, the RAMDISK applies for the kernel virtual address space from ZONE _ NORMAL, and the theoretical maximum value is 880 MB. However, when the Linux system is actually operated, the assignable kernel virtual address space is only 96MB at most. With the increase of applications in the Linux system and the increase of peripheral devices mounted on the hardware system, the 96MB kernel virtual address space has not been very satisfactory for the increasing demand of the physical address space, so that the capacity of the RAMDISK in the prior art is greatly limited.

Disclosure of Invention

The invention aims to provide a block device IO request processing method of a data center, which is used for reducing excessive kernel virtual address space application of an RAMDISK to a Linux system and breaking through the limit of the RAMDISK capacity.

In order to achieve the above object, the present invention provides a block device IO request processing method for a data center, including the following steps:

s1, loading a block device driver by a user;

s2, generating a block device IO request by the file system according to the access request of the user to the block device to access the block device;

s3, the block device responds and executes the block device IO request.

As a further improvement of the present invention, the step S1 specifically includes:

s11, generating an initialization mapping page table;

s12, initializing a spin lock;

s13, registering a block device;

s14, creating and constructing request parameters;

s15, binding construction request parameters;

s16, creating and initializing a disk;

and S17, adding the disk into the Linux system.

As a further improvement of the present invention, the step S11 specifically includes: the block device circularly applies for the kernel virtual address space from the Linux kernel and judges whether the application is successful or not;

if so, aligning the kernel virtual address space obtained by applying according to 2MB as a boundary, establishing a mapping relation with the physical address space of the block device, and then storing the mapping relation to the structure body;

if not, returning and reporting error information to the block equipment;

and finally, establishing a plurality of mapping relations between the physical address space of the block device and the kernel virtual address space, and generating an initialized mapping page table.

As a further improvement of the present invention, the step S11 of "saving the mapping relationship to the structure" specifically includes:

and storing the initial information and the end information of the kernel virtual address space and the physical address space of the block device to a structural body.

As a further improvement of the present invention, the number of mapping relationships established between the physical address space of the block device and the kernel virtual address space is greater than or equal to four and less than or equal to eight.

As a further improvement of the present invention, the "execute block device IO request" in step S3 specifically includes:

the physical address space of the block data to be accessed in the block device IO request generated by the file system is searched and/or newly created in the initialized mapping page table generated in step S11, and the mapping relation matching with the physical address space of the block device is updated, and then block data transmission is performed to complete the block device IO request.

As a further improvement of the present invention, the step S14 specifically includes:

judging whether the physical address space corresponding to the block data to be accessed by the block device IO request is out of bounds compared with the physical address space of the actual block device by the Linux system;

if not, traversing each segment of the block device IO request in the physical address space of the block device to map the data corresponding to the block device IO request to the kernel virtual address space, and then performing block data transmission until the block device IO request is executed;

if yes, the block device returns error information to end the block device IO request.

As a further improvement of the invention, a mutual exclusion locking operation is also included before the 'block data transmission'; a mutually exclusive lock unlock operation is also included after the "block data transfer".

As a further improvement of the present invention, the "block data transmission" specifically includes: judging whether the Cache is hit or not;

if the block is hit in the Cache, further judging whether the IO request of the block device is a read operation;

if not, finding and disconnecting the mapping which is not used for the longest time, storing the kernel virtual address space attribute in the disconnected mapping relation, finally establishing a new mapping relation between the physical address space corresponding to the block data to be transmitted and the disconnected kernel virtual address space through the attribute of the kernel virtual address space reserved in the global variable, storing the attribute of the kernel virtual address space in the newly established mapping relation into the global variable, and updating the mapping page table.

As a further improvement of the present invention, the "block data transmission" further includes: judging whether the block device IO request is read operation;

if yes, writing the block data into the memory;

if not, writing the block data into the disk.

Compared with the prior art, the invention has the beneficial effects that: in the invention, the kernel virtual address space obtained by applying the RAMDISK to the Linux system is aligned according to 2MB as the boundary, and the mapping relation which is not used for the longest time is searched and disconnected to establish the dynamic mapping relation between the physical address space of the block device and the kernel virtual address space, thereby effectively reducing the number of virtual address spaces applied by the RAMDISK to the Linux kernel and breaking through the limitation of the RAMDISK capacity.

Drawings

Fig. 1 is a flowchart of a block device IO request processing method of a data center according to the present invention;

FIG. 2 is a table of initialized mapping pages generated by a block device;

FIG. 3 is a model diagram of a mapping relationship between a kernel virtual address space in the initialized mapping page table shown in FIG. 2 and a physical address space of block data corresponding to a block device IO request;

FIG. 4 is a table of updated mapping tables generated by dynamic mapping of block devices;

FIG. 5 is a model diagram of a mapping relationship between a kernel virtual address space and a physical address space of block data corresponding to a block device IO request in the dynamically generated mapping page table shown in FIG. 3;

FIG. 6 is a flowchart of the present invention wherein the user loads the device driver of the block in step S1;

FIG. 7 is a logic diagram of the present invention for generating the initialized mapping page table in step S11;

FIG. 8 is a logic diagram of the creation of request parameters in step S14 according to the present invention;

FIG. 9 is a logic diagram of a block data transfer process of the present invention;

fig. 10 is a graph showing the change of the IOPS value when random access and sequential access are performed when different numbers of mapping relationships are respectively established in the present invention.

Detailed Description

The present invention is described in detail with reference to the embodiments shown in the drawings, but it should be understood that these embodiments are not intended to limit the present invention, and those skilled in the art should understand that functional, methodological, or structural equivalents or substitutions made by these embodiments are within the scope of the present invention.

SUMMARY：

A File System (FS) is a type of object store that can hold either files (where the object ID is its pathname) or objects (where the object is referenced by some ID of the File System).

The RAMDISK may further include an interface module (not shown) connected to a bus of the computer system, a control chip, and a volatile memory connected to at least the control chip. The control chip can be an Application Specific Integrated Circuit (ASIC) and can also be a Field Programmable Gate Array (FPGA). The RAMDISK can be used for executing data access operation by CPU path line of computer through system bus via interface module.

The embodiments of the present invention are illustrated by RAMDISK running Linux system. In the Linux environment, from the bottom layer to the top layer, the overall structure is as follows: hardware layer, driver layer, Operating System (OS), File System (FS), Virtual File System (VFS), and USER layer (USER). The user sends an instruction through the virtual file system, and realizes storage operation or operation of data in various physical devices (a memory, a hard disk, network equipment, graphic processing equipment and the like) in the hardware layer through the operating system and the driver layer.

Due to the characteristics of the RAMDISK technical architecture, a queue request is not needed to realize the data caching function, and the data caching function can be directly accessed by a CPU; and the CPU performs the transmission operation in the unit of block data when accessing the RAMDISK, but before the transmission of the block data, whether the physical address space corresponding to the block data is mapped to the kernel virtual address space by the page table needs to be confirmed.

Please refer to fig. 1, which illustrates a block device IO request processing method of a data center, including the following steps: s1, loading a block device driver by a user; s2, generating a block device IO request by the file system according to the access request of the user to the block device to access the block device; s3, the block device responds and executes the block device IO request.

As shown in fig. 6, in the present embodiment, the step S1 specifically includes: s11, generating an initialization mapping page table; s12, initializing a spin lock; s13, registering a block device; s14, creating and constructing request parameters; s15, binding construction request parameters; s16, creating and initializing a disk; and S17, adding the disk into the Linux system.

As shown in fig. 7, in the present embodiment, step S11 specifically includes:

step 201 is executed first: the block device circularly applies for a kernel virtual address space from a Linux kernel; then, the jump execution step 202: to determine whether the application was successful.

If the application is successful, skipping to execute step 203, and aligning the kernel virtual address space obtained by the application according to 2MB as a boundary; then, continuing to execute step 204, and establishing a mapping relation between the kernel virtual address space obtained by applying and the physical address space of the block device; then, continuing to execute step 205 to store the mapping relationship to the structure; finally, the execution step 206 is skipped to determine whether the block device applies the kernel virtual address space operation to the Linux system for loop completion. If the execution is finished, the operation is finished; if not, returning to and continuing to execute step 201.

If the application is unsuccessful, the block device returns and reports error information, and then the operation is finished.

Finally, a plurality of mapping relations are established between the physical address space of the block device and the kernel virtual address space, and an initialization mapping page table (namely, the initialization mapping page table shown in fig. 2) is generated.

Referring to fig. 2 and fig. 3 together, in the initialized mapping page table generated in step S11, v0 to v3 respectively represent kernel virtual address spaces, and h _1 to h _3 respectively represent physical address spaces of block data in the memory to be operated by the block device IO request that establishes the initialized mapping relationship with the kernel virtual address spaces.

The step S11 of "storing the mapping relationship to the structure" specifically includes: and storing the initial information and the end information of the kernel virtual address space and the physical address space of the block device to a structural body.

In this embodiment, the "execute block device IO request" in step S3 is specifically:

If the block data to be accessed can find the kernel virtual address space corresponding to the block data in the initialized mapping page table, further judging whether the block data is read operation or write operation. If the data is read operation, writing the data into a physical address space in the memory; if the data is not a read operation (i.e. a write operation), the data is written into a disk of the computer.

In this embodiment, only when the physical address space of the block data cannot be searched in the initialized mapping page table generated in step S11, the mapping that is not used last time is searched and broken, and the space attribute of the kernel virtual address in the broken mapping relationship is saved, however, a new mapping relationship is created in the Linux kernel for the physical address space of the block data, and a mapping page table with a new mapping relationship between the kernel virtual address space in the dynamically generated mapping page table and the physical address space of the block data corresponding to the block device IO request is formed as shown in fig. 4 and fig. 5.

In fig. 4 and 5, it is assumed that the physical address space of the block data corresponding to the block device IO request is h _ n, and this physical address space is not found in the initialized mapping page table formed in step S11 in fig. 2 and 3. At this time, if the page mapping relationship established between v3 and h _3 conforms to the least recently used situation, the page mapping relationship between v3 and h _3 is disconnected, and a new page mapping relationship is established between v3 and h _ n instead.

And the method is analogized and circularly repeated, so that the large-capacity RAMDISK is mapped by a small amount of Linux kernel virtual address space. Generally speaking, as long as the capability of the operating system to the addressing space of the memory allows, the RAMDISK can expand the capacity without limit, thereby breaking through the limit of the RAMDISK capacity, so that the RAMDISK formed by the invention can support more file systems between the operating system layer and the hardware layer, so as to satisfy the increase of the data of the file systems formed by the increasingly-increased peripheral devices of the computer, prevent the Linux kernel virtual address space from extruding the user space, and indirectly improve the performance of the computer using the RAMDISK.

As shown in fig. 8, in the present embodiment, the step S14 specifically includes:

in step 301, a block device IO request is generated according to an instruction sent to the computer by the user through the virtual file system, and then step 302 is executed to determine whether a physical address space corresponding to block data to be accessed by the block device IO request is out of bounds compared with a physical address space of an actual block device.

If not (i.e., not out of bounds), executing step 303, traversing each segment of the block device IO request in the physical address space of the block device; step 304 is then performed to map the data corresponding to the block device IO request to the kernel virtual address space, and step 306 is then performed to perform block data transmission until the block device IO request is executed.

If yes (and out of bounds), the block device returns an error message to end the block device IO request.

Finally, the step 308 is executed to determine whether the traversal is completed, and if so, the process of executing the IO request of the block device is reported to be finished.

Preferably, before performing step 306 "block data transmission", the method further comprises performing step 305 a mutual exclusion locking operation; the method also includes executing 307 a mutual exclusive lock unlock operation after executing 306 the block data transmission. In this embodiment, step 305 and step 306 are respectively executed before and after the block data transmission, so as to ensure that only one thread can access the object (i.e. the block data) at any time, and the integrity of the block data operation can be effectively ensured, so that the Linux system based on the RAMDISK has ideal stability and reliability during the data operation.

Referring to fig. 9, in this embodiment, step 306 "block data transmission" in fig. 8 specifically includes: step 401 is executed to determine whether the Cache is hit.

If yes, indicating that the Cache is hit, skipping to execute step 405 to further determine whether the block device IO request is a read operation.

If not, the Cache is not hit on the surface, the step 402 is skipped to, the mapping which is not used for the longest time is searched and disconnected, and the kernel virtual address space attribute in the disconnected mapping relation is stored; then, step 403 is executed to establish a new mapping relationship between the physical address space corresponding to the block data to be transmitted and the disconnected kernel virtual address space through the attribute of the kernel virtual address space reserved in the global variable, and to store the attribute of the kernel virtual address space in the newly established mapping relationship into the global variable; then, step 404 is executed to further map the page table; finally, jumping to execute step 405 to determine whether the block device IO request is a read operation.

Specifically, after the determination in step 405 is performed, if yes, it indicates that the block device IO request is a read operation, and step 406 is performed to write the block data corresponding to the block device IO request into the memory, and then the process is ended.

If not, it indicates that the block device IO request is a write operation, and step 407 is executed to write the block data corresponding to the block device IO request to the disk, and then the process ends.

In the present embodiment, an update time parameter of the block data is saved in the global variable, and it is through this update time parameter that the Linux kernel determines which page mapping relationship in the initialization mapping page table is the "least recently used" page mapping relationship as shown in fig. 2.

As shown in fig. 10, in this embodiment, the number of mapping relationships established between the physical address space of the block device and the kernel virtual address space is greater than or equal to four and less than or equal to eight, and preferably four mapping relationships are established.

In RAMDISK, when a user sends an instruction through a virtual file system and accesses data in a physical memory through an operating system and a driver layer, the user operates according to Block data (Block) as a unit; however, in a specific physical memory address segment, Page data (Page) is still used as the minimum unit for access. Therefore, a specific physical address space of a specific physical memory address segment in the RAMDISK is quickly found, and a random access mechanism is adopted in the process.

In the process of random access, the relationship between the number of mapping relationships established between the physical address space of the block device and the kernel virtual address space and the resource consumption of the computer is in inverse proportion, that is, the greater the number of the established mapping relationships, the lower the IOPS value (Input/Output Per Second) of the computer, and the lower the performance of the computer.

However, when the block data corresponding to the block device IO request is found in the specific physical address space of the RAMDISK, sequential access of data in units of Page data (Page) is started. In the computer sequential access mechanism, if the number of mapping relationships established between the physical address space of a block device and the kernel virtual address space is larger, the access speed of page data is higher.

In the prior art, when a Linux system based on a RAMDISK technical framework runs actually, the distributed kernel virtual address space is limited to 96 MB; therefore, it is also of no substantial effect to blindly increase the number of mappings established between the physical address space of the block device and the kernel virtual address space.

As can be seen from the "sequential access IOPS value" curve 501 in fig. 10, when the number of established mappings reaches four or more, the IOPS value at the time of sequential access substantially tends to be saturated.

Meanwhile, when the number of mappings established between the physical address space of the block device and the kernel virtual address space is three or less, it is the highest region in the "random access IOPS value" curve 502, but after combining the "sequential access IOPS value" curve 501 in fig. 10, we find that in this case, the access speed of the executed Page data (Page) is also relatively low.

Further, we have found that in the "random access IOPS value" curve 502 in fig. 10, when the number of mappings established between the physical address space of the block device and the kernel virtual address space is four to eight, the IOPS value at the time of random access does not change much, while the IOPS value at the time of sequential access is also higher.

Finally, when the number of the mapping relations established between the physical address space of the block device and the kernel virtual address space is more than eight, the IOPS value in random access begins to decrease; at the same time, the IOPS value in the sequential access does not increase because of saturation.

The occupancy rate of computer resources is less when the quantity of the mapping relation established between the physical address space and the kernel virtual address space of the block device is less; therefore, in the present embodiment, the number of mapping relationships established between the physical address space of the block device and the kernel virtual address space is four.

In the invention, the kernel virtual address space obtained by applying the RAMDISK to the Linux system is aligned according to 2MB as the boundary, and the mapping relation which is not used for the longest time is searched and disconnected to establish the dynamic mapping relation between the physical address space of the block device and the kernel virtual address space, thereby effectively reducing the number of virtual address spaces applied by the RAMDISK to the Linux kernel and breaking through the limitation of the RAMDISK capacity.

In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules or units is only one logical division, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, a network device, or the like) or a processor (processor) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.

The above-listed detailed description is only a specific description of a possible embodiment of the present invention, and they are not intended to limit the scope of the present invention, and equivalent embodiments or modifications made without departing from the technical spirit of the present invention should be included in the scope of the present invention.

It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned.

Furthermore, it should be understood that although the present description refers to embodiments, not every embodiment may contain only a single embodiment, and such description is for clarity only, and those skilled in the art should integrate the description, and the embodiments may be combined as appropriate to form other embodiments understood by those skilled in the art.

Claims

1. A block device IO request processing method of a data center is characterized by comprising the following steps:

s1, loading a block device driver by a user;

s3, responding and executing the block device IO request by the block device;

the step S1 specifically includes:

s11, generating an initialization mapping page table;

s12, initializing a spin lock;

s13, registering a block device;

s14, creating and constructing request parameters;

s15, binding construction request parameters;

s16, creating and initializing a disk;

s17, adding the disk into the Linux system;

the step S11 specifically includes: the block device circularly applies for the kernel virtual address space from the Linux kernel and judges whether the application is successful or not;

if not, returning and reporting error information to the block equipment;

2. The method according to claim 1, wherein the step S11 of "saving the mapping relationship to the structure" is specifically:

3. The method of claim 1, wherein the number of mapping relationships established between the physical address space of the block device and the kernel virtual address space is greater than or equal to four and less than or equal to eight.

4. The method according to claim 1, wherein the "execute block device IO request" in step S3 is specifically:

5. The method according to claim 1 or 4, wherein the step S14 is specifically:

6. The method of claim 5, further comprising a mutually exclusive locking operation prior to said "block data transfer"; a mutually exclusive lock unlock operation is also included after the "block data transfer".

7. The method according to claim 5, wherein the "block data transmission" is specifically: judging whether the Cache is hit or not;

8. The method of claim 7, wherein the "block data transmission" further comprises: judging whether the block device IO request is read operation;

if yes, writing the block data into the memory;

if not, writing the block data into the disk.