CN113296685B - Data processing method and device and computer readable storage medium - Google Patents
Data processing method and device and computer readable storage medium Download PDFInfo
- Publication number
- CN113296685B CN113296685B CN202010472738.3A CN202010472738A CN113296685B CN 113296685 B CN113296685 B CN 113296685B CN 202010472738 A CN202010472738 A CN 202010472738A CN 113296685 B CN113296685 B CN 113296685B
- Authority
- CN
- China
- Prior art keywords
- metadata
- memory
- large page
- detection data
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 23
- 230000015654 memory Effects 0.000 claims abstract description 252
- 238000001514 detection method Methods 0.000 claims abstract description 97
- 238000000034 method Methods 0.000 claims abstract description 37
- 238000012545 processing Methods 0.000 claims description 33
- 238000013507 mapping Methods 0.000 claims description 22
- 238000005192 partition Methods 0.000 claims description 16
- 238000004590 computer program Methods 0.000 claims description 7
- 238000012795 verification Methods 0.000 abstract description 6
- 238000009792 diffusion process Methods 0.000 abstract description 2
- 230000008569 process Effects 0.000 description 18
- 238000002955 isolation Methods 0.000 description 11
- 238000010586 diagram Methods 0.000 description 6
- 238000007726 management method Methods 0.000 description 6
- 238000004422 calculation algorithm Methods 0.000 description 4
- 238000006243 chemical reaction Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 230000005055 memory storage Effects 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000005856 abnormality Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 230000007723 transport mechanism Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
- G06F3/0619—Improving the reliability of storage systems in relation to data integrity, e.g. data losses, bit errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
Abstract
The application discloses a data processing method and device and a computer readable storage medium, wherein the data processing method comprises the following steps: creating metadata detection data for the metadata, wherein the metadata is stored in a first memory, the metadata detection data is stored in a second memory, and physical memory spaces of the first memory and the second memory are isolated from each other; receiving a metadata access request of a client, and detecting whether metadata is correct or not based on metadata detection data; when the metadata is correct, the metadata access request of the client is processed based on the metadata. The method and the device ensure the data correctness during metadata access, improve the metadata verification efficiency and avoid dirty data diffusion or the situation that a user reads wrong data.
Description
Technical Field
The present disclosure relates to the field of data storage technologies, but not limited to, and in particular, to a data processing method and apparatus, and a computer readable storage medium.
Background
In a storage system, metadata is data describing data, and correctness of the metadata is related to data security of the entire storage system. In order to ensure the correctness of the metadata, a data checking mechanism needs to be added to the metadata.
The current common practice is to add metadata check fields in each metadata unit to ensure the data correctness of the whole metadata unit, but if the calculation and comparison of the check fields are carried out every time metadata access is carried out, certain loss is caused to the performance, so that the current storage system generally does not carry out metadata correctness check before metadata access, but calculates the metadata check value only when metadata is updated and persists into the storage medium for checking silence errors in the storage medium. However, if the metadata memory itself is dirty due to memory trampling or the like, the actual data block accessed according to the dirty metadata may be erroneous, which may result in loss of user data or reading of erroneous data.
Disclosure of Invention
The application provides a data processing method and device and a computer readable storage medium, which can ensure the data correctness during metadata access and improve the metadata verification efficiency.
The embodiment of the application provides a data processing method, which comprises the following steps: creating metadata detection data for metadata, wherein the metadata is stored in a first memory, the metadata detection data is stored in a second memory, and physical memory spaces of the first memory and the second memory are isolated from each other; receiving a metadata access request of a client, and detecting whether the metadata is correct or not based on the metadata detection data; and when the metadata is correct, processing the metadata access request of the client based on the metadata.
In some possible implementations, the method further includes: and when the metadata is wrong, returning the metadata to the client.
In some possible implementations, the method further includes: and receiving a metadata update request, and synchronously updating the metadata and the metadata detection data.
In some possible implementations, the first memory is a small page memory and the second memory is a large page memory.
In some possible implementations, the method further includes: creating a large page memory pool for each metadata partition, wherein the metadata partition comprises a plurality of metadata, one large page memory pool is used for storing metadata detection data of a plurality of metadata of one metadata partition, the large page memory pool comprises a plurality of large pages and a plurality of sections of virtual memory spaces, each large page is mapped to one section of virtual memory space, the sections of virtual memory spaces in one large page memory pool are continuous, and the base addresses of the virtual memory spaces in different large page memory pools are different.
In some possible implementations, the method further includes: and creating a memory pool mapping table for each large page memory pool, wherein the memory pool mapping table is used for describing the mapping relation between each large page and each segment of virtual memory space in the large page memory pool.
In some possible implementations, the metadata detection data is a copy of metadata.
The detecting whether the metadata is correct based on the metadata detection data includes: detecting whether the metadata and the metadata copy are consistent; when the metadata is consistent with the metadata copy, judging that the metadata is correct; and when the metadata and the metadata copy are inconsistent, judging that the metadata is wrong.
In some possible implementations, the metadata detection data is a metadata check value.
The detecting whether the metadata is correct based on the metadata detection data includes: calculating a check value of the metadata, and detecting whether the calculated check value is consistent with the check value of the metadata; when the calculated check value is consistent with the metadata check value, judging that the metadata is correct; and judging that the metadata is wrong when the calculated check value is inconsistent with the metadata check value.
The embodiment of the application also provides a data processing device, which comprises a processor and a memory, wherein the processor is used for executing a computer program stored in the memory to realize the steps of the data processing method.
Embodiments of the present application also provide a computer readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement the steps of the data processing method according to any one of the above.
The embodiment of the application also provides a data processing device, which comprises a first memory, a second memory, a metadata detection module and a metadata processing module, wherein: the metadata detection module is used for creating metadata detection data for metadata, the metadata is stored in the first memory, the metadata detection data is stored in the second memory, and physical memory spaces of the first memory and the second memory are isolated from each other; receiving a metadata access request of a client, and detecting whether the metadata is correct or not based on the metadata detection data; notifying a metadata processing module when the metadata is correct; and the metadata processing module is used for receiving the notification of the metadata detection module and processing the metadata access request of the client based on the metadata.
In some possible implementations, the data processing apparatus further includes a metadata update module, where the metadata update module is configured to receive a metadata update request and update the metadata and metadata detection data synchronously.
The data processing method and device and the computer readable storage medium comprise the steps of creating metadata detection data for metadata, storing the metadata in a first memory, storing the metadata detection data in a second memory, isolating physical memory spaces of the first memory and the second memory from each other, receiving a metadata access request of a client, and detecting whether the metadata is correct or not based on the metadata detection data; when the metadata is correct, the metadata access request of the client is processed based on the metadata, so that the data correctness during metadata access is ensured, the metadata verification efficiency is improved, and dirty data diffusion or the situation that a user reads wrong data is avoided.
Additional features and advantages of the application will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the application. Other advantages of the present application may be realized and attained by the structure particularly pointed out in the written description and drawings.
Drawings
The accompanying drawings are included to provide an understanding of the technical aspects of the present application, and are incorporated in and constitute a part of this specification, illustrate the technical aspects of the present application and together with the examples of the present application, and not constitute a limitation of the technical aspects of the present application.
FIG. 1 is a flow chart of a data processing method according to an embodiment of the present application;
FIG. 2 is a schematic diagram of a memory storage structure of metadata and metadata detection data according to an embodiment of the present application;
FIG. 3 is a schematic diagram of the overall structure of a large page memory pool according to an embodiment of the present application;
FIG. 4 is a schematic diagram of a virtual memory space in a large page memory pool according to an embodiment of the present application;
FIG. 5 is a schematic diagram of a memory mapping table in a large-page memory pool according to an embodiment of the present application;
FIG. 6 is a flowchart illustrating a metadata update process according to an embodiment of the present application;
FIG. 7 is a schematic diagram of a data processing apparatus according to an embodiment of the present application;
fig. 8 is a schematic structural diagram of another data processing apparatus according to an embodiment of the present application.
Detailed Description
The present application describes a number of embodiments, but the description is illustrative and not limiting and it will be apparent to those of ordinary skill in the art that many more embodiments and implementations are possible within the scope of the embodiments described herein. Although many possible combinations of features are shown in the drawings and discussed in the detailed description, many other combinations of the disclosed features are possible. Any feature or element of any embodiment may be used in combination with or in place of any other feature or element of any other embodiment unless specifically limited.
The present application includes and contemplates combinations of features and elements known to those of ordinary skill in the art. The embodiments, features and elements of the present disclosure may also be combined with any conventional features or elements to form a unique inventive arrangement as defined in the claims. Any feature or element of any embodiment may also be combined with features or elements from other inventive arrangements to form another unique inventive arrangement as defined in the claims. Thus, it should be understood that any of the features shown and/or discussed in this application may be implemented alone or in any suitable combination. Accordingly, the embodiments are not to be restricted except in light of the attached claims and their equivalents. Further, various modifications and changes may be made within the scope of the appended claims.
Furthermore, in describing representative embodiments, the specification may have presented the method and/or process as a particular sequence of steps. However, to the extent that the method or process does not rely on the particular order of steps set forth herein, the method or process should not be limited to the particular sequence of steps described. Other sequences of steps are possible as will be appreciated by those of ordinary skill in the art. Accordingly, the particular order of the steps set forth in the specification should not be construed as limitations on the claims. Furthermore, the claims directed to the method and/or process should not be limited to the performance of their steps in the order written, and one skilled in the art can readily appreciate that the sequences may be varied and still remain within the spirit and scope of the embodiments of the present application.
As shown in fig. 1, an embodiment of the present application provides a data processing method, which includes steps 101 to 103.
Step 101 comprises: metadata detection data is created for the metadata, the metadata is stored in the first memory, the metadata detection data is stored in the second memory, and physical memory spaces of the first memory and the second memory are isolated from each other.
It should be noted that data in any storage system is divided into real data (data) and metadata (metadata). The actual data refers to data actually stored by the user. Metadata refers to system data that describes characteristics of a file, block device, or object in a storage system, such as access rights, owner, and distribution information for the data blocks. Metadata is often referred to as "data about data," i.e., data that is used to describe other data. For example, a file of a user is referred to as "data", and in a storage system, metadata of the file is generally used to indicate the name, size, creation time, modification time, user to whom the file belongs, storage location, and so on.
The data processing method provided by the embodiment of the application can be applied to any type of storage server containing metadata, the storage server can provide data access service for a plurality of different applications, and clients of the different applications can send metadata access requests to the storage server through different types of service interfaces so as to acquire metadata. The storage server may be a data server, and at this time, metadata provided by the data server may be metadata included in the data server itself; the storage server may be a metadata server, in which case the metadata provided by the metadata server may be metadata contained by one or more other data servers. For a metadata access request, a plurality of metadata attribute items may be included, each having its attribute parameters, such as the name, size, etc. of the object to be accessed. For example, for a block interface, its object name to be accessed may be a volume ID, while for a file interface, its object name to be accessed may be an Inode ID, and for different types of service interfaces, the attribute items included in the metadata access request may be different.
In one exemplary embodiment, the first memory is a small page memory and the second memory is a large page memory.
The operating system manages the memory by using multi-level page tables and pages, and each page of the operating system defaults to 4 Kilobytes (KB), namely a small page memory or a common memory. Compared with the traditional small page memory or common memory, the Linux operating system supports a large page memory with more than 2 Megabytes (MB), which is called as a large page memory, and the memory management and access efficiency can be improved through the large page memory. The large page memory is different from the small page memory or the common memory, and is exported to the user process for management and use through the hugetlbfs special file system. hugetlbfs is a virtual file system where files in hugetlbfs do not support read/write system calls (e.g., read () or write (), etc.), and access to it is typically done in the form of a memory map mmap (memory map). mmap is a method of memory mapping files, i.e., mapping a file or other object to the virtual address space of a process. The memory mapped by hugetlbfs is hugepage large page memory.
As shown in fig. 2, the metadata is stored by using the small page memory, and the metadata detection data is stored by using the large page memory, so that the metadata and the metadata detection data are ensured to be isolated from each other in the physical memory space, and the mutual influence is avoided, and the correctness of the metadata can be effectively detected based on the metadata detection data.
In one exemplary embodiment, the metadata detection data is a metadata copy or metadata check value.
In this embodiment, the metadata copy is backup data of metadata, i.e., the metadata copy is identical to the metadata; the metadata check value is a check value calculated on the metadata by a check algorithm, for example, the check value may be a Message Digest 5 (md5) value, etc., and the specific check algorithm and its calculation process are well known to those skilled in the art, and will not be described herein. When metadata detection data uses metadata copies, the data processing process is simpler and more efficient; when the metadata detection data uses the metadata check value, the metadata detection data occupies less memory space.
In an exemplary embodiment, before creating the metadata detection data for the metadata, the data processing method further includes: the file system is divided into a plurality of metadata partitions, each of which is divided into a plurality of metadata blocks, each of which includes a preset number (e.g., 1024 or other number) of metadata units.
In one exemplary embodiment, creating metadata detection data for metadata includes: creating metadata detection data for metadata in units of metadata blocks; alternatively, metadata detection data is created for metadata in units of metadata units.
The architecture of the Linux operating system is divided into a user mode and a kernel mode, wherein the kernel mode is essentially software, controls hardware resources of a computer and provides an environment for running upper-layer application programs. The user mode is the active space of the upper application program, and the execution of the application program must depend on the resources provided by the kernel, including CPU resources, memory resources, I/O resources and the like. In order for an upper layer application to have access to these resources, the kernel must provide an interface for the upper layer application to access: i.e. a system call. The storage system of the application refers to a user state storage system, namely a storage system in which a storage software stack is completely realized in a user state.
In an exemplary embodiment, the method further comprises: a large page memory pool is created for each metadata partition, the metadata partition comprises a plurality of metadata, a large page memory pool is used for storing metadata detection data of the metadata partition, the large page memory pool comprises a plurality of large pages and a plurality of sections of virtual memory spaces, each large page is mapped to one section of virtual memory space, the sections of virtual memory spaces in the large page memory pool are continuous, and base addresses (namely first addresses of first sections of virtual memory spaces) of the virtual memory spaces in different large page memory pools are different.
In a storage system, data is classified into many types, such as metadata and user actual data, which all require memory for storage management. The security of memory management is critical for a storage system, and memory isolation between different types of data is guaranteed, so that the security of memory management can be effectively improved, the mutual influence of different types of data memories is avoided, and the data security of the storage system is enhanced. The traditional memory management method adopts a common memory pooling scheme to perform memory isolation, the method does not consider the spatial continuity of virtual addresses and physical addresses, memories among memory pools can be accessed beyond boundaries and trampled mutually, the real isolation among virtual memories and physical memories can not be realized, and serious memory problems such as mutual trampling of memories among different data types can still be caused.
It should be noted that, the physical memory refers to a real physical medium memory storage space, and the address describing the physical memory is a physical address. The virtual memory is ultimately mapped to a specific physical address for access. The memory accessed by the user process is a virtual memory, and the address describing the virtual memory is a virtual address.
According to the method and the device, the large page memory mmap of different physical addresses is mapped to different virtual memory spaces, so that isolation of physical memory and virtual memory among different data types is realized, and the security of the memory of the storage system is improved.
The overall structure of the large page memory pool is shown in fig. 3, the large page memory pools are memory isolated units, the physical memory of each large page memory pool is composed of a stack of physical large page memories, each physical large page memory is mapped to a section of virtual memory space with the same size, and the physical memories among the large page memory pools are isolated. The virtual memory of each large page memory pool is a section of continuous virtual address space, as shown in fig. 4, the base addresses of the continuous virtual memory spaces of different large page memory pools are different, so as to ensure the virtual memory isolation between the different large page memory pools.
In an exemplary embodiment, the method further comprises: and creating a memory pool mapping table for each large page memory pool, wherein the memory pool mapping table is used for describing the mapping relation between each large page and each segment of virtual memory space in the large page memory pool.
As shown in fig. 5, each large page memory pool has an independent memory mapping table, and the memory mapping table is used for converting virtual addresses into physical addresses, so that the address conversion efficiency is improved by creating a memory pool mapping table for each large page memory pool, and the mutual influence of the virtual memory address to the physical memory address conversion between the large page memory pools can be avoided, thereby realizing memory isolation in the process of memory conversion.
Step 102 comprises: and receiving a metadata access request of the client, and detecting whether the metadata is correct or not based on the metadata detection data.
In an exemplary embodiment, when the metadata detection data is a metadata copy, detecting whether the metadata is correct based on the metadata detection data includes:
detecting whether the metadata is consistent with the metadata copy;
when the metadata is consistent with the metadata copy, judging that the metadata is correct;
when the metadata and the metadata copy are inconsistent, the metadata is judged to be wrong.
In an exemplary embodiment, when the metadata detection data is a metadata check value, detecting whether the metadata is correct based on the metadata detection data includes:
calculating a check value of the metadata, and detecting whether the calculated check value is consistent with the check value of the metadata;
when the calculated check value is consistent with the metadata check value, judging that the metadata is correct;
and when the calculated check value is inconsistent with the metadata check value, judging that the metadata is wrong.
In one exemplary embodiment, the metadata access request may be a metadata read request or a metadata modification request.
The correctness of the metadata is related to the data safety of the storage system, the embodiment of the application can discover the abnormality of the metadata in time by adding a verification mechanism for the metadata, prevent the problem of the abnormal data from being spread, ensure the data correctness during the metadata access, and in addition, the verification process of the embodiment of the application processes the memory data, thereby improving the verification efficiency of the metadata.
Step 103 comprises: when the metadata is correct, the metadata access request of the client is processed based on the metadata.
In an exemplary embodiment, the metadata access request may include access information, where the access information may include a user name or account, and when the user name or account meets an access right, the metadata access request indicates that the metadata access request meets the access right, and the user name or account is allowed to access metadata associated with the access right; and rejecting the metadata access request when the user name or the account number does not accord with the access right.
In an exemplary embodiment, as shown in fig. 1, the data processing method further includes:
step 104: when the metadata is wrong, the metadata is wrong and returned to the client.
When metadata access is performed, the data correctness of the metadata is checked through metadata detection, and when the metadata correctness is detected based on the metadata detection data, follow-up operation is performed, so that the data safety is ensured; when the metadata is detected to be incorrect based on the metadata detection data, the metadata error is returned, and the user is timely notified, so that the dirty data is prevented from being further spread.
Metadata servers can be generally divided into two categories depending on system organization: the unit data server is used for providing metadata service for the whole system, metadata access requests of all clients are sent to the metadata server and executed, and when metadata errors occur, the unit data server can further read metadata in a disk and generate memory structure metadata corresponding to the disk structure; another type is a metadata server, in which metadata access requests are divided into one of the metadata servers for processing, and when metadata errors occur, the metadata access requests may be re-divided into other metadata servers for processing.
In an exemplary embodiment, as shown in fig. 6, the data processing method further includes: a metadata update request is received, and metadata detection data are synchronously updated.
In the embodiment of the application, when the metadata needs to be updated, the metadata and the metadata detection data are updated synchronously, so that the consistency of the metadata and the metadata detection data is ensured.
Based on the same inventive concept, an embodiment of the present application further provides a data processing apparatus, including a processor and a memory, the processor being configured to execute a computer program stored in the memory to implement the steps of the data processing method according to any one of the above.
Based on the same inventive concept, the embodiments of the present application also provide a computer-readable storage medium storing a computer program which, when executed by a processor, implements the steps of the data processing method as described in any one of the above.
Based on the same inventive concept, as shown in fig. 7, the embodiment of the present application further provides a data processing apparatus, including a first memory 701, a second memory 702, a metadata detection module 703, and a metadata processing module 704, where:
a metadata detection module 703, configured to create metadata detection data for metadata, where the metadata is stored in the first memory 701, the metadata detection data is stored in the second memory 702, and physical memory spaces of the first memory 701 and the second memory 702 are isolated from each other; receiving a metadata access request of a client, and detecting whether metadata is correct or not based on metadata detection data; when the metadata is correct, notifying a metadata processing module;
and the metadata processing module 704 is configured to receive the notification from the metadata detection module, and process the metadata access request of the client based on the metadata.
In one exemplary embodiment, the first memory 701 is a small page memory and the second memory 702 is a large page memory.
In one exemplary embodiment, the metadata detection data is a metadata copy or metadata check value.
In this embodiment, the metadata copy is backup data of metadata, i.e., the metadata copy is identical to the metadata; the metadata check value is a check value calculated on the metadata by a check algorithm, for example, the check value may be an MD5 value, etc., and the specific check algorithm and its calculation process are well known to those skilled in the art, and will not be described herein. When metadata detection data uses metadata copies, the data processing process is simpler and more efficient; when the metadata detection data uses the metadata check value, the metadata detection data occupies less memory space.
In an exemplary embodiment, the metadata detection module 703 is further configured to: the file system is divided into a plurality of metadata partitions, each of which is divided into a plurality of metadata blocks, each of which includes a preset number (e.g., 1024 or other number) of metadata units.
In one exemplary embodiment, the metadata detection module 703 creates metadata detection data for metadata in units of metadata blocks; alternatively, metadata detection data is created for metadata in units of metadata units.
In an exemplary embodiment, the metadata detection module 703 is further configured to: a large page memory pool is created for each metadata partition, the large page memory pool is used for storing metadata detection data of a plurality of metadata of one metadata partition, the large page memory pool comprises a plurality of large pages and a plurality of sections of virtual memory spaces, each large page is mapped to one section of virtual memory space, the sections of virtual memory spaces in the large page memory pool are continuous, and base addresses (namely first addresses of the first section of virtual memory spaces) of the virtual memory spaces in different large page memory pools are different.
In an exemplary embodiment, the metadata detection module 703 is further configured to: and creating a memory pool mapping table for each large page memory pool, wherein the memory pool mapping table is used for describing the mapping relation between each large page and each segment of virtual memory space in the large page memory pool.
The data processing device adopts the large page memory pools as basic units for memory isolation, and the physical memory of each large page memory pool consists of a stack of different physical large page memories, so that the physical memory isolation among the large page memory pools is realized. The virtual memory of each large page memory pool is a section of continuous virtual address space, and the continuous virtual address space of each large page memory pool has independent base addresses, so that the virtual memory isolation among the large page memory pools is realized. And meanwhile, a memory mapping table for converting virtual addresses into physical addresses is created for each large page memory pool, so that memory isolation in the memory conversion process is realized.
The data processing device adds the consideration of the continuity of the virtual address space and the physical address space, effectively avoids the problem of memory stepping caused by memory continuity and beyond access, truly realizes isolation among memory pools, virtual memories and physical memories, avoids serious memory problems, and improves the security of the memory of the storage system.
In an exemplary embodiment, when the metadata detection data is a metadata copy, the metadata detection module 703 detects whether the metadata is correct based on the metadata detection data, including:
detecting whether the metadata is consistent with the metadata copy; when the metadata is consistent with the metadata copy, judging that the metadata is correct; when the metadata and the metadata copy are inconsistent, the metadata is judged to be wrong.
In an exemplary embodiment, when the metadata detection data is a metadata check value, the metadata detection module 703 detects whether the metadata is correct based on the metadata detection data, including:
calculating a check value of the metadata, and detecting whether the calculated check value is consistent with the check value of the metadata; when the calculated check value is consistent with the metadata check value, judging that the metadata is correct; and when the calculated check value is inconsistent with the metadata check value, judging that the metadata is wrong.
In one exemplary embodiment, the metadata access request may be a metadata read request or a metadata modification request.
In an exemplary embodiment, the metadata access request may include access information, where the access information may include a user name or an account number, and the metadata processing module 704 is specifically configured to: detecting whether a user name or an account number in the metadata access request accords with the access right, allowing the user name or the account number to access metadata associated with the access right when the user name or the account number accords with the access right, and rejecting the metadata access request when the user name or the account number does not accord with the access right.
In an exemplary embodiment, the metadata detection module 703 is further configured to: when the metadata is wrong, the metadata is wrong and returned to the client.
When the data processing device processes the metadata access request, the metadata detection module 703 detects the correctness of the metadata based on the metadata, and when the metadata is detected to be correct based on the metadata detection data, the metadata processing module 704 is notified to perform subsequent operations, so that the data security is ensured; when the metadata is detected to be incorrect based on the metadata detection data, the metadata error is returned, and the user is timely notified, so that the dirty data is prevented from being further spread.
In an exemplary embodiment, as shown in fig. 8, the data processing apparatus further includes a metadata update module 705, where the metadata update module 705 is configured to receive a metadata update request, and update metadata and metadata detection data synchronously.
In this embodiment, when the metadata needs to be updated, the metadata updating module 705 synchronously updates the metadata and the metadata detection data, so as to ensure that the metadata and the metadata detection data are consistent.
Those of ordinary skill in the art will appreciate that all or some of the steps, systems, functional modules/units in the apparatus, and methods disclosed above may be implemented as software, firmware, hardware, and suitable combinations thereof. In a hardware implementation, the division between the functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be performed cooperatively by several physical components. Some or all of the components may be implemented as software executed by a processor, such as a digital signal processor or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). The term computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, as known to those skilled in the art. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital Versatile Disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. Furthermore, as is well known to those of ordinary skill in the art, communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.
Claims (7)
1. A method of data processing, comprising:
creating metadata detection data for metadata, wherein the metadata is stored in a first memory, the metadata detection data is stored in a second memory, and physical memory spaces of the first memory and the second memory are mutually isolated, wherein the first memory is a small page memory, the second memory is a large page memory, and the metadata detection data is a metadata copy or a metadata check value;
receiving a metadata access request of a client, and detecting whether the metadata is correct or not based on the metadata detection data;
when the metadata is correct, processing a metadata access request of the client based on the metadata;
wherein the method further comprises: receiving a metadata update request, and synchronously updating the metadata and metadata detection data; creating a large page memory pool for each metadata partition, wherein the metadata partition comprises a plurality of metadata, one large page memory pool is used for storing metadata detection data of a plurality of metadata of one metadata partition, the large page memory pool comprises a plurality of large pages and a plurality of virtual memory spaces, each large page is mapped to one section of virtual memory space, the plurality of virtual memory spaces in one large page memory pool are continuous, and the base addresses of the virtual memory spaces in different large page memory pools are different; and creating a memory pool mapping table for each large page memory pool, wherein the memory pool mapping table is used for describing the mapping relation between each large page and each segment of virtual memory space in the large page memory pool.
2. The data processing method of claim 1, wherein the method further comprises: and when the metadata is wrong, returning the metadata to the client.
3. The data processing method according to claim 1, wherein the metadata detection data is a metadata copy;
the detecting whether the metadata is correct based on the metadata detection data includes:
detecting whether the metadata and the metadata copy are consistent;
when the metadata is consistent with the metadata copy, judging that the metadata is correct;
and when the metadata and the metadata copy are inconsistent, judging that the metadata is wrong.
4. The data processing method according to claim 1, wherein the metadata detection data is a metadata check value;
the detecting whether the metadata is correct based on the metadata detection data includes:
calculating a check value of the metadata, and detecting whether the calculated check value is consistent with the check value of the metadata;
when the calculated check value is consistent with the metadata check value, judging that the metadata is correct;
and judging that the metadata is wrong when the calculated check value is inconsistent with the metadata check value.
5. A data processing apparatus comprising a processor and a memory, the processor being arranged to execute a computer program stored in the memory to carry out the steps of the data processing method according to any one of claims 1 to 4.
6. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the data processing method according to any one of claims 1 to 4.
7. The data processing device is characterized by comprising a first memory, a second memory, a metadata detection module, a metadata processing module and a metadata updating module, wherein:
the metadata detection module is configured to create metadata detection data for metadata, where the metadata is stored in the first memory, the metadata detection data is stored in the second memory, and physical memory spaces of the first memory and the second memory are isolated from each other, where the first memory is a small page memory, the second memory is a large page memory, and the metadata detection data is a metadata copy or a metadata check value; receiving a metadata access request of a client, and detecting whether the metadata is correct or not based on the metadata detection data; notifying the metadata processing module when the metadata is correct; creating a large page memory pool for each metadata partition, wherein the metadata partition comprises a plurality of metadata, one large page memory pool is used for storing metadata detection data of a plurality of metadata of one metadata partition, the large page memory pool comprises a plurality of large pages and a plurality of virtual memory spaces, each large page is mapped to one section of virtual memory space, the plurality of virtual memory spaces in one large page memory pool are continuous, and the base addresses of the virtual memory spaces in different large page memory pools are different; creating a memory pool mapping table for each large page memory pool, wherein the memory pool mapping table is used for describing the mapping relation between each large page and each segment of virtual memory space in the large page memory pool;
the metadata processing module is used for receiving the notification of the metadata detection module and processing the metadata access request of the client based on the metadata;
the metadata updating module is used for receiving a metadata updating request and synchronously updating the metadata and the metadata detection data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010472738.3A CN113296685B (en) | 2020-05-29 | 2020-05-29 | Data processing method and device and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010472738.3A CN113296685B (en) | 2020-05-29 | 2020-05-29 | Data processing method and device and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113296685A CN113296685A (en) | 2021-08-24 |
| CN113296685B true CN113296685B (en) | 2023-12-26 |
Family
ID=77318050
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010472738.3A Active CN113296685B (en) | 2020-05-29 | 2020-05-29 | Data processing method and device and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113296685B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114461388B (en) * | 2022-01-13 | 2024-01-12 | 苏州浪潮智能科技有限公司 | Block storage service metadata configuration method, device, equipment and medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1622060A (en) * | 2003-11-26 | 2005-06-01 | 微软公司 | Lazy flushing of translation lookaside buffers |
| CN102486751A (en) * | 2010-12-01 | 2012-06-06 | 安凯(广州)微电子技术有限公司 | Method for realizing virtual big page through small page NANDFLASH on micro memory system |
| CN107807792A (en) * | 2017-10-27 | 2018-03-16 | 郑州云海信息技术有限公司 | A kind of data processing method and relevant apparatus based on copy storage system |
| CN108804350A (en) * | 2017-04-27 | 2018-11-13 | 华为技术有限公司 | A kind of memory pool access method and computer system |
| CN110825492A (en) * | 2019-10-31 | 2020-02-21 | 海光信息技术有限公司 | Method, device, equipment and storage medium for isolating secure data memory |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7475183B2 (en) * | 2005-12-12 | 2009-01-06 | Microsoft Corporation | Large page optimizations in a virtual machine environment |
| US9323689B2 (en) * | 2010-04-30 | 2016-04-26 | Netapp, Inc. | I/O bandwidth reduction using storage-level common page information |
| US10552344B2 (en) * | 2017-12-26 | 2020-02-04 | Intel Corporation | Unblock instruction to reverse page block during paging |
-
2020
- 2020-05-29 CN CN202010472738.3A patent/CN113296685B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1622060A (en) * | 2003-11-26 | 2005-06-01 | 微软公司 | Lazy flushing of translation lookaside buffers |
| CN102486751A (en) * | 2010-12-01 | 2012-06-06 | 安凯(广州)微电子技术有限公司 | Method for realizing virtual big page through small page NANDFLASH on micro memory system |
| CN108804350A (en) * | 2017-04-27 | 2018-11-13 | 华为技术有限公司 | A kind of memory pool access method and computer system |
| CN107807792A (en) * | 2017-10-27 | 2018-03-16 | 郑州云海信息技术有限公司 | A kind of data processing method and relevant apparatus based on copy storage system |
| CN110825492A (en) * | 2019-10-31 | 2020-02-21 | 海光信息技术有限公司 | Method, device, equipment and storage medium for isolating secure data memory |
Non-Patent Citations (4)
| Title |
|---|
| Xu, J 等.NOVA: A Log-structured File System for Hybrid Volatile/Non-volatile Main Memories.14TH USENIX CONFERENCE ON FILE AND STORAGE TECHNOLOGIES (FAST '16).2016,全文. * |
| 一种基于副本机制的智能存储系统I/O优化方法;刘霖;赵跃龙;谢晓玲;王国华;;沈阳工业大学学报(第04期);全文 * |
| 一种支持大页的层次化DRAM/NVM混合内存系统;陈吉;刘海坤;王孝远;张宇;廖小飞;金海;;计算机研究与发展(第09期);全文 * |
| 新型非易失存储的安全与隐私问题研究综述;徐远超;闫俊峰;万虎;孙凤芸;张伟功;李涛;;计算机研究与发展(第09期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113296685A (en) | 2021-08-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3502877B1 (en) | Data loading method and apparatus for virtual machines | |
| US9015417B2 (en) | Deduplication-aware page cache | |
| CN110096227B (en) | Data storage method, data processing device, electronic equipment and computer readable medium | |
| US10157214B1 (en) | Process for data migration between document stores | |
| US10356150B1 (en) | Automated repartitioning of streaming data | |
| CN102938784A (en) | Method and system used for data storage and used in distributed storage system | |
| CN108733311B (en) | Method and apparatus for managing storage system | |
| US8195877B2 (en) | Changing the redundancy protection for data associated with a file | |
| CN113448938A (en) | Data processing method and device, electronic equipment and storage medium | |
| US20230273859A1 (en) | Storage system spanning multiple failure domains | |
| CN111309245B (en) | Hierarchical storage writing method and device, reading method and device and system | |
| US20150169623A1 (en) | Distributed File System, File Access Method and Client Device | |
| CN113296685B (en) | Data processing method and device and computer readable storage medium | |
| US20180260155A1 (en) | System and method for transporting a data container | |
| CN114968127A (en) | Object calculation storage system, data processing method, client and storage medium | |
| CN109388651B (en) | Data processing method and device | |
| CN107943615B (en) | Data processing method and system based on distributed cluster | |
| US20120151005A1 (en) | Image file download method | |
| US11222036B1 (en) | Data warehouse access reporting | |
| CN109542860B (en) | Service data management method based on HDFS and terminal equipment | |
| CN108228842B (en) | Docker mirror image library file storage method, terminal, device and storage medium | |
| CN105068896A (en) | Data processing method and device based on RAID backup | |
| US11580078B2 (en) | Providing enhanced security for object access in object-based datastores | |
| CN112800123B (en) | Data processing method, device, computer equipment and storage medium | |
| CN112486517B (en) | Application program installation method and system, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |