CN117155759B

CN117155759B - Data processing method, device, computer equipment and storage medium

Info

Publication number: CN117155759B
Application number: CN202311403986.2A
Authority: CN
Inventors: 葛凯凯; 罗维; 张智
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2023-10-27
Filing date: 2023-10-27
Publication date: 2024-01-05
Anticipated expiration: 2043-10-27
Also published as: CN117155759A

Abstract

The embodiment of the application discloses a data processing method, a data processing device, computer equipment and a storage medium, and belongs to the technical field of computers. The embodiment of the application can be applied to various scenes such as cloud storage, cloud computing, big data and the like. The method comprises the following steps: receiving a metadata migration instruction; identifying metadata of the file system layer by layer in the first metadata pool based on the tree structure to obtain metadata of files of each layer in the file system and metadata of folders of each layer; and migrating the metadata of each layer of files and the metadata of each layer of folders in the file system to a second metadata pool layer by layer. According to the scheme, when the storage capacity of the first metadata pool is insufficient, only metadata of the file system is migrated to the second metadata storage pool, data of the file system does not need to be migrated, normal storage of the metadata is guaranteed, the migrated data size can be obviously reduced, and metadata migration efficiency is improved.

Description

Data processing method, device, computer equipment and storage medium

Technical Field

The embodiment of the application relates to the technical field of computers, in particular to a data processing method, a data processing device, computer equipment and a storage medium.

Background

With the development of big data and cloud computing, distributed storage is becoming a mainstream data storage mode. In a distributed storage system, data is stored in a data pool of a storage cluster and metadata is stored in a metadata pool of the storage cluster. The data is a file or a folder, and the metadata is description data such as the name, the size, the creation time and the like of the file or the folder. Under the condition that the storage capacity of the metadata pool is insufficient, how to ensure the normal storage of the metadata is a technical problem to be solved.

In the related art, when the storage capacity of the metadata pool is insufficient, data and metadata in the storage cluster can only be migrated to the temporary cluster. Then, a larger metadata pool is created in the storage cluster. And then, respectively migrating the data and the metadata stored in the temporary cluster to a data pool in the storage cluster and a newly created metadata pool, thereby realizing cross-pool migration of the metadata in the storage cluster.

However, the storage amount of data is far greater than that of metadata, and by adopting the method, a great amount of time is consumed in the process of migrating the data, so that the efficiency of cross-pool migration of the metadata is too low.

Disclosure of Invention

The embodiment of the application provides a data processing method, a data processing device, computer equipment and a storage medium, which can only transfer metadata of a file system to a second metadata pool with larger storage capacity layer by layer when the storage capacity of a first metadata pool is insufficient, can obviously reduce the transferred data amount and improve the efficiency of metadata transfer. The technical scheme is as follows:

in one aspect, a data processing method is provided, the method including:

receiving a metadata migration instruction, wherein the metadata migration instruction is used for indicating metadata stored in a first metadata pool to be migrated to a second metadata pool, the first metadata pool is used for storing metadata of a file system according to a tree structure, the metadata is used for describing corresponding files or folders, the tree structure is used for representing the levels of the files and folders in the file system, and the levels of the files or folders in the same layer in the tree structure are the same;

identifying metadata of the file system layer by layer in the first metadata pool based on the tree structure to obtain metadata of files of each layer in the file system and metadata of folders of each layer;

And migrating the metadata of the files of each layer and the metadata of the folders of each layer in the file system to the second metadata pool layer by layer, wherein the storage capacity of the second metadata pool is larger than that of the first metadata pool.

In another aspect, there is provided a data processing apparatus, the apparatus comprising:

the system comprises a receiving module, a storage module and a storage module, wherein the receiving module is used for receiving a metadata migration instruction, the metadata migration instruction is used for indicating metadata stored in a first metadata pool to be migrated to a second metadata pool, the first metadata pool is used for storing metadata of a file system according to a tree structure, the metadata is used for describing corresponding files or folders, the tree structure is used for representing the levels of the files and folders in the file system, and the levels of the files or folders in the same layer in the tree structure are the same;

the identification module is used for identifying the metadata of the file system layer by layer in the first metadata pool based on the tree structure to obtain the metadata of each layer of files in the file system and the metadata of each layer of folders;

and the migration module is used for migrating the metadata of each layer of files and the metadata of each layer of folders in the file system to the second metadata pool layer by layer, and the storage capacity of the second metadata pool is larger than that of the first metadata pool.

In some embodiments, the identifying module is configured to identify, based on the tree structure, a node object of a root node in the first metadata pool, where the root node is located at a first level of the tree structure, the node object of the root node stores metadata of at least one first level node, where the first level node is a child node of the root node, where the first level node is located at a second level of the tree structure, where the root node is used to represent a folder, and where the first level node is used to represent a file or a folder; for any ith level node, in the case that the ith level node is used for representing a folder, identifying a node object of the ith level node in the first metadata pool, wherein metadata of at least one (i+1) th level node is stored in the node object of the ith level node, the (i+1) th level node is a child node of the ith level node, the (i+1) th level node is located at the (i+2) th level of the tree structure, and i is an integer not less than 1; and under the condition that the tree structure has n levels, responding to the node object of the n-1 level node, and ending the layer-by-layer identification process.

In some embodiments, the identifying module is configured to determine, in the first metadata pool, a root node, an index number of the root node, and at least one fragment number of the root node, based on the tree structure, the fragment number being configured to identify a node object of the root node; and identifying at least one node object of the root node in the first metadata pool based on the index number of the root node and at least one fragment number of the root node, wherein the name of the at least one node object corresponds to the at least one fragment number one by one.

In some embodiments, the apparatus further comprises:

the first obtaining module is configured to obtain, in response to obtaining a node object of a multi-level node in the tree structure, for a node object of any node, metadata of at least one child node of the node based on at least one key value pair stored in the node object, where the at least one key value pair is used to store metadata of the at least one child node, a key in the key value pair is a name of a corresponding child node, and a value in the key value pair is metadata of the corresponding child node.

In some embodiments, for any child node, in the case that the child node is used for representing a folder, an index number and a fragment number of the child node are also stored in a key value pair corresponding to the child node;

the identification module is used for acquiring, for any ith node, an index number and at least one fragment number stored in a key value pair corresponding to the ith node when the ith node is used for representing a folder; at least one node object of the ith level node is identified in the first metadata pool based on the index number and at least one fragment number.

In some embodiments, the apparatus further comprises:

and the second acquisition module is used for acquiring system metadata in the first metadata pool, wherein the system metadata comprises metadata of the root node and metadata of metadata service, and the metadata service is used for maintaining a tree structure in the first metadata pool.

In some embodiments, the migration module is configured to migrate the system metadata to the second metadata pool; migrating node objects of a plurality of nodes in the tree structure to the second metadata pool according to the order of the levels of the nodes from high to low, wherein the level of the root node is highest, and the level of the nth level node is lowest; and ending the layer-by-layer migration process in response to successful migration of node objects of the plurality of n-1 level nodes to the second metadata pool.

In some embodiments, the migration module is configured to, for a node object of any node, update a plurality of key-value pairs in the node object; acquiring statistical information of the node in the node object, wherein the statistical information is used for representing the number of child nodes contained in the node; updating metadata distribution information and father node distribution information in the node object, wherein the metadata distribution information is used for identifying a metadata pool for storing the node object, and the father node distribution information is used for identifying a metadata pool for storing the metadata of the node; and migrating the updated plurality of key value pairs, the statistical information, the updated metadata distribution information and the updated parent node distribution information into the second metadata pool.

In some embodiments, the migration module is configured to, for a node object of any node, read a plurality of key-value pairs in the node object; for any key value pair, performing inverse sequence analysis on the values in the key value pair to obtain analyzed metadata; the read-write attribute of the metadata is changed, so that the metadata is forbidden to be read and written; and serializing the changed metadata to obtain the value in the updated key value pair.

In some embodiments, for any target object, the target object is named an index number and a fragment number of a corresponding node, the apparatus further comprises:

the construction module is used for constructing a first set based on the names of a plurality of target objects in the layer-by-layer migration process, wherein the first set is a set of used index numbers;

the screening module is used for screening a second set from all index numbers based on the first set, wherein the second set is a set of unused index numbers;

a storage module for storing the second set in the second metadata pool;

an allocation module, configured to allocate an index number to a node from the second set in response to an operation of creating the node in the second metadata pool.

In some embodiments, the apparatus further comprises:

and the creation module is used for responding to the metadata migration instruction and creating the second metadata pool in a storage cluster, wherein the storage cluster comprises the first metadata pool and at least one data pool, and the data pool is used for storing files and folders of the file system.

In some embodiments, the apparatus further comprises:

an updating module, configured to update, in response to successful migration of metadata of the file system to the second metadata pool, the metadata pool associated with the file system from the first metadata pool to the second metadata pool in a file system view, where the file system view is configured to store an association relationship between the file system and the metadata pool;

and the synchronization module is used for synchronizing the updated file system view to a plurality of clients so that the clients can read and write the metadata of the file system in the metadata pool associated with the file system according to the updated file system view.

In another aspect, a computer device is provided, the computer device comprising a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to implement the data processing method as described in the above aspect.

In another aspect, there is provided a computer readable storage medium having stored therein at least one computer program loaded and executed by a processor to implement a data processing method as described in the above aspect.

In another aspect, a computer program product is provided, comprising a computer program, which is loaded and executed by a processor to implement the data processing method as described in the above aspect.

The embodiment of the application provides a data processing scheme. Since the first metadata pool stores metadata according to a tree structure, the hierarchical structure of files and folders in the file system can be intuitively determined through the tree structure. Therefore, when the storage capacity of the first metadata pool is insufficient, the computer equipment can only migrate the metadata of the file system to the second metadata pool with larger storage capacity layer by layer according to the tree structure, and then the metadata can be stored in the new metadata pool, so that the normal storage of the metadata is ensured. In addition, the method can obviously reduce the data quantity of migration and improve the efficiency of metadata migration because the method does not need to migrate the data of the file system.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic diagram of an implementation environment of a data processing method according to an embodiment of the present application;

FIG. 2 is a flow chart of a data processing method according to an embodiment of the present application;

FIG. 3 is a flow chart of another data processing method provided in an embodiment of the present application;

FIG. 4 is a schematic diagram of metadata provided by an embodiment of the present application;

FIG. 5 is a schematic diagram of a tree structure according to an embodiment of the present application;

FIG. 6 is a schematic diagram of identifying metadata layer by layer provided in an embodiment of the present application;

FIG. 7 is a schematic diagram of migration system metadata provided by an embodiment of the present application;

FIG. 8 is a schematic diagram of updating distribution information according to an embodiment of the present application;

FIG. 9 is a schematic diagram of a migration object node according to an embodiment of the present application;

FIG. 10 is a schematic diagram of an index number provided in an embodiment of the present application;

FIG. 11 is a schematic diagram of a synchronous file system view provided in an embodiment of the present application;

FIG. 12 is a schematic diagram of reading and writing metadata according to an embodiment of the present application;

FIG. 13 is a schematic diagram of a data processing apparatus according to an embodiment of the present disclosure;

FIG. 14 is a schematic diagram of another data processing apparatus according to an embodiment of the present application;

fig. 15 is a schematic structural diagram of a terminal according to an embodiment of the present application;

fig. 16 is a schematic structural diagram of a server according to an embodiment of the present application.

Detailed Description

For the purposes of making the objects, technical solutions and advantages of the embodiments of the present application more apparent, the embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

The terms "first," "second," and the like in this application are used to distinguish between identical or similar items that have substantially the same function and function, and it should be understood that there is no logical or chronological dependency between the "first," "second," and "nth" terms, nor is it limited to the number or order of execution.

The term "at least one" in this application means one or more, and the meaning of "a plurality of" means two or more.

It should be noted that, information (including but not limited to user equipment information, user personal information, etc.), data (including but not limited to data for analysis, stored data, presented data, etc.), and signals referred to in this application are all authorized by the user or are fully authorized by the parties, and the collection, use, and processing of relevant data is required to comply with relevant laws and regulations and standards of relevant countries and regions. For example, documents and folders referred to in this application are obtained with sufficient authorization.

In order to facilitate understanding, terms related to the present application are explained below.

Cloud computing (clouding) is a computing model that distributes computing tasks across a large pool of computers, enabling various application systems to acquire computing power, storage space, and information services as needed. The network that provides the resources is referred to as the "cloud". Resources in the cloud are infinitely expandable in the sense of users, and can be acquired at any time, used as needed, expanded at any time and paid for use as needed.

As a basic capability provider of cloud computing, a cloud computing resource pool (abbreviated as a cloud platform, generally called IaaS (Infrastructure as a Service, infrastructure as a service) platform) is established, and multiple types of virtual resources are deployed in the resource pool for external clients to select for use. The cloud computing resource pool mainly comprises: computing devices (which are virtualized machines, including operating systems), storage devices, network devices.

According to the logic function division, a PaaS (Platform as a Service ) layer can be deployed on the IaS layer, a SaaS (Software as a Service ) layer can be deployed on the PaaS layer, and the SaaS can also be directly deployed on the IaS. PaaS is a platform on which software runs, such as a database, web container, etc. SaaS is a wide variety of business software such as web portals, sms mass senders, etc. Generally, saaS and PaaS are upper layers relative to IaaS.

Cloud storage (cloud storage) is a new concept that extends and develops in the concept of cloud computing, and a distributed cloud storage system (hereinafter referred to as a storage system for short) refers to a storage system that integrates a large number of storage devices (storage devices are also referred to as storage nodes) of various types in a network to work cooperatively through application software or application interfaces through functions such as cluster application, grid technology, and a distributed storage file system, so as to provide data storage and service access functions for the outside.

At present, the storage method of the storage system is as follows: when creating logical volumes, each logical volume is allocated a physical storage space, which may be a disk composition of a certain storage device or of several storage devices. The client stores data on a certain logical volume, that is, the data is stored on a file system, the file system divides the data into a plurality of parts, each part is an object, the object not only contains the data but also contains additional information such as a data Identification (ID) and the like, the file system writes each object into a physical storage space of the logical volume, and the file system records storage position information of each object, so that when the client requests to access the data, the file system can enable the client to access the data according to the storage position information of each object.

The process of allocating physical storage space for the logical volume by the storage system specifically includes: physical storage space is divided into stripes in advance according to the set of capacity measures for objects stored on a logical volume (which measures tend to have a large margin with respect to the capacity of the object actually to be stored) and redundant array of independent disks (RAID, redundant Array of Independent Disk), and a logical volume can be understood as a stripe, whereby physical storage space is allocated for the logical volume.

Big data (Big data) refers to a data set which cannot be captured, managed and processed by a conventional software tool within a certain time range, and is a massive, high-growth-rate and diversified information asset which needs a new processing mode to have stronger decision-making ability, insight discovery ability and flow optimization ability. With the advent of the cloud age, big data has attracted more and more attention, and special techniques are required for big data to effectively process a large amount of data within a tolerant elapsed time. Technologies applicable to big data include massively parallel processing databases, data mining, distributed file systems, distributed databases, cloud computing platforms, the internet, and scalable storage systems.

The data processing method provided by the embodiment of the application can be executed by computer equipment. In some embodiments, the computer device is a terminal or a server. An implementation environment of the data processing method provided in the embodiment of the present application is described below by taking a computer device as an example of a server. Fig. 1 is a schematic diagram of an implementation environment of a data processing method according to an embodiment of the present application, with reference to fig. 1, where the implementation environment includes: a terminal 101 and a server 102. The terminal 101 and the server 102 can be directly or indirectly connected through wired or wireless communication, and the present application is not limited herein.

In some embodiments, terminal 101 is, but is not limited to, a smart phone, tablet, notebook, desktop, smart voice interaction device, smart home appliance, in-vehicle terminal, aircraft, etc. The terminal 101 installs and runs an application program supporting data storage. The user logs in to the application through the terminal 101 and stores user data using a data storage service of the application. The application is associated with the server 102 and background services are provided by the server 102.

In some embodiments, the server 102 is a stand-alone physical server, can be a server cluster or a distributed system formed by a plurality of physical servers, and can also be a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs (Content Delivery Network, content delivery networks), and basic cloud computing services such as big data and artificial intelligence platforms. In some embodiments, server 102 receives stored data uploaded by terminal 101 through an application. The server 102 stores the storage data uploaded by the plurality of terminals 101 through a distributed storage system. The server 102 may also provide data read-write services to the plurality of terminals 101 through a distributed storage system. In some embodiments, the server 102 takes on primary computing work and the terminal 101 takes on secondary computing work; alternatively, the server 102 takes on secondary computing work and the terminal 101 takes on primary computing work; alternatively, a distributed computing architecture is used for collaborative computing between the server 102 and the terminal 101.

Those skilled in the art will recognize that the number of terminals may be greater or lesser. Such as the above-mentioned terminals may be only one, or the above-mentioned terminals may be several tens or hundreds, or more. The number of terminals and the device type are not limited in the embodiment of the present application.

Fig. 2 is a flowchart of a data processing method according to an embodiment of the present application, and as shown in fig. 2, in the embodiment of the present application, an example of execution by a server is described. The data processing method includes the following steps.

201. The server receives a metadata migration instruction, wherein the metadata migration instruction is used for indicating metadata stored in a first metadata pool to be migrated to a second metadata pool, the first metadata pool is used for storing metadata of a file system according to a tree structure, the metadata is used for describing corresponding files or folders, the tree structure is used for representing the levels of the files and folders in the file system, and the levels of the files or folders in the same layer in the tree structure are the same.

In the embodiment of the application, the server stores the storage data uploaded by the plurality of terminals through the distributed storage system. The stored data may be divided into data and metadata. The data is data such as a file and a folder, and the metadata is data for describing the data such as the name, the size, the creation time and the like of the file or the folder. In a distributed storage system, data and metadata are stored in pools, i.e., the data and metadata are stored separately using respective resource pools. The resource pool storing data is called a data pool, and the resource pool storing metadata is called a metadata pool.

When the storage capacity of the old metadata pool is insufficient, in order to ensure normal storage of metadata, the distributed storage system may send a metadata migration instruction to the server, instructing the server to migrate metadata of the file system to a new metadata pool with larger storage capacity. For convenience of description, the old metadata pool will be referred to as a first metadata pool and the new metadata pool will be referred to as a second metadata pool. The file system comprises data and metadata uploaded by at least one terminal. The server may store data belonging to the same file system in the same data pool; metadata belonging to the same file system is stored in the same metadata pool, thereby facilitating management of stored data.

202. The server identifies metadata of the file system layer by layer in the first metadata pool based on the tree structure, and obtains metadata of files of each layer in the file system and metadata of folders of each layer.

In the embodiment of the application, in the first metadata pool, metadata of a file system is stored according to a tree structure. The tree structure of the storage metadata is the same as the directory structure of the storage data. Each level of nodes in the tree structure represents each level of files or each level of folders in the directory structure, respectively. And the server identifies the metadata of each node layer by layer in a first metadata pool of metadata to be migrated according to the tree structure of the stored metadata, and obtains the metadata of each layer of files and the metadata of each layer of folders in the file system.

203. The server migrates metadata of files of each layer and metadata of folders of each layer in the file system layer by layer to a second metadata pool, wherein the storage capacity of the second metadata pool is larger than that of the first metadata pool.

In the embodiment of the application, after obtaining metadata of files and folders of each layer in a file system, the server migrates the metadata layer by layer from a first layer (top layer) to a metadata pool with larger storage capacity. After the migration is completed, the server stores the metadata of the file system through the second metadata pool, and further, the terminal can read and write the metadata of the file system in the second metadata pool, so that the condition that the metadata is failed to read and write due to insufficient storage capacity of the first metadata pool is avoided, and normal reading and writing of the metadata are ensured.

The embodiment of the application provides a data processing method. Since the first metadata pool stores metadata according to a tree structure, the hierarchical structure of files and folders in the file system can be intuitively determined through the tree structure. Therefore, when the storage capacity of the first metadata pool is insufficient, the terminal can only migrate the metadata of the file system to the second metadata pool with larger storage capacity layer by layer according to the tree structure, and the metadata can be stored in the new metadata pool later, so that the metadata can be ensured to be normally stored. In addition, the method can obviously reduce the data quantity of migration and improve the efficiency of metadata migration because the method does not need to migrate the data of the file system.

Fig. 3 is a flowchart of another data processing method provided in the embodiment of the present application, and as shown in fig. 3, the embodiment of the present application is described by taking a server as an example. The data processing method includes the following steps.

301. The server receives a metadata migration instruction, wherein the metadata migration instruction is used for indicating metadata stored in a first metadata pool to be migrated to a second metadata pool, the first metadata pool is used for storing metadata of a file system according to a tree structure, the metadata is used for describing corresponding files or folders, the tree structure is used for representing the levels of the files and folders in the file system, and the levels of the files or folders in the same layer in the tree structure are the same.

In the embodiment of the application, the server stores the storage data uploaded by the plurality of terminals through the distributed storage system. Wherein there are multiple storage clusters in the distributed storage system. At least one data pool and a metadata pool are included within the storage cluster. The data pool is used for storing data of the file system, and the metadata pool is used for storing metadata of the file system. The data and metadata may be collectively referred to as stored data. The data is files, folders, etc., the metadata is names, sizes, creation times, etc. of the files or folders, and is used to describe the data. The file system comprises data and metadata uploaded by at least one terminal. The server may store data belonging to the same file system in the same data pool within the storage cluster; metadata belonging to the same file system is stored in the same metadata pool in the storage cluster, so that the server can manage the stored data through the distributed storage system.

For any metadata pool in the distributed storage system, in the case that the remaining storage capacity of the metadata pool is smaller than the storage capacity threshold, the distributed storage system sends a metadata migration instruction to the server to instruct the server to migrate metadata stored in the metadata pool to another new metadata pool with larger capacity. The storage capacity threshold may be a preset value, such as 1GB, 2GB, or 5GB, which is not limited in the embodiment of the present application. For convenience of description, a metadata pool of migrated metadata will be referred to as a first metadata pool, and a new metadata pool will be referred to as a second metadata pool.

In the first metadata pool, metadata of the file system is stored according to a tree structure. In the corresponding data pool, the data is stored in a directory structure of the data of the file system. The tree structure of the storage metadata is the same as the directory structure of the storage data. Each level of nodes in the tree structure represents each level of files or each level of folders in the directory structure, respectively.

302. The server is responsive to the metadata migration instructions to create a second metadata pool in the storage cluster, the storage cluster comprising a first metadata pool and at least one data pool, the data pool for storing files and folders of the file system, the storage capacity of the second metadata pool being greater than the storage capacity of the first metadata pool.

In an embodiment of the application, the server creates a second metadata pool in the storage cluster in response to the data migration instruction. The storage capacity of the second metadata pool is greater than the storage capacity of the first metadata pool. In addition, at least one data pool is included within the storage cluster. The data pool is used to store data of the file system.

By creating a second metadata pool with larger storage capacity, the server can not only migrate metadata of the file system from the first metadata pool to the second metadata pool, but also store metadata of the file system, which is subsequently uploaded by the terminal, in the second metadata pool. Therefore, by creating the second metadata pool, the condition that the metadata read-write fails due to insufficient storage capacity of the first metadata pool is avoided, and the normal read-write of the metadata is ensured.

In some embodiments, the data migration instruction also carries an ID (Identity Document, identification number) of the first metadata pool, an ID of the second metadata pool, and a storage capacity of the second metadata pool. For example, when the storage capacity of the second metadata pool carried by the data migration instruction is 100GB, the server determines, according to the data migration instruction, the storage cluster in which the first metadata pool is located by using the ID of the first metadata pool, and creates, by using a create pool instruction (create pool), a second metadata pool with a storage capacity of 100GB in the storage cluster in which the first metadata pool is located.

303. The server identifies metadata of the file system layer by layer in the first metadata pool based on the tree structure, and obtains metadata of files of each layer in the file system and metadata of folders of each layer.

In the embodiment of the application, the server identifies metadata of each node in the tree structure layer by layer in the first metadata pool according to the tree structure storing metadata of the file system, and obtains metadata of each layer of files in the file system and metadata of each layer of folders.

In order to more clearly explain the process of the server identifying metadata of the file system layer by layer in the first metadata pool, the process of the server identifying metadata layer by layer will be described in detail below through the following steps (a) - (d).

First, metadata of a file system will be briefly described. Metadata of a file system mainly includes: object (dir), directory entry (Denry), inode (index). Fig. 4 is a schematic diagram of metadata provided in an embodiment of the present application. As shown in fig. 4, a plurality of directory entries are included in a node object (dir) of a node. Each directory entry corresponds to a child node of the node. The directory entry is used to store the name of the file (folder) corresponding to the child node. The Inode is used for storing metadata such as the size, creation time, creator, access right and the like of a file (folder) corresponding to the child node. Therefore, it can be considered that two types of metadata, namely, a child node directory entry and an Inode, are stored in a parent node object. In addition, when the child node is a folder, the node object name of the child node is stored in the Inode of the child node. The node object of the child node stores metadata of the child node.

The following describes a procedure for the server to identify metadata layer by layer.

(a) The server obtains system metadata in a first metadata pool. Wherein the system MetaData includes MetaData of the root node and MetaData of MetaData services (MDS). Wherein the root node in the tree structure corresponds to the root directory in the directory structure. The metadata of the root node includes description data such as name, size, creation time, creator, read-write authority, and the like of the root directory. The metadata service is for maintaining a tree structure in the first metadata pool. The metadata of the metadata service includes nodes to be deleted in the tree structure of the first metadata pool.

In some embodiments, the server may obtain metadata for the root node and metadata for the metadata service based on the name of the metadata. Wherein the name of the metadata consists of an index number and a fragment number. The index number is a unique identifier allocated to the file or the folder by the distributed storage system in the process of creating the file or the folder. In addition, when the metadata of the node is large, the metadata of the node may be stored in a plurality of pieces. Wherein different slices of the node can be distinguished by the slice number.

For example, in the case where the index number of the root node is 1 and there are two fragment numbers of 00000000 and 00000001 of the root node, the server determines that the names of the two metadata of the root node are 1.00000000.Inode and 1.00000001.Inode, respectively. In the case where the index numbers of the metadata service are 100 and 101 and there are a plurality of fragment numbers xxxx of the metadata service, the server determines that the names of a plurality of metadata of the metadata service are 100.Xxxx.inode and 101.Xxxx.inode, respectively. Then, the server acquires metadata of the root node and metadata of the metadata service in the first metadata pool by the names of the metadata.

(b) The server identifies node objects of the root node in the first metadata pool according to the tree structure. The root node is located at the first layer of the tree structure, and metadata of at least one first-level node is stored in a node object of the root node. The first level node is a child node of the root node. The first level nodes are located at a second level of the tree structure. The root node is used to represent a folder, and the first level node is used to represent a file or folder.

In some embodiments, the name of the node object is also composed of an index number and a fragment number. Since metadata of a plurality of child nodes is stored in a node object of a node. Therefore, in the case where there are a plurality of child nodes in a node, in order to reduce the amount of metadata stored in the node object, a plurality of node objects may be allocated to the node. Metadata of at least one child node of the node is stored within each node object. Wherein different node objects of the node can be distinguished by the fragment number.

In some embodiments, the server identifies the root node object by its name. The server determines a root node, an index number of the root node and at least one fragment number of the root node in a first metadata pool according to the tree structure. The index number of the root node is a unique identification allocated to the root directory by the distributed storage system. The fragment number is used to identify the different node objects of the root node. The server determines the name of at least one node object of the root node according to the index number of the root node and the at least one fragment number of the root node. The server then identifies at least one node object of the root node in a first metadata pool based on the name of the at least one node object. Wherein the name of at least one node object corresponds to at least one fragment number one-to-one. Metadata of at least one child node of the root node is stored in a node object of the root node. Therefore, the problem of identifying the node object is converted into the keyword searching problem by searching the node object by the name, the time of identifying the node object in the first metadata pool is reduced, and the efficiency of identifying the node object is improved.

For example, in the case where the index number of the root node is 1 and there are two fragment numbers of 00000000 and 00000001 of the root node, the server determines that the names of the two node objects of the root node are 1.00000000 and 1.00000001, respectively. Then, the server identifies the node object in the first metadata pool by the name of the node object, and takes the identified node object as the node object of the root node.

(c) After the server determines the node object of the root node, the node object of each level of nodes is identified layer by layer. For any i-level node, in the case that the i-level node is used to represent a folder, it is indicated that the i-level node has a node object, and metadata of at least one child node (a file or a folder contained in the folder) is stored in the node object of the i-level node. Thus, the server identifies node objects for the ith level of nodes in the first metadata pool. The node object of the ith node stores metadata of at least one (i+1) th node. At least one i+1th level node is a child node of the i level node, the i+1th level node is located at the i+2th level of the tree structure, and i is an integer not less than 1.

In some embodiments, metadata for a node is stored in a node object by a key-value pair. Accordingly, in response to obtaining a node object of a multi-level node in the tree structure, for the node object of any node, the server may obtain metadata of at least one child node of the node according to at least one key value pair stored in the node object. Wherein one key value pair is used to store metadata of one child node. The key (key) in the key-value pair is the name of the corresponding child node, i.e. the directory entry (Denry) of the child node. The value (value) in the key-value pair is the metadata of the corresponding child node, that is, the Inode of the child node. Wherein the key-value pairs may be stored in an omap file (a file for storing key-value pairs) of the node object. The storage format of the metadata can be standardized by storing two metadata, namely the directory entry and the Inode of the node in a key value pair mode. The server or the terminal can conveniently acquire the metadata of different types of nodes by reading the key value pairs in the nodes, and the efficiency of acquiring the metadata is improved.

In some embodiments, the values in the key-value pair may also include a type field. The type field can indicate the type of node to which the key value pair corresponds. For example, for a key value pair of any node, in the case where the value of the type field in the value is 0, this node is indicated as representing a file, and there are no child nodes. In the case of a value of 1 in the value type field, this indicates that the node is used to represent a folder, and that there is at least one child node for this node.

In some embodiments, for any child node of the parent node, in the case that the child node is used to represent a folder, the index number and the fragment number of the child node are also stored in the key value pair corresponding to the child node. Thus, the server may determine the name of the node object of any child node in the key value pair stored in the parent node object. Then, the node object of the child node is identified in the first metadata pool according to the name of the node object of the child node. For example, for any i-th level node, if the i-th level node is a folder, the server acquires, in a key value pair corresponding to the i-th level node, an index number and at least one fragment number stored in the key value pair. The server then identifies at least one node object for the level i node in the first metadata pool based on the index number and the at least one fragment number.

Fig. 5 is a schematic diagram of a tree structure according to an embodiment of the present application. As shown in fig. 5, the root node includes 3 child nodes: /home,/lib and/data. The 3 child nodes are first level nodes. The 3 first level nodes are located at the second level of the tree structure. The first level node/lib includes 2 child nodes: /lib/lua and/lib/rpm. The 2 child nodes are second level nodes. The first level node/data includes 1 child node: data/log. The child node is also a second level node. The 3 second level nodes are located at the third level of the tree structure.

Fig. 6 is a schematic diagram of identifying metadata layer by layer according to an embodiment of the present application. As shown in fig. 6, the server identifies a node object named 1.00000000 in the first metadata pool according to the index number and the fragment number of the root directory. The node object is the node object of the root node. In the node object of the root node, the metadata of three first-level nodes of/home,/lib and/data are respectively stored through 3 key value pairs. Wherein, the key of the key value pair is the directory entry (name) of the first level node, and the value of the key value pair is the Inode of the first level node. Then, in the case where the first level node/lib is a folder, the server acquires the index number of/lib of 100001 and the fragment numbers of/lib of 00000000 and 00000001 in the Inode of/lib. The server identifies two node objects of/lib in the first metadata pool according to the index number and the fragment number of/lib. Wherein, in the node object named 100001.00000000, the metadata of the three second level nodes are stored by 3 key value pairs, namely, lib/local,/lib/lua and/lib/rpm. In the node object named 100001.00000001, metadata of three second level nodes are stored by 3 key value pairs,/lib/py 2.6,/lib/py 2.7, and/lib/py 3.6, respectively. The six second level nodes are all/lib child nodes.

(d) In the case of a tree structure having n levels, the server ends the layer-by-layer identification process in response to obtaining node objects for the n-1 level nodes. Since the metadata of the n-th level node is stored in the node object of the n-1-th level node. Therefore, in response to the server obtaining the node object of the n-1 level node, the server can obtain metadata of each level node in the tree structure from the node objects of the levels node.

Through the layer-by-layer identification process introduced in the steps (a) - (d), the server can identify the metadata of each level of nodes in the tree structure, thereby obtaining the metadata of each layer of files and the metadata of each layer of folders in the file system. Through layer-by-layer identification, metadata of a file system can be identified quickly, a tree structure for storing the metadata can be reserved, and the metadata can be tidied or migrated according to the tree structure conveniently.

304. The server migrates the metadata of each layer of files and the metadata of each layer of folders in the file system layer by layer to a second metadata pool.

In the embodiment of the application, after the server obtains the metadata of each layer of files and the metadata of each layer of folders in the file system through layer-by-layer identification, the server transfers the metadata to a second metadata pool with larger storage capacity layer by layer according to the tree structure of the metadata.

In order to more clearly describe the process of the server migrating metadata of the file system layer by layer to the second metadata pool, the following describes the process of the server migrating metadata layer by layer in detail through the following steps (1) - (3).

(1) The server migrates the system metadata to a second metadata pool. Wherein the system metadata includes metadata of the root node and metadata of the metadata service.

For example, fig. 7 is a schematic diagram of metadata of a migration system according to an embodiment of the present application. As shown in fig. 7, the server migrates metadata of the root node and metadata of the metadata service stored in the first metadata pool to the second metadata pool. The names of the metadata before and after migration are the same. For example, in both the first metadata pool and the second metadata pool, the name of the metadata of the root node is 1.0000000x.inode. Where 0000000x is used to represent the different fragment numbers.

(2) The server migrates node objects of the plurality of nodes in the tree structure to the second metadata pool in the order of the node levels from high to low. Wherein the level of the root node is highest. In the case of a tree structure having n levels in total, the level of the nth level node is lowest. Therefore, the server firstly migrates the node object of the root node to the second element tree pool, and finally migrates the node object of the n-1 level node to the second element data pool.

For the node object of any level node, the server may migrate the node object to the second metadata pool through the following steps (2.1) - (2.4).

(2.1) for the node object of any level of nodes, the server obtains a plurality of key value pairs stored in the node object, and updates the plurality of key value pairs in the node object.

In some embodiments, the server first reads a plurality of key-value pairs stored in the node object. Wherein, for any key value pair, the value in the key value pair is a sequence obtained by serializing metadata. Therefore, the server performs inverse-serialization analysis on the values in the key value pair to obtain the analyzed metadata. Then, the server changes the read-write attribute of the metadata, and prohibits the read-write of the metadata. And the server sequences the metadata after the read-write attribute is changed, and the value in the updated key value pair is obtained again.

For example, the server implements prohibition of a read-write request for metadata by setting the export_pin attribute of the metadata to-1. It should be noted that, a storage cluster typically includes a plurality of metadata services. The metadata service is used to process read and write requests for metadata. Accordingly, the export_pin attribute in the metadata is used to identify which metadata service is used to process read and write requests for the metadata. Therefore, by setting the export_pin attribute of the metadata to-1, the identification of the metadata service can be canceled, that is, the respective metadata services are prohibited from processing the read-write request for the metadata, thereby realizing prohibition of the read-write of the metadata.

In some embodiments, the server may read the plurality of key-value pairs stored in the node object in the omap file of the node object via a read_all_directory function (a data read function).

(2.2) the server obtains the statistical information of the corresponding node in the node object. The statistics information is used to represent the number of child nodes included in the node, that is, the statistics information can identify the number of files and child folders included in the folder corresponding to the node.

In some embodiments, the server may read the statistics of the node in the omaptader file of the node object via a read_fnode function (a data reading function).

(2.3) the server updating the metadata distribution information and the parent node distribution information in the node object. Wherein the metadata distribution information is used to identify a metadata pool in which the node object is stored. The parent node distribution information is used to identify a metadata pool that stores metadata for the node. Specifically, the server updates the metadata pool identified by the metadata distribution information and the parent node distribution information from the first metadata pool to the second metadata pool.

In some embodiments, the server may read metadata distribution information (layout) and parent node distribution information (backtrace) in the extended properties file (xattr) of the node object through a get_layout_parent_xattr function (a data reading function).

Fig. 8 is a schematic diagram of updating distribution information according to an embodiment of the present application. As shown in fig. 8, new_matadata_pool_id is the ID of the second metadata pool. The layout.pool.id is the ID of the metadata pool identified by the metadata distribution information. backtrace.pool.id is the ID of the metadata pool identified by the parent node distribution information.

(2.4) the server migrating the updated plurality of key-value pairs, the statistical information, the updated metadata distribution information, and the updated parent node distribution information to a second metadata pool. For example, the server migrates the updated plurality of key-value pairs to the omap file of the corresponding node object in the second metadata pool; migrating the statistical information to an omapeader file of the node object; and migrating the updated metadata distribution information and the updated parent node distribution information to the extended attribute file of the node object.

The names before and after the migration of the node object are the same. For example, in both the first metadata pool and the second metadata pool, the node object of the root node is named 1.0000000x. Wherein 0000000x is used to represent different fragment numbers of a node object.

Through the steps (2.1) - (2.4), the server can migrate the node object of any node from the first metadata pool to the second metadata pool, and in the process of migrating the node object, the read-write request of the metadata stored in the node object is forbidden, so that migration failure caused by the read-write request is avoided. In addition, the metadata distribution information of the node object is updated in the migration process, so that the terminal or other servers can determine the metadata pool for storing the node object through the latest metadata distribution information, and the metadata of the node object is prevented from being read in the wrong metadata pool.

Fig. 9 is a schematic diagram of a migration object node according to an embodiment of the present application. As shown in fig. 9, for any object node in the first metadata pool, the server migrates a plurality of key value pairs, statistical information, metadata distribution information and parent node distribution information stored in the object node to a corresponding object node in the second metadata pool, thereby implementing migration of the object node to the second metadata pool.

(3) In the case of a tree structure having n levels, the server ends the layer-by-layer migration process in response to successful migration of node objects of a plurality of n-1 level nodes to the second metadata pool. Since the metadata of the n-th level node is stored in the node object of the n-1-th level node. Thus, in the event that the server successfully migrates node objects of a plurality of n-1 level nodes to the second metadata pool, the server successfully migrates metadata of the file system to the second metadata pool.

Through the layer-by-layer migration process introduced in the steps (1) - (3), the server can not only quickly migrate the metadata of each layer of files and each layer of folders in the file system to the second metadata pool, but also maintain the tree structure for storing the metadata in the migration process, so that the metadata can be conveniently and continuously stored in the second metadata pool according to the same tree structure.

In some embodiments, in response to the server successfully migrating metadata of the file system to the second metadata pool, the server deletes the first metadata pool and metadata stored in the first metadata pool in the storage cluster, thereby freeing up a portion of storage space in the storage cluster.

In some embodiments, the server is also able to collect index numbers that have been used during migration of the metadata. Since the server identifies the node object in the first metadata pool by the name of the node object in the process of migrating the metadata layer by layer, and the name of the node object contains the index number of the node. Therefore, the server can determine a plurality of index numbers to be used according to names of a plurality of target objects to be migrated in the layer-by-layer migration process. The server then builds a first set from the plurality of index numbers that are used. The first set is the set of index numbers that have been used. Then, the server screens out unused index numbers from all index numbers according to the first set, and constructs a second set according to the unused index numbers. The second set is a set of unused index numbers. The server stores the second set in the second metadata pool, and in turn, in response to creating the node in the second metadata pool, the server may assign an unused index number to the node from the second set. In addition, the server may also determine the name of the node object of the newly created node using the index number.

By storing the second set in the second metadata pool, when the node is newly created, an index number can be randomly allocated to the newly created node in the second set, and the index number is not used by other nodes, so that the collision of the index numbers of the nodes is avoided.

In some embodiments, the first set may be a discontinuous set. Accordingly, the server gathers a plurality of used index numbers through the intermittent collection. For example, in the case where index numbers 12, 20, and 21 have been used, the server builds a break set { <12,1>, <20,2> }. Where <12,1> in the break set indicates that starting with index number 12,1 consecutive index number has been used, i.e. index number 12 has been used. The <20,2> in the break set indicates that starting from index number 20,2 consecutive index numbers have been used, i.e. index numbers 20 and 21 have been used.

In some embodiments, at least one metadata service is also included in the storage cluster. Each metadata service is responsible for index number assignment for an area. Wherein, there are a plurality of index numbers in each region. In response to the server successfully building the first set, the metadata service determines a plurality of unused index numbers within the corresponding region from the used index numbers recorded in the first set. The metadata service then builds a set of unused index numbers within the region (i.e., a second set within the region) and stores the set in a second metadata pool through an object named mdsx_inotable. Wherein mdsx_inotable is used to represent a second set of x-th metadata service (MDS) constructs. For example, fig. 10 is a schematic diagram of an index number provided in an embodiment of the present application. As shown in fig. 10, all index numbers are equally divided into four parts, each part including 2 ⁴⁰ And index numbers. Wherein the index number of the first part is used for assigning to system metadata. The index numbers of the remaining three parts are assigned to other nodes by corresponding metadata services.

In some embodiments, after the server migration metadata is complete, the server is also able to update the file system associated metadata pool in the file system view. The file system view is used to record the file systems stored in the distributed storage system and the data pools and metadata pools associated with each file system. The file system associated data pool is stored with file system data, and the file system associated metadata pool is stored with file system metadata. Thus, in response to the server successfully migrating metadata of the file system to the second metadata pool, the server updates the metadata pool associated with the file system from the first metadata pool to the second metadata pool in the file system view. The server then synchronizes the updated file system view to a plurality of clients (terminals) such that the clients read and write metadata of the file system in a metadata pool associated with the file system based on the updated file system view. By updating the file system view in time after metadata is migrated and synchronizing the updated file system view to the client, metadata read-write failure caused by the fact that the client reads and writes metadata in an old metadata pool can be avoided.

For example, fig. 11 is a schematic diagram of a synchronous file system view provided in an embodiment of the present application. As shown in fig. 11, a distributed storage system includes a plurality of MON clusters (a type of cluster for managing storage clusters). The server sends the updated file system view to the leader node (master node) of the MON cluster. The updated file system view is then synchronized by the leader node of the MON cluster to the other nodes of the MON cluster via the paxos protocol (a synchronization protocol). In response to receiving the updated file system view, any node of the MON cluster sends the updated file system view to the plurality of target clients. The target client is a terminal or a server subscribed to the file system view update. Accordingly, after the target client receives the updated file system view, the target client determines a metadata pool associated with the file system according to the file system view. Fig. 12 is a schematic diagram of reading and writing metadata according to an embodiment of the present application. As shown in fig. 12, in the process of reading and writing metadata from the metadata pool by the target client, the target client sends a metadata read-write request to the metadata service, and the metadata service processes the read-write request to realize that the target client reads and writes metadata in the metadata pool. In addition, in the process of reading and writing data from the data pool by the target client, the target client directly sends a data reading and writing request to the data pool, and the data pool processes the data reading and writing request, so that the target client reads and writes data in the data pool.

Fig. 13 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application. Referring to fig. 13, the apparatus includes: a receiving module 1301, an identifying module 1302, and a migrating module 1303.

A receiving module 1301, configured to receive a metadata migration instruction, where the metadata migration instruction is configured to instruct migration of metadata stored in a first metadata pool to a second metadata pool, where the first metadata pool is configured to store metadata of a file system according to a tree structure, the metadata is configured to describe a corresponding file or folder, the tree structure is configured to represent a hierarchy of the file and folder in the file system, and the hierarchy of the file or folder in the same layer in the tree structure is the same;

The identifying module 1302 is configured to identify metadata of the file system layer by layer in the first metadata pool based on the tree structure, so as to obtain metadata of files of each layer in the file system and metadata of folders of each layer;

the migration module 1303 is configured to migrate, layer by layer, metadata of each layer of files and metadata of each layer of folders in the file system to a second metadata pool, where a storage capacity of the second metadata pool is greater than a storage capacity of the first metadata pool.

In some embodiments, the identifying module 1302 is configured to identify, based on a tree structure, a node object of a root node in a first metadata pool, where the root node is located at a first level of the tree structure, the node object of the root node stores metadata of at least one first level node, the first level node is a child node of the root node, the first level node is located at a second level of the tree structure, the root node is used to represent a folder, and the first level node is used to represent a file or a folder; for any ith node, under the condition that the ith node is used for representing a folder, identifying a node object of the ith node in a first metadata pool, wherein metadata of at least one (i+1) th node is stored in the node object of the ith node, the (i+1) th node is a child node of the ith node, the (i+1) th node is positioned at the (i+2) th layer of the tree structure, and i is an integer not less than 1; in the case of a tree structure having n levels, the process of layer-by-layer identification is ended in response to obtaining node objects for the n-1 th level nodes.

In some embodiments, the identifying module 1302 is configured to determine, based on the tree structure, a root node, an index number of the root node, and at least one fragment number of the root node in the first metadata pool, the fragment number being used to identify a node object of the root node; at least one node object of the root node is identified in the first metadata pool based on the index number of the root node and at least one fragment number of the root node, and a name of the at least one node object corresponds to the at least one fragment number one-to-one.

In some embodiments, fig. 14 is a schematic structural diagram of another data processing apparatus provided in an embodiment of the present application, as shown in fig. 14, where the apparatus further includes:

the first obtaining module 1304 is configured to, in response to obtaining a node object of a plurality of levels of nodes in the tree structure, obtain, for a node object of any node, metadata of at least one child node of the node based on at least one key-value pair stored in the node object, where the at least one key-value pair is used to store metadata of the at least one child node, a key in the key-value pair is a name of the corresponding child node, and a value in the key-value pair is metadata of the corresponding child node.

In some embodiments, for any child node, in the case that the child node is used to represent a folder, the index number and the fragment number of the child node are also stored in the key value pair corresponding to the child node;

The identifying module 1302 is configured to, for any i-th level node, obtain, when the i-th level node is used for representing a folder, an index number and at least one fragment number stored in a key value pair corresponding to the i-th level node; at least one node object of the ith level node is identified in the first metadata pool based on the index number and the at least one fragment number.

In some embodiments, the apparatus further comprises:

a second obtaining module 1305, configured to obtain system metadata in the first metadata pool, where the system metadata includes metadata of the root node and metadata of a metadata service, and the metadata service is configured to maintain a tree structure in the first metadata pool.

In some embodiments, the migration module 1303 is configured to migrate the system metadata to the second metadata pool; according to the order of the levels of the nodes from high to low, migrating the node objects of a plurality of nodes in the tree structure to a second metadata pool, wherein the level of the root node is highest, and the level of the nth level node is lowest; and ending the layer-by-layer migration process in response to successful migration of the node objects of the plurality of n-1 level nodes to the second metadata pool.

In some embodiments, the migration module 1303 is configured to update, for a node object of any node, a plurality of key value pairs in the node object; acquiring statistical information of the nodes from the node objects, wherein the statistical information is used for representing the number of child nodes contained in the nodes; updating metadata distribution information and parent node distribution information in the node object, wherein the metadata distribution information is used for identifying a metadata pool of the storage node object, and the parent node distribution information is used for identifying the metadata pool of the storage node; and migrating the updated plurality of key value pairs, the statistical information, the updated metadata distribution information and the updated parent node distribution information into a second metadata pool.

In some embodiments, the migration module 1303 is configured to, for a node object of any node, read a plurality of key value pairs in the node object; for any key value pair, performing inverse sequence analysis on the values in the key value pair to obtain analyzed metadata; the read-write attribute of the metadata is changed, so that the metadata is forbidden to be read and written; and serializing the changed metadata to obtain the value in the updated key value pair.

In some embodiments, for any target object, the name of the target object is the index number and the fragment number of the corresponding node, and the apparatus further includes:

a building module 1306, configured to build, in a layer-by-layer migration process, a first set based on names of a plurality of migrated target objects, where the first set is a set of used index numbers;

a screening module 1307, configured to screen a second set from all index numbers based on the first set, where the second set is a set of unused index numbers;

a storage module 1308 for storing the second set in a second metadata pool;

an allocation module 1309 for allocating an index number to a node from the second set in response to an operation of creating the node in the second metadata pool.

In some embodiments, the apparatus further comprises:

a creation module 1310 for creating a second metadata pool in a storage cluster in response to the metadata migration instruction, the storage cluster comprising a first metadata pool and at least one data pool, the data pool for storing files and folders of a file system.

In some embodiments, the apparatus further comprises:

an updating module 1311, configured to update, in response to successful migration of metadata of the file system to the second metadata pool, the metadata pool associated with the file system from the first metadata pool to the second metadata pool in a file system view, where the file system view is used to store an association relationship between the file system and the metadata pool;

and a synchronization module 1312, configured to synchronize the updated file system view to the plurality of clients, so that the clients read and write metadata of the file system from and into a metadata pool associated with the file system according to the updated file system view.

The embodiment of the application provides a data processing device. Since the first metadata pool stores metadata according to a tree structure, the hierarchical structure of files and folders in the file system can be intuitively determined through the tree structure. Therefore, when the storage capacity of the first metadata pool is insufficient, the computer equipment can only migrate the metadata of the file system to the second metadata pool with larger storage capacity layer by layer according to the tree structure, and then the metadata can be stored in the new metadata pool, so that the normal storage of the metadata is ensured. In addition, the method can obviously reduce the data quantity of migration and improve the efficiency of metadata migration because the method does not need to migrate the data of the file system.

It should be noted that: the data processing apparatus provided in the above embodiment is only exemplified by the division of the above functional modules, and in practical application, the above functional allocation may be performed by different functional modules according to needs, that is, the internal structure of the terminal is divided into different functional modules, so as to perform all or part of the functions described above. In addition, the embodiments of the data processing apparatus and the data processing method provided in the foregoing embodiments belong to the same concept, and specific implementation processes of the embodiments of the method are detailed in the method embodiments, which are not repeated herein.

The embodiment of the application also provides a terminal, which comprises a processor and a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor to realize the data processing method provided by the embodiment.

Fig. 15 is a schematic structural diagram of a terminal according to an embodiment of the present application.

The terminal 1500 includes: a processor 1501 and a memory 1502.

The processor 1501 may include one or more processing cores, such as a 4-core processor, an 8-core processor, or the like. The processor 1501 may be implemented in at least one hardware form of DSP (Digital Signal Processing ), FPGA (Field Programmable Gate Array, field programmable gate array), PLA (Programmable Logic Array ). The processor 1501 may also include a main processor, which is a processor for processing data in an awake state, also called a CPU (Central Processing Unit ), and a coprocessor; a coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 1501 may be integrated with a GPU (Graphics Processing Unit, data processing interactor) for responsible for rendering and rendering of the content required to be displayed by the display screen. In some embodiments, the processor 1501 may also include an AI (Artificial Intelligence ) processor for processing computing operations related to machine learning.

Memory 1502 may include one or more computer-readable storage media, which may be non-transitory. Memory 1502 may also include high-speed random access memory, as well as non-volatile memory, such as one or more magnetic disk storage devices, flash memory storage devices. In some embodiments, a non-transitory computer readable storage medium in memory 1502 is used to store at least one computer program for being possessed by processor 1501 to implement the data processing methods provided by the method embodiments herein.

In some embodiments, the terminal 1500 may further optionally include: a peripheral interface 1503 and at least one peripheral device. The processor 1501, memory 1502 and peripheral interface 1503 may be connected by a bus or signal lines. The individual peripheral devices may be connected to the peripheral device interface 1503 via a bus, signal lines, or circuit board. Optionally, the peripheral device comprises: at least one of radio frequency circuitry 1504, a display 1505, a camera assembly 1506, audio circuitry 1507, and a power supply 1508.

A peripheral interface 1503 may be used to connect I/O (Input/Output) related at least one peripheral device to the processor 1501 and the memory 1502. In some embodiments, processor 1501, memory 1502, and peripheral interface 1503 are integrated on the same chip or circuit board; in some other embodiments, either or both of the processor 1501, the memory 1502, and the peripheral interface 1503 may be implemented on separate chips or circuit boards, which is not limited in this embodiment.

The Radio Frequency circuit 1504 is configured to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1504 communicates with a communication network and other communication devices via electromagnetic signals. The radio frequency circuit 1504 converts an electrical signal into an electromagnetic signal for transmission, or converts a received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 1504 includes: antenna systems, RF transceivers, one or more amplifiers, tuners, oscillators, digital signal processors, codec chipsets, subscriber identity module cards, and so forth. The radio frequency circuit 1504 may communicate with other devices via at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: metropolitan area networks, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and/or WiFi (Wireless Fidelity ) networks. In some embodiments, the radio frequency circuit 1504 may also include NFC (Near Field Communication, short range wireless communication) related circuits, which are not limited in this application.

Display 1505 is used to display a UI (User Interface). The UI may include graphics, text, icons, video, and any combination thereof. When display screen 1505 is a touch display screen, display screen 1505 also has the ability to collect touch signals at or above the surface of display screen 1505. The touch signal may be input to the processor 1501 as a control signal for processing. At this point, display 1505 may also be used to provide virtual buttons and/or a virtual keyboard, also referred to as soft buttons and/or a soft keyboard. In some embodiments, the display 1505 may be one, disposed on the front panel of the terminal 1500; in other embodiments, the display 1505 may be at least two, respectively disposed on different surfaces of the terminal 1500 or in a folded design; in other embodiments, display 1505 may be a flexible display disposed on a curved surface or a folded surface of terminal 1500. Even more, the display 1505 may be arranged in a non-rectangular irregular pattern, i.e., a shaped screen. The display screen 1505 may be made of LCD (Liquid Crystal Display ), OLED (Organic Light-Emitting Diode) or other materials.

The camera assembly 1506 is used to capture images or video. Optionally, the camera assembly 1506 includes a front camera and a rear camera. The front camera is disposed on the front panel of the terminal 1500, and the rear camera is disposed on the rear surface of the terminal 1500. In some embodiments, the at least two rear cameras are any one of a main camera, a depth camera, a wide-angle camera and a tele camera, so as to realize that the main camera and the depth camera are fused to realize a background blurring function, and the main camera and the wide-angle camera are fused to realize a panoramic shooting and Virtual Reality (VR) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 1506 may also include a flash. The flash lamp can be a single-color temperature flash lamp or a double-color temperature flash lamp. The dual-color temperature flash lamp refers to a combination of a warm light flash lamp and a cold light flash lamp, and can be used for light compensation under different color temperatures.

The audio circuitry 1507 may include a microphone and a speaker. The microphone is used for collecting sound waves of users and the environment, converting the sound waves into electric signals, inputting the electric signals to the processor 1501 for processing, or inputting the electric signals to the radio frequency circuit 1504 for voice communication. For purposes of stereo acquisition or noise reduction, a plurality of microphones may be respectively disposed at different portions of the terminal 1500. The microphone may also be an array microphone or an omni-directional pickup microphone. The speaker is used to convert electrical signals from the processor 1501 or the radio frequency circuit 1504 into sound waves. The speaker may be a conventional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, not only the electric signal can be converted into a sound wave audible to humans, but also the electric signal can be converted into a sound wave inaudible to humans for ranging and other purposes. In some embodiments, the audio circuit 1507 may also include a headphone jack.

The power supply 1508 is used to power the various components in the terminal 1500. The power source 1508 may be alternating current, direct current, disposable battery, or rechargeable battery. When the power source 1508 includes a rechargeable battery, the rechargeable battery may support wired or wireless charging. The rechargeable battery may also be used to support fast charge technology.

In some embodiments, the terminal 1500 also includes one or more sensors 1509. The one or more sensors 1509 include, but are not limited to: an acceleration sensor 1510, a gyro sensor 1511, a pressure sensor 1512, an optical sensor 1513, and a proximity sensor 1515.

The acceleration sensor 1510 may detect the magnitudes of accelerations on three coordinate axes of a coordinate system established with the terminal 1500. For example, the acceleration sensor 1510 may be used to detect components of gravitational acceleration in three coordinate axes. The processor 1501 may control the display screen 1505 to display the user interface in either a landscape view or a portrait view based on the gravitational acceleration signal collected by the acceleration sensor 1510. The acceleration sensor 1510 may also be used for acquisition of motion data of a game or user.

The gyro sensor 1511 may detect a body direction and a rotation angle of the terminal 1500, and the gyro sensor 1511 may collect a 3D motion of the user to the terminal 1500 in cooperation with the acceleration sensor 1510. The processor 1501, based on the data collected by the gyro sensor 1511, may implement the following functions: motion sensing (e.g., changing UI according to a tilting operation by a user), image stabilization at shooting, game control, and inertial navigation.

The pressure sensor 1512 may be disposed on a side frame of the terminal 1500 and/or below the display 1505. When the pressure sensor 1512 is disposed on the side frame of the terminal 1500, a grip signal of the terminal 1500 by the user may be detected, and the processor 1501 performs a left-right hand recognition or a shortcut operation according to the grip signal collected by the pressure sensor 1512. When the pressure sensor 1512 is disposed at the lower layer of the display screen 1505, the processor 1501 controls the operability control on the UI interface according to the pressure operation of the user on the display screen 1505. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

The optical sensor 1513 is used to collect the ambient light intensity. In one embodiment, processor 1501 may control the display brightness of display screen 1505 based on the intensity of ambient light collected by optical sensor 1513. Optionally, when the ambient light intensity is high, the display brightness of the display screen 1505 is turned up; when the ambient light intensity is low, the display luminance of the display screen 1505 is turned down. In another embodiment, the processor 1501 may also dynamically adjust the shooting parameters of the camera assembly 1506 based on the ambient light intensity collected by the optical sensor 1513.

A proximity sensor 1514, also referred to as a distance sensor, is provided on the front panel of the terminal 1500. The proximity sensor 1514 is used to collect the distance between the user and the front of the terminal 1500. In one embodiment, when the proximity sensor 1514 detects a gradual decrease in the distance between the user and the front of the terminal 1500, the processor 1501 controls the display 1505 to switch from the on-screen state to the off-screen state; when the proximity sensor 1514 detects that the distance between the user and the front surface of the terminal 1500 gradually increases, the processor 1501 controls the display screen 1505 to switch from the off-screen state to the on-screen state.

Those skilled in the art will appreciate that the structure shown in fig. 15 is not limiting and that more or fewer components than shown may be included or certain components may be combined or a different arrangement of components may be employed.

Fig. 16 is a schematic structural diagram of a server provided in the embodiments of the present application, where the server 1600 may have a relatively large difference due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 1601 and one or more memories 1602, where at least one computer program is stored in the memories 1602, and the at least one computer program is loaded and executed by the processor 1601 to implement the data processing method provided in the respective method embodiments described above. Of course, the server 1600 may also have a wired or wireless network interface, a keyboard, an input/output interface, etc. for performing input/output, and the server 1600 may also include other components for implementing functions of the device, which are not described herein.

The embodiment of the application also provides a computer readable storage medium, and at least one computer program is stored in the computer readable storage medium, and the at least one computer program is loaded and executed by a processor to implement the data processing method provided by the embodiment.

The embodiments of the present application also provide a computer program product, including a computer program, where the computer program is loaded and executed by a processor to implement the data processing method provided in the foregoing embodiments.

It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program for instructing relevant hardware, where the program may be stored in a computer readable storage medium, and the storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.

The foregoing description of the embodiments is merely an optional embodiment and is not intended to limit the embodiments, and any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application are intended to be included in the scope of the present application.

Claims

1. A method of data processing, the method comprising:

identifying node objects of each level of nodes layer by layer in the first metadata pool based on the tree structure to obtain metadata of files of each level and metadata of folders of each level in the file system, wherein nodes in the tree structure are used for representing files or folders in the file system, and metadata of a plurality of child nodes of the nodes are stored in the node objects of any node;

2. The method according to claim 1, wherein identifying node objects of each level of nodes layer by layer in the first metadata pool based on the tree structure, to obtain metadata of files of each level and metadata of folders of each level in the file system, comprises:

Based on the tree structure, in the first metadata pool, identifying a node object of a root node, wherein the root node is located at a first layer of the tree structure, metadata of at least one first-level node is stored in the node object of the root node, the first-level node is a child node of the root node, the first-level node is located at a second layer of the tree structure, the root node is used for representing a folder, and the first-level node is used for representing a file or a folder;

for any ith level node, in the case that the ith level node is used for representing a folder, identifying a node object of the ith level node in the first metadata pool, wherein metadata of at least one (i+1) th level node is stored in the node object of the ith level node, the (i+1) th level node is a child node of the ith level node, the (i+1) th level node is located at the (i+2) th level of the tree structure, and i is an integer not less than 1;

and under the condition that the tree structure has n levels, responding to the node object of the n-1 level node, and ending the layer-by-layer identification process.

3. The method of claim 2, wherein the identifying node objects of a root node in the first metadata pool based on the tree structure comprises:

Determining a root node, an index number of the root node and at least one fragment number of the root node in the first metadata pool based on the tree structure, wherein the fragment number is used for identifying a node object of the root node;

and identifying at least one node object of the root node in the first metadata pool based on the index number of the root node and at least one fragment number of the root node, wherein the name of the at least one node object corresponds to the at least one fragment number one by one.

4. The method according to claim 2, wherein the method further comprises:

and responding to the node object of the multilevel nodes in the tree structure, for the node object of any node, acquiring metadata of at least one child node of the node based on at least one key value pair stored in the node object, wherein the at least one key value pair is used for storing the metadata of the at least one child node, a key in the key value pair is the name of the corresponding child node, and a value in the key value pair is the metadata of the corresponding child node.

5. The method according to claim 4, wherein for any child node, in the case that the child node is used to represent a folder, the key value pair corresponding to the child node further stores an index number and a fragment number of the child node;

The identifying, for any i-th level node, in the first metadata pool, a node object of the i-th level node in a case that the i-th level node is used to represent a folder, including:

for any ith node, under the condition that the ith node is used for representing a folder, acquiring an index number and at least one fragment number stored in a key value pair corresponding to the ith node;

at least one node object of the ith level node is identified in the first metadata pool based on the index number and at least one fragment number.

6. The method according to claim 2, wherein the method further comprises:

in the first metadata pool, system metadata is acquired, wherein the system metadata comprises metadata of the root node and metadata of metadata services, and the metadata services are used for maintaining a tree structure in the first metadata pool.

7. The method of claim 6, wherein migrating the metadata of the files of each layer and the metadata of the folders of each layer in the file system layer by layer to the second metadata pool comprises:

migrating the system metadata to the second metadata pool;

Migrating node objects of a plurality of nodes in the tree structure to the second metadata pool according to the order of the levels of the nodes from high to low, wherein the level of the root node is highest, and the level of the nth level node is lowest;

and ending the layer-by-layer migration process in response to successful migration of node objects of the plurality of n-1 level nodes to the second metadata pool.

8. The method of claim 7, wherein migrating the node object of any node to the second metadata pool comprises:

for a node object of any node, updating a plurality of key value pairs in the node object;

acquiring statistical information of the node in the node object, wherein the statistical information is used for representing the number of child nodes contained in the node;

updating metadata distribution information and father node distribution information in the node object, wherein the metadata distribution information is used for identifying a metadata pool for storing the node object, and the father node distribution information is used for identifying a metadata pool for storing the metadata of the node;

and migrating the updated plurality of key value pairs, the statistical information, the updated metadata distribution information and the updated parent node distribution information into the second metadata pool.

9. The method of claim 8, wherein for a node object of any node, updating a plurality of key-value pairs in the node object comprises:

for a node object of any node, reading a plurality of key value pairs in the node object;

for any key value pair, performing inverse sequence analysis on the values in the key value pair to obtain analyzed metadata;

the read-write attribute of the metadata is changed, so that the metadata is forbidden to be read and written;

and serializing the changed metadata to obtain the value in the updated key value pair.

10. The method of claim 7, wherein for any target object, the target object is named an index number and a fragment number of a corresponding node, the method further comprising:

in the layer-by-layer migration process, constructing a first set based on names of a plurality of target objects to be migrated, wherein the first set is a set of index numbers which are already used;

screening a second set from all index numbers based on the first set, wherein the second set is an unused index number set;

storing the second set in the second metadata pool;

In response to creating a node in the second metadata pool, an index number is assigned to the node from the second set.

11. The method according to claim 1, wherein the method further comprises:

in response to the metadata migration instruction, the second metadata pool is created in a storage cluster, the storage cluster including the first metadata pool and at least one data pool for storing files and folders of the file system.

12. The method according to claim 1, wherein the method further comprises:

in response to successful migration of metadata of the file system to the second metadata pool, updating the metadata pool associated with the file system from the first metadata pool to the second metadata pool in a file system view, the file system view being used to store an association between the file system and the metadata pool;

and synchronizing the updated file system view to a plurality of clients so that the clients read and write metadata of the file system in a metadata pool associated with the file system according to the updated file system view.

13. A data processing apparatus, the apparatus comprising:

the identifying module is used for identifying node objects of each level of nodes layer by layer in the first metadata pool based on the tree structure to obtain metadata of files of each level and metadata of folders of each level in the file system, wherein the nodes in the tree structure are used for representing the files or folders in the file system, and the metadata of a plurality of child nodes of the nodes are stored in the node objects of any node;

14. A computer device, characterized in that it comprises a processor and a memory in which at least one computer program is stored, which is loaded and executed by the processor to implement the data processing method according to any of claims 1 to 12.

15. A computer readable storage medium, characterized in that at least one computer program is stored in the computer readable storage medium, which is loaded and executed by a processor for realizing the data processing method according to any of claims 1 to 12.