CN116820347A

CN116820347A - Distributed object processing method and device

Info

Publication number: CN116820347A
Application number: CN202310874147.2A
Authority: CN
Inventors: 黄梦龙; 李云龙; 陈亮
Original assignee: Hunan Happly Sunshine Interactive Entertainment Media Co Ltd
Current assignee: Hunan Happly Sunshine Interactive Entertainment Media Co Ltd
Priority date: 2023-07-17
Filing date: 2023-07-17
Publication date: 2023-09-29

Abstract

The application provides a processing method and a processing device of a distributed object, which are applied to first equipment provided with a metadata database, wherein the method comprises the following steps: obtaining a user request, wherein the user request is a request for processing a target object mounted on an operating system in a distributed object; and responding to the user request to process metadata of the target object, wherein the metadata comprises attribute information of the target object.

Description

Distributed object processing method and device

Technical Field

The present application relates to the field of information technologies, and in particular, to a method and an apparatus for processing a distributed object.

Background

An object store is a data store in which each unit of data (called an "object") is stored as a discrete unit. These objects may be virtually any type of data: pdf, video, audio, text, website data, or any other file type.

In contrast to file storage, these objects are stored in a single flat structure, with no file-interlayer sub-structure. In object storage, all objects are stored in a flat address space, unlike the nested hierarchies used for file storage. Furthermore, all default and custom metadata is stored in a flat address space with unique identifiers along with the object itself (not as part of a separate file system table or index).

In the prior art, a user uses an open-source tool such as rclone, s3fs and the like to mount a bucket of a distributed object storage minio on an operating system such as windows/macos, specifically on a file system of a local disk of the user, and then performs some conventional operations on a bucket directory or a file, including directory renaming, file copying and the like, so that the file needs to be repeatedly uploaded.

Disclosure of Invention

In view of this, the present application provides a method and apparatus for processing a distributed object, as follows:

a method of processing a distributed object for use with a first device provided with a metadata database, the method comprising:

obtaining a user request, wherein the user request is a request for processing a target object mounted on an operating system in a distributed object;

and responding to the user request to process metadata of the target object, wherein the metadata comprises attribute information of the target object.

Optionally, in the method, the obtaining the user request includes:

a user request sent by a processor is received, wherein the user request is generated by the processor based on input operation of an input device, and the processor runs an operating system.

Optionally, the method, wherein the user request includes any one of a copy file request and a directory renaming request, and the responding to the user request processes metadata of the target object includes:

Determining a preset key value pair based on the storage position of the target object in the user request;

and updating the key value of the preset key value pair based on the request content of the user request, wherein the preset key value pair comprises at least one item of metadata of the target object.

Optionally, in the above method, when the user request is a file copying request, updating the key value of the preset key value pair based on the request content of the user request includes:

inserting a first key value pair into the metadata database, wherein the first key value pair comprises a storage position of the target object after being copied in the distributed object database;

and adding one to the count information in the preset key value pair and the count information in the first key value pair respectively, wherein the count value represents the number of the files pointing to the same object.

Optionally, in the above method, when the user request is a directory renaming request, determining the preset key value pair based on the storage location of the target object in the user request includes:

obtaining key value pairs of at least one object contained in the renamed directory;

correspondingly, the updating the key value of the preset key value pair based on the request content requested by the user comprises the following steps:

Updating target field information in the key value pair of the at least one object according to the renamed name, wherein the target field information characterizes the storage position of the at least one object in the distributed object database.

Optionally, in the method, the user request is a file deletion request, and the processing the metadata of the target object in response to the user request includes:

determining a storage position of a target object in the user request in a distributed object database;

requesting, by an operating system based on the storage location control, deletion of the target object from the distributed object database;

and deleting the preset key value pair corresponding to the target object.

Optionally, in the method, the user request is a query file request, and the processing the metadata of the target object in response to the user request includes:

and determining the storage position of the target object in the distributed object database in the user request, and feeding back to an operating system so that the operating system requests the target object from the distributed object database based on the storage position.

A method of processing a distributed object for use with a second device provided with an operating system, the method comprising:

analyzing the user request to obtain the type of the user request, wherein the type comprises a first type related to metadata of an object and a second type different from the first type;

if the type of the user request is a first type, requesting to a metadata database based on the user request, so that the metadata database responds to the user request to process the metadata of the target object, wherein the metadata stored in the metadata database are metadata corresponding to each distributed object in a distributed object database;

and if the type of the user request is a second type, requesting the distributed object database based on the user request, so that the distributed object database responds to the user request to process the target object.

A processing apparatus of a distributed object, applied to a first device provided with a metadata database, comprising:

the first obtaining module is used for obtaining a user request, wherein the user request is a request for processing a target object which is mounted on an operating system in a distributed object;

And the response module is used for responding to the user request and processing metadata of the target object, wherein the metadata comprises attribute information of the target object.

A processing apparatus of a distributed object, applied to a second device provided with an operating system, comprising:

the second obtaining module is used for obtaining a user request, wherein the user request is a request for processing a target object which is mounted on an operating system in a distributed object;

an analysis module, configured to analyze the user request to obtain a type of the user request, where the type includes a first type related to metadata of an object and a second type different from the first type;

the request module is used for requesting a metadata database based on the user request if the type of the user request is a first type, so that the metadata database responds to the user request to process the metadata of the target object, and the metadata stored in the metadata database are metadata corresponding to each distributed object in the distributed object database; and if the type of the user request is a second type, requesting the distributed object database based on the user request, so that the distributed object database responds to the user request to process the target object.

In summary, the method and apparatus for processing a distributed object provided in this embodiment include: obtaining a user request, wherein the user request is a request for processing a target object mounted on an operating system in a distributed object; and responding to the user request to process metadata of the target object, wherein the metadata comprises attribute information of the target object. When a user request for processing a target object mounted on an operating system in a distributed object is obtained, the metadata database responds to the target request based on the attribute information of the target object, the metadata of the target object is processed, the process does not need to process the data of the target object, and the process of re-storing or uploading the file object in the distributed database is not needed, so that the storage space, the bandwidth and the like of the distributed object database are saved.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for the description of the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present application, and that other drawings may be obtained according to the provided drawings without inventive effort for a person skilled in the art.

FIG. 1 is a flowchart of an embodiment 1 of a method for processing a distributed object according to the present application;

FIG. 2 is a schematic diagram of file storage to object storage in embodiment 1 of a method for processing a distributed object according to the present application;

fig. 3 is an application scenario schematic diagram of embodiment 1 of a method for processing a distributed object provided by the present application;

FIG. 4 is a flowchart of an embodiment 2 of a method for processing a distributed object according to the present application;

FIG. 5 is a flowchart of an embodiment 3 of a method for processing a distributed object according to the present application;

FIG. 6 is a flowchart of an embodiment 4 of a method for processing a distributed object according to the present application;

FIG. 7 is a flowchart of an embodiment 5 of a method for processing a distributed object according to the present application;

FIG. 8 is a flowchart of an embodiment 6 of a method for processing a distributed object according to the present application;

FIG. 9 is a flowchart of an embodiment 7 of a method for processing a distributed object according to the present application;

FIG. 10 is a flowchart of an embodiment 8 of a method for processing a distributed object according to the present application;

FIG. 11 is a schematic diagram of an application scenario of a method for processing a distributed object according to the present application;

FIG. 12 is a schematic diagram of an embodiment 1 of a distributed object processing apparatus according to the present application;

Fig. 13 is a schematic structural diagram of an embodiment 2 of a distributed object processing apparatus according to the present application.

Detailed Description

The following description of the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.

As shown in fig. 1, a flowchart of an embodiment 1 of a method for processing a distributed object according to the present application is applied to a first device provided with a metadata database, and includes the following steps:

step S101: obtaining a user request;

wherein the user request is a request for processing a target object mounted on an operating system in a distributed object.

The target object in this embodiment is an object mounted on the operating system in a distributed manner.

In a specific implementation, the storage object is composed of distributed object storage software and a linux storage server, and video/picture/files are kept in a distributed object storage database, which is called an object.

Wherein the object is a basic unit of data storage in the distributed object storage system, an object is actually a combination of any type of data such as a file/picture/video and a set of attribute information, and the object metadata refers to attribute information, and the attribute information includes, but is not limited to, name, size, label and the like.

Shown in fig. 2 is a schematic diagram of file storage to object storage, comprising: a plurality of videos, pictures and files are respectively stored in the linux server storage, the videos, pictures and files are called objects, and one video, one graph or one file is respectively called one object.

Specifically, the process of storing a video/picture/file into an object store is as follows: the user/administrator creates a bucket; the user submits a video (such as a first period of the program A. Vod) to a certain bucket of the object storage cluster, such as/bucket 1, the object storage cluster calculates which linux nodes of the cluster the video should be stored to (the system can split the video to obtain split multi-part data) according to the name of the video by utilizing a hash algorithm, and then the split data is stored to the corresponding linux nodes.

Accordingly, in the object store, the actual storage location of the video in the object store can be determined after knowing the name and path of the video.

Specifically, the user performs an operation on an object displayed in a directory form in the device interface provided with the operating system, and based on the operation, the object name and the path of the object can be determined, so as to determine the storage position of the target object.

In the present application, the object store is described by using a distributed object database as a name.

The bucket is a logic concept, which is equivalent to a top-level directory, and the object storage achieves the aim of isolation by creating different buckets for different services/users.

Specifically, the obtaining the user request includes:

Specifically, an operating system is run in the processor, and based on the operating system, each object is mounted on the operating system.

Wherein the user performs an operation on an object mounted on the operating system through an input device connected to the processor.

Among these operations are various operations that may be performed on an object, such as copying a file, renaming a directory, querying a file, deleting a file, modifying a file, or adding a new file, etc.

It should be noted that, part of the operations relate to operations performed on metadata of the object, part of operations performed on the object data itself, and accordingly, in this embodiment, processing performed on a request for performing operations on metadata of the object is performed, and the operations performed on the object data itself are responded by a distributed object database, which will be explained in the following embodiments, which will not be described in detail in this embodiment.

The user request is a request related to processing metadata of the target object, and a request response mode not related to metadata processing will be described in the following embodiments, which will not be described in detail in this embodiment.

Step S102: and processing the metadata of the target object in response to the user request.

Wherein the metadata contains attribute information of the target object.

Wherein, since the user request is an operation performed on the metadata of the target object, the metadata database responds to the user request and processes the metadata of the target object.

It should be noted that, in the following embodiments, the detailed lifetime of the process of processing the metadata of the target object is performed, which is not described in detail in this embodiment.

As shown in fig. 3, an application scenario of the method for processing a distributed object in this embodiment is shown, where the application scenario includes a distributed object database mini 301, a metadata database dis302, and a client303.

The distributed object database is an open source object storage database minio, and minio is an object storage service based on Apache License v2.0 open source protocol. It is compatible with amazon S3 cloud storage service interface, suitable for storing large capacity unstructured data such as pictures, videos, log files, backup data and container/virtual machine images, etc., while an object can be of arbitrary size, varying from a few kb (kilobytes) to a maximum of 5T (terabytes).

The metadata database adopts a Redis database, and is a high-performance key-value database which is completely open-source and complies with BSD (Berkeley Software Distribution, berkeley software suite) protocol.

Wherein, the Redis database has the following characteristics: the Redis supports the persistence of data, can store the data in the memory in the disk, can load again and use while restarting; redis not only supports simple key-value type data, but also provides storage of list (list), set (set), zset (ordered set), hash (hash) and other data structures; redis supports backup of data, i.e., master-slave mode data backup.

Since Redis is a database storing kv key value pairs, is an implementation of a metadata database, a corresponding value can be obtained through a key, or the value is stored through a key, the key is unique, if the value is the content of a character string type, the key value pairs are in a Redis calling string type, if the value is the content format of an array/queue, the type is called a list type in the Redis, and the metadata database in the application adopts two types of data types, namely the string type and the list type of the Redis.

The content of the data types in the metadata database involved in the processing of the distributed objects will be described in detail in the following embodiments, which will not be described in detail in this embodiment.

Specifically, all data of the service are stored in a distributed object database, and after the file name is hashed, the data is determined to be stored in which nodes, and generally, EC (Erasure Coding) is used to ensure that the data is high in availability.

Wherein, the metadata database stores mapping information of all data of the distributed object, and particularly adopts a redis key value pair.

Wherein the client is set in the operating system and modified on the open source rclone. Wherein the client is configured to request a metadata-related request from the metadata database for metadata in the distributed object database.

In summary, the method for processing a distributed object provided in this embodiment includes: obtaining a user request, wherein the user request is a request for processing a target object mounted on an operating system in a distributed object; and responding to the user request to process metadata of the target object, wherein the metadata comprises attribute information of the target object. When a user request for processing a target object mounted on an operating system in a distributed object is obtained, the metadata database responds to the target request based on the attribute information of the target object, the metadata of the target object is processed, the process does not need to process the data of the target object, and the process of re-storing or uploading the file object in the distributed database is not needed, so that the storage space, the bandwidth and the like of the distributed object database are saved.

As shown in fig. 4, a flowchart of an embodiment 2 of a method for processing a distributed object according to the present application includes the following steps:

step S401: obtaining a user request;

step S401 is identical to the corresponding steps in embodiment 1, and is not described in detail in this embodiment.

Step S402: determining a preset key value pair based on the storage position of the target object in the user request;

wherein the user request includes any one of a copy file request and a directory rename request.

Wherein the metadata database is a redis database.

Specifically, the redis database stores key value pairs of each object in the distributed object database.

When the target object in the user request is a file, the file is mounted on the operating system in a barrel, the storage position of the target object in the distributed object database is determined, and correspondingly, a preset key value pair corresponding to the storage position is determined in the metadata database.

The preset key value pair corresponding to the storage position comprises a key value pair corresponding to the target file object or a key value pair related to a catalog to which the target file object belongs.

The directory to which the target file object belongs is a bucket directory to which the target file object belongs.

In a specific implementation, the open source in the user request includes a name of the target object, a storage position of the target object in the distributed object database is determined in the metadata database based on the name, and then a corresponding preset key value pair is determined based on the storage position.

The metadata database utilizes the string type and list type supported by redis itself, and realizes three formats needed by business based on the two types.

The first format is string type inside redis for storing copy reference counts.

If the files in a certain bucket directory are copied, a string format for counting is generated, wherein the key is generally a randomly generated character string, the character strings adopted by different data objects are different, and the value is used for indicating the number of the files of the data objects which are simultaneously pointed to the object storage.

Examples: (random-str, 2), key is random-str, representing a randomly generated string; value is 2, which indicates that there are two files pointing to the data object stored by the object at the same time;

the second format is a string type inside redis for storing metadata such as files/video/pictures, and is a string type.

In the string type format, the key is used to identify the storage address of the storage object, specifically, a certain file in a certain directory under a certain bucket, and the value includes four fields including the actual name, modification time, count key and other metadata information of the object in the storage. Wherein, the counting key is a data object stored with the same object pointed to by the counting files at the same time.

For example, one metadata format is as follows: (/ bucket1/dir1/obj1, { "name": "/bucket1/dir1/obj1", "model": 81922211, "count-ref": "," attr ":" }), wherein key is/bucket 1/dir1/obj1, representing a directory of dir1 under bucket1, there is a file of obj1 in the directory, the value, when saved in redis, is of the string type, and when client or service is in use, will convert this value into a mapped type, there are four fields in it, name represents the actual name in the object store, and the actual name is: the value of the modification time is represented by the value of the modification time, the value of the modification time is represented by the reference count key in the first format, the value of the modification time is represented by the character string of the modification time, the value of the modification time is represented by attr, the modification time is represented by the value of the modification time, the modification time is represented by the character string of the modification time, the modification time is represented by the value of the modification time, and the modification time is represented by the character string of the modification time, the modification time is represented by the value of the modification time, and the modification time is represented by other metadata information, such as the modification time.

If a certain file obj1 is copied and pasted, the example becomes: (/ bucket1/dir1/obj1, { "name": "/bucket1/dir1/obj1", "model": 81922211, "count-ref": "random-str", "attr": "}"), and/bucket 1/dir1/obj1 indicate names within the distributed object store, the random-str pointing to a string in the first format that references a count key.

The third format is list type inside redis for storing files/directory lists inside the directory. Specifically including the bucket directory and enumerating objects in the directory

For example, (/ bucket1/dir1/, { obj1 }), where/bucket 1/dir 1/represents the directory dir1 within the bucket ending with "/", which currently has an object obj1 within.

It should be noted that, in the present application, the response to the object processing requested by the user based on the metadata database is based on the above three formats.

For example, the object file object is obj1, and the storage position of the object file object is determined to be/bucket 1/dir1/obj1 in the distributed object database, the corresponding preset key value pair is (/ bucket1/dir1/obj1, { "name": "/bucket1/dir1/obj1", "model": 81922211, "count-ref": "," attr ":" }), and the corresponding preset key value pair adopts string type.

Step S403: and updating the key value of the preset key value pair based on the request content requested by the user.

Wherein the preset key value pair contains at least one item of metadata of the target object.

Wherein the user request is analyzed for user requested content, such as copy files, target renames, etc.

The response to the user request content may be implemented by processing metadata, and no operation is required to be performed on the object data in the distributed object database, so in this embodiment, the response to the user request is implemented by updating the key value of the preset key value pair corresponding to the target object based on the user request content.

The following embodiments will be described in detail with respect to the response process of renaming the copy file and directory, which will not be described in detail in this embodiment.

In summary, the method for processing a distributed object provided in this embodiment includes: determining a preset key value pair based on the storage position of the target object in the user request; and updating the key value of the preset key value pair based on the request content of the user request, wherein the preset key value pair comprises at least one item of metadata of the target object. In this embodiment, a corresponding preset key value pair is determined based on a storage position of a target object requested by a user in a distributed object database, and a key value of the preset key value pair is updated based on content requested by the user, so that response to the content requested by the user is realized by processing metadata of the target object in a metadata database, and operation on object data in the distributed object database is not required, and data migration or file re-uploading is not required to the distributed object database.

As shown in fig. 5, a flowchart of an embodiment 3 of a method for processing a distributed object according to the present application includes the following steps:

step S501: obtaining a user request;

step S502: determining a preset key value pair based on the storage position of the target object in the user request;

steps S501-502 are identical to the corresponding steps in embodiment 2, and are not described in detail in this embodiment.

Step S503: inserting a first key value pair in a metadata database;

the first key value pair comprises a storage position of the target object after being copied in the distributed object database.

If the user request is a copy file request, after determining a target object corresponding to the copy file request, determining a preset key value pair corresponding to the target object in the metadata database.

Wherein the preset key value pair is the storage position of the copied target object in the distributed object database.

The target request is a copy file request that is aimed at copying and pasting an object stored in the distributed object database from one storage location to another.

In this embodiment, a first key pair is inserted into the metadata database, where the first key pair includes a storage location of the target object after copying in the distributed database and other related attribute information.

Since the metadata of the object is stored in the metadata database, when operations such as accessing the object in the distributed object database are performed, the metadata of the object in the metadata database is accessed, so in this embodiment, the object data in the distributed object database is not copied, but a first key value pair is added in the metadata database, so as to create a corresponding entry from the metadata angle, and avoid repeated storage in the distributed object database.

The first key value pair is different from the preset key value pair only in storage position.

The storage path of the target object in the distributed object database is/socket 1/dir2/obj1, the new path is/socket 1/dir2/obj2, and correspondingly, a key value pair is inserted in the metadata database, and the storage path in the key value pair is/socket 1/dir2/obj2

Step S504: and adding one to the counting information in the preset key value pair and the counting information in the first key value pair respectively.

The preset key value pair contains counting information, the counting information can be represented by a counting key value pair, wherein the key can be a random character, and the value represents the number of files pointing to the same object.

Since the current request is to copy the file object, the number of files pointing to the same object in the metadata database is two.

Correspondingly, the count information of the preset key value pair and the first key value pair is respectively increased by one.

Specifically, the count key value pair is (random, 2), where key is a random string random1, and value is 2, indicating that two files point to objects stored in the same underlying object.

If the file is first copied, inserting a counting key value pair after the key value pair of the file object, wherein the counting key value pair is value 2; if the file is not first copied, 1 is added to the value of the count key value pair of the file object.

In an implementation, if the target object has already been copied once, the count information in the preset key pair is changed from 2 plus 1 to 3, and correspondingly, the count information in the first key pair is also changed to 3.

For example, copy file/bucket 1/dir1/obj1 to/bucket 1/dir2/obj2, insert a file metadata key value pair (/ bucket1/dir2/obj2, { "name": "/bucket1/dir1/obj1", "model": 81922211, "count-ref": "random1", "attr": "}") in the metadata database: after the character string in the value is converted into a mapping type structural body, the value corresponding to the name field is the name of the file in the object storage actually, and the value corresponding to the count-ref field is as follows: random1, a character string of keys of the counting key value pair; after the key value pair of which the metadata key is/socket 1/dir1/obj1 is modified, and the character string in the value is converted into a mapping type structure, the value of the counter-ref field in the value is changed into random1, and finally the content of the key value pair value is (/ socket 1/dir1/obj1, { "name": "/socket 1/dir1/obj1", "model": ", 81922211": "counter-ref": "random1", "attr": "}), and the character string in the dis is saved; the key is the directory name, here the key/dir 2/, the value is the file or directory list in the directory, and only one element, namely obj2, is in the list.

In summary, the method for processing a distributed object provided in this embodiment includes: when the user request is a file copying request, a first key value pair is inserted into a metadata database, wherein the first key value pair comprises a storage position of the target object after being copied in a distributed object database; and adding one to the count information in the preset key value pair and the count information in the first key value pair respectively, wherein the count value represents the number of the files pointing to the same object. In this embodiment, if the user request is to copy the object in the distributed object database, a first key value pair corresponding to the storage position of the target object in the distributed object database after copying is inserted into the metadata database for the object, and one is added to the count information in the first key value pair and the preset key value pair to characterize that the number of files pointing to the same object is two after the target object is copied at the time, so that repeated storage of the objects in the distributed object database is not required, and the storage space in the distributed object database is saved.

As shown in fig. 6, a flowchart of an embodiment 4 of a method for processing a distributed object according to the present application includes the following steps:

Step S601: obtaining a user request;

step S601 is identical to the corresponding steps in embodiment 2, and is not described in detail in this embodiment.

Step S602: obtaining key value pairs of at least one object contained in the renamed directory;

wherein the user request is a directory rename request.

Wherein the target object is one or more objects under a certain directory in the bucket.

It should be noted that, because the content such as a file/video is associated with its name in the storage location of the distributed object storage cluster, once its name and path are determined, its location (linux node) in which the object storage cluster is stored is also determined. When the bucket is mounted on a file system of a user by rclone software, the bucket is expressed as a disk or a directory, when the user renames the directory, because the object storage has no concept of the directory, when the position of the file is calculated, a hash value of the complete path name of the file is calculated in the distributed object database, the complete path such as "/bucket 1/program A first period. Therefore, in the prior art, the file needs to be copied to a new location, but in the present embodiment, only the metadata of the file in the metadata database needs to be processed, and the file itself does not need to be copied to the new location.

Specifically, when renaming the directory, renaming is required for all objects under the directory.

Wherein, key value pairs corresponding to one object or a plurality of objects contained in the renamed directory are determined.

Specifically, the user request is a key of a key value pair obtained by splicing the old directory name and the belonging bucket based on renaming operation by the operating system, and content included in the user request obtained by the first device executing the embodiment includes the key and directory renaming information.

For example, if the user renames dir1 to be new-dir1, assuming that there are three files obj1, obj2, obj3 under the directory, the operating system receives the user renaming request to obtain an old directory name dir1 and a new directory name new-dir1, and then, based on the old directory name and the bucket concatenation, a key is (/ bucket1/dir1/, { obj1, obj2, obj3 }). Wherein the metadata database traverses the value corresponding to the list type key value based on the key value key, and determines key value pairs of three objects from the obj1 value ibj 3.

Step S603: and updating the target field information in the key value pair of the at least one object according to the renamed name.

Wherein the target field information characterizes a storage location of the at least one object in a distributed object database.

After obtaining the renamed new directory name, updating the target field information in the preset key value pair, wherein the target field information is the old directory name before updating and is the new directory name after updating.

Specifically, the corresponding key value pairs in the metadata database in the directory are modified, and the key value pairs of all the objects related to the directory are modified.

As with obj1 above, the key value pair that it had previously stored in the metadata service was (/ bucket1/dir1/obj1, { "name": "/bucket1/dir1/obj1", "modtime":81922211 "," count-ref ":", "attr": "}), and the modified key value pair was (/ bucket1/new-dir1/obj1, {" name ":"/bucket1/dir1/obj1"," modtime ": 81922211", "count-ref": "" "; the obj2 is similar, the key value is (/ bucket1/dir1/obj2, { "name": "/bucket1/dir1/obj2", "modtime":81922211, "count-ref": "," attr ":"), the modified key value pair is (/ bucket1/new-dir1/obj2, { "name": "/bucket1/dir1/obj2", "modtime":81922211 "," count-ref ":", "attr": "}). Wherein, the key value pair of the metadata of each file in the directory in the redis, the directory hierarchy structure represented by the key, and the value corresponding to the corresponding name field in the value represent the name in the object storage.

It should be noted that, the list type key value pair of each object needs to be modified to a new directory key value pair.

For example, the key value pair (/ socket 1/dir1/, { obj1, obj2, obj3 }) with list type key being/socket 1/dir 1/is changed to (/ socket/1/new-dir 1/, { obj1, obj2, obj3 }), and the value of this key value pair is unchanged.

In summary, the method for processing a distributed object according to this embodiment includes: obtaining at least one object contained in the renamed directory; and updating the key value information in the key value pair of the at least one object according to the renamed name, wherein the key value pair at least comprises the key value information. In this embodiment, renaming is performed on the directory in the bucket, and only the meta key value of the object related to the directory in the metadata database needs to update the target field information to a new directory content, so that the processing is not required on the object data in the distributed object database, and the storage space in the distributed object database is saved.

As shown in fig. 7, a flowchart of an embodiment 5 of a method for processing a distributed object according to the present application includes the following steps:

Step S701: obtaining a user request;

step S701 corresponds to the corresponding step in embodiment 1, and is not described in detail in this embodiment.

Step S702: determining a storage position of a target object in the user request in a distributed object database;

when the user request is a delete file request, the file in the distributed object database and the corresponding key value pair in the metadata database need to be processed.

After obtaining the user request forwarded by the operating system, the storage position of the target object of the request in the distributed object database is queried in the metadata database.

Specifically, after receiving the file deletion request, the operating system splices the file name of the target deletion with the barrel and the directory to which the file name belongs to obtain a key of a redis key value, and sends the key of the redis key value and the deletion request to the metadata database. The metadata database obtains metadata of a key value pair corresponding to the key value, and after characters in the value are converted into a structure body of a black type, name fields in the structure body are used for obtaining names in the distributed object database where the file is located.

For example, the user requests to delete the file name obj1, specifically, the file name/bucket 1/dir1/obj1, the operating system will stitch into a key of the key value of redis based on the file name and bucket and directory, namely: the key (/ socket 1/dir1/obj 1) requests a metadata database, the metadata database determines key value pairs corresponding to the key, specifically (/ socket 1/dir1/obj1, { "name": "/socket 1/dir1/obj1," "model": "(81922211)," count-ref ":", "attr": "}"), and after converting the character strings in the value into a mapping type structure, the name field is used for obtaining the name of the distributed object database where the file is located: and (3) obtaining the real name of the file in the distributed object database as the box 1/dir1/obj1.

Step S703: requesting, by an operating system based on the storage location control, deletion of the target object from the distributed object database;

specifically, the metadata database stores names in the distributed object database where the target deleted file is determined, and then the stored names are fed back to the operating system, so that the operating system requests the distributed object storage database to delete the corresponding file, and feeds back the deleting result.

If the deletion is successful, continuing the next step, if the deletion is failed, returning a failure result, and ending the flow.

Step S704: and deleting the preset key value pair corresponding to the target object.

And deleting the preset key value pair corresponding to the determined target object.

If the file of the target object is a copied file, and correspondingly, the preset key value pair corresponding to the target object also has count information, the key value pair of other backup files is updated.

Specifically, the preset key value pair of the target object that is alive is (/ bucket1/dir1/obj1, { "name": "/bucket1/dir1/obj1", "modtime":81922211 "," count-ref ":", "attr": "}"), if the file is copied and pasted before, the corresponding count-ref field in the value is not empty, the field value of the count-ref is reduced by 1, and the preset key value pair corresponding to the target object is deleted, and if the file is not copied and pasted before, the corresponding count-ref field in the value is directly deleted.

In summary, the method for processing a distributed object provided in this embodiment, where the user request is a delete file request, includes: determining a storage position of a target object in the user request in a distributed object database; requesting, by an operating system based on the storage location control, deletion of the target object from the distributed object database; and deleting the preset key value pair corresponding to the target object. In this embodiment, after determining the storage position of the target object in the distributed object database in the user request, the metadata database controls the operating system to request the distributed object database to delete the target object, after deleting successfully, the preset key value pair corresponding to the target object in the metadata database is deleted, and when deleting the file in the distributed object database, the corresponding metadata in the metadata database needs to be processed to ensure the unification of the metadata database and the distributed object database.

As shown in fig. 8, a flowchart of an embodiment 6 of a method for processing a distributed object according to the present application includes the following steps:

step S801: obtaining a user request;

step S801 is identical to the corresponding steps in embodiment 1, and is not described in detail in this embodiment.

Step S802: and determining a storage position of a target object in the distributed object database in the user request, and feeding back to an operating system, so that the operating system requests the target object from the distributed object database based on the storage position.

Wherein the user request is a query file request, and queries a target object in the distributed object database.

Specifically, the user requests to inquire obj1, the operating system determines/bucket 1/dir1/obj1 based on the file obj1 and the bucket and directory to which the file belongs, adopts the/bucket 1/dir1/obj1 as a key, and the metadata database inquires a key value pair corresponding to the key, specifically, the key value pair is the metadata of (/ bucket1/dir1/obj1, { "name": "/bucket1/dir1/obj1", "model": 81922211 "," count-ref ":", "attr": "}), converts a character string in a value into a structure of a mapping type, and then obtains the name of the distributed object database where the file is located through a name field: socket 1/dir1/obj1.

After the metadata database determines the name corresponding to the target object, the name characterizes the storage position of the target object in the distributed object database, and the storage position is fed back to the operating system.

And after the operating system receives the storage position, requesting the target file object from the distributed object database based on the storage position.

The operation system determines the name of the file in the distributed object database based on the metadata of the query file stored in the metadata database, and the operation information queries the object in the distributed object database based on the name of the file in the distributed object database obtained by the query.

If the object data needs to be downloaded, the data of the object can be obtained from the distributed object database to the cache area of the operating system.

In summary, the method for processing a distributed object provided in this embodiment, where the user request is a query file request, includes: and determining the storage position of the target object in the distributed object database in the user request, and feeding back to an operating system so that the operating system requests the target object from the distributed object database based on the storage position. In this embodiment, based on a user request to query a file, the metadata database queries the storage location of the target file object in the distributed object database and feeds the storage location back to the operating system, and the operating system requests the target file object from the distributed object database based on the storage location, so that it is ensured that any object data in the distributed object database can be queried through the metadata database.

As shown in fig. 9, a flowchart of an embodiment 7 of a method for processing a distributed object according to the present application includes the following steps:

step S901: obtaining a user request;

step S901 is identical to the corresponding steps in embodiment 1, and is not described in detail in this embodiment.

Step S902: after uploading the newly added file to a distributed object database based on an operating system, obtaining a storage position of the newly added object in the distributed object database;

wherein the user request is a newly added file.

The user copies a non-mount point file to a certain directory on the operating system, and the operating system generates a user request to request a new file.

The operating system firstly uploads the newly added file to the distributed object database, and after the distributed object database is fed back with the uploading success information, the operating system requests to the metadata database to create metadata information (key value pair) corresponding to the newly added file.

Specifically, after the operating system uploads the newly added file to the distributed object database, the distributed object database feeds back a storage position (storage name) where the newly added file is mounted in the distributed object database in an object form, the operating system sends the storage position to a metadata database, and the metadata database creates metadata information of the newly added object based on the storage name.

Step S903: and creating metadata information corresponding to the new object based on the storage position.

Wherein, the metadata database creates metadata information for the newly added object after obtaining the storage position of the newly added object in the distributed object database.

Specifically, the newly created metadata information includes string type key value pairs and list type directory metadata key value pairs.

The method comprises the steps that a user requests to newly add a file obj1 into a bucket mounting catalog/bucket 1/dir1, a distributed object database is successfully fed back and stored, an operating system requests to create metadata information from a metadata database based on the storage position of a corresponding object in the distributed object database, the corresponding object is stored in the distributed object database, the metadata database inserts a key value pair and a newly created list type target metadata key value pair, the inserted key value pair is (/ bucket1/dir1/obj1, { "name"/bucket 1/dir1/obj1"," model ": 81922211", "attr": "), the key is a file path name, and after a character string in a value is converted into a mapping type structure, a value corresponding to a name field is the actual name of the distributed object; the list metadata key value pair of the newly built list type is (/ socket/dir 1/, { obj1 }), the key is the name of the list, the value is the file or list of the list in the list, and only one element, namely obj1, is in the list.

In summary, the method for processing a distributed object according to this embodiment, where the user request is a new file, includes: after uploading the newly added file to a distributed object database based on an operating system, obtaining a storage position of the newly added object in the distributed object database; and creating metadata information corresponding to the new object based on the storage position. In this embodiment, after the operating system requests to upload the new file to the distributed object database successfully, the operating system requests to create metadata information from the metadata database based on the storage position of the new file in the distributed object database as the new object, so as to provide a basis for processing the object in the distributed object database based on the metadata information in the metadata database.

Corresponding to the above embodiment of the method for processing a distributed object applied to a first device having metadata, the present application further provides an embodiment of a method for processing a distributed object applied to a second device provided with an operating system.

As shown in fig. 10, a flowchart of an embodiment 8 of a method for processing a distributed object according to the present application is applied to a second device provided with an operating system, and includes the following steps:

Step S1001: obtaining a user request;

It should be noted that the method in this embodiment is specifically implemented by a client applied to an operating system, and the relationship between the client and the distributed object database and the metadata database are shown in fig. 3.

Specifically, the client program has the following specific functions: firstly, acquiring the real name of a file in a distributed object database from the metadata database; 2 is that the directory renaming and file pasting copy operation does not request the distributed object database, and instead requests the metadata database; and 3, uploading the file, namely uploading data to a distributed object database, and creating metadata to a metadata database after success.

Wherein the second device sets/connects an input means based on which a user inputs an operation of processing an object in a bucket mounted in an operating system.

The specific method of execution of the client will be described in detail in the following examples.

Specifically, the step S1001 includes:

Step S10011: determining a target object corresponding to the triggering operation;

the user inputs a triggering operation through the input device, displays content in a display screen of the second device, and determines a target object corresponding to the triggering operation.

Wherein, the content displayed in the display screen is an object displayed in a catalog form, a barrel can comprise a plurality of target objects, and a catalog object comprises a plurality of file objects.

Specifically, the target object is one or more objects mounted in a bucket of the operating system.

Step S10012: obtaining a storage position of the target object in a distributed object database based on a barrel to which the target object belongs and the name of the target object;

the name of the target object can be determined based on the operation of the user for processing the target object, and further, the storage position of the target object in the distributed object database is determined based on the bucket or the catalog and the bucket to which the target object belongs.

For example, it is determined that the target object is a file object, its name is obj1, its directory is dir1, its bucket is socket 1, and the three are spliced to obtain that the storage location of the file object in the distributed object database is/socket 1/dir1/obj1.

Step S10013: and obtaining a user request based on the request triggered by the triggering operation and the storage position of the target object.

The request triggered by the triggering operation is received, and the request may be various operations performed on the objects in the distributed object database, such as deleting a file, modifying a file, adding a file, and the like, which are not limited in this embodiment.

After the storage position of the target object is determined, the storage position is combined with a request set triggered by a triggering operation, and a user request containing request content and the request object is obtained.

Step S1002: analyzing the user request to obtain the type of the user request;

wherein the types include a first type related to metadata of the object and a second type different from the first type.

And analyzing the content requested by the user to obtain the type of the user request.

In the application, the type of the user request is analyzed according to whether the request content is related to the metadata of the object in the distributed object database.

Wherein if the requested content is related to metadata of the object, it is determined that the user request is of a first type, or the requested content is unrelated to metadata of the object, and only processing of the object data itself is involved, it is determined that the user request is of a second type.

For example, the first type of user request includes at least a copy file request, a query file request, a delete file request, a directory rename request; the second type of user request includes at least an add file request and a modify file request.

Step S1003: if the type of the user request is a first type, requesting a metadata database based on the user request, so that the metadata database responds to the user request to process the metadata of the target object;

the metadata stored in the metadata database is metadata corresponding to each distributed object in the distributed object database.

And when the type of the user request is the first type and is related to the metadata of the object, the user request is sent to a metadata database, so that the metadata database responds to the user request and processes the metadata of the target attribute corresponding to the user request.

In the process of responding to the user request, please refer to the foregoing method embodiment applied to the first device, which is not described in detail in this embodiment.

Step S1004: and if the type of the user request is a second type, requesting the distributed object database based on the user request, so that the distributed object database responds to the user request to process the target object.

And when the type of the user request is the second type and is irrelevant to the metadata of the object, the user request is sent to the distributed object database, so that the distributed object database responds to the user request and processes the target object corresponding to the user request.

For example, if the user request is a modify file request, the modify file does not involve modifying metadata of the file object, the modified file and the user request are uploaded to the distributed object database, so that the target object in the distributed object database is updated from the file before modification to the file after modification.

In summary, the method for processing a distributed object provided in this embodiment includes: obtaining a user request, wherein the user request is a request for processing a target object mounted on an operating system in a distributed object; analyzing the user request to obtain the type of the user request, wherein the type comprises a first type related to metadata of an object and a second type different from the first type; if the type of the user request is a first type, requesting to a metadata database based on the user request, so that the metadata database responds to the user request to process the metadata of the target object, wherein the metadata stored in the metadata database are metadata corresponding to each distributed object in a distributed object database; and if the type of the user request is a second type, requesting the distributed object database based on the user request, so that the distributed object database responds to the user request to process the target object. In this embodiment, after the second device provided with the operating system obtains the user request, the type of the user request is determined, if the type belongs to the first type related to the metadata of the object, the metadata database is requested to process the metadata of the target object, and if the type belongs to the second type unrelated to the metadata of the object, the distributed object database is requested to process the target object, so that different response modes for different types of requests are realized, and the access pressure of the distributed object database is reduced.

Corresponding to the embodiment of the processing method of the distributed object provided by the application, the application also provides an application scene embodiment of the processing method of the distributed object.

As shown in fig. 11, a schematic view of an application scenario of a method for processing a distributed object provided by the present application, where the scenario includes the following structures: an operating system 1101, a metadata database 1102, and a distributed object database 1103. The operating system, in particular the personal PC (personal computer) file system, comprises a K disc in which a client program is arranged, which client program generates user requests based on received input information and performs interaction procedures with a metadata database and a distributed object database based on different user requests.

The scenario involves a plurality of interaction flows 1-4, including:

scheme 1: importing metadata;

wherein the distributed object database imports metadata of each object therein into the metadata database.

Scheme 2: acquiring target list/object metadata;

the client program obtains corresponding target list/object metadata from the metadata database based on a user request, and specifically may be string key value pairs and list key value peering.

Scheme 3: renaming the catalogue;

wherein the client program controls the metadata database to update the key value pairs of the object based on the renamed information such that the names of the updated key value pairs and the renamed key value pairs are known.

Scheme 4: uploading the object.

The client program uploads the file to the distributed object database, and the distributed object database imports the storage position of the file object into the metadata database through the client program, and the metadata database creates metadata of the file object.

Corresponding to the embodiment of the processing method of the distributed object provided by the application, the application also provides an embodiment of a device applying the processing method of the distributed object.

As shown in fig. 12, a schematic structural diagram of an embodiment 1 of a distributed object processing apparatus according to the present application is applied to a first device, where the first device is provided with a metadata database, and the apparatus includes the following structures: a first obtaining module 1201 and a response module 1202;

wherein, the first obtaining module 1201 is configured to obtain a user request, where the user request is a request for processing a target object that is installed on an operating system with a distributed object;

The response module 1202 is configured to process metadata of the target object in response to the user request, where the metadata includes attribute information of the target object.

Optionally, the first obtaining module is specifically configured to:

Optionally, the user request includes any one of a copy file request and a directory renaming request, and the response module includes:

a determining unit, configured to determine a preset key value pair based on a storage location of a target object in the user request;

and the updating unit is used for updating the key value of the preset key value pair based on the request content of the user request, wherein the preset key value pair comprises at least one item of metadata of the target object.

Optionally, when the user request is a file copying request, the updating unit is specifically configured to:

Optionally, when the user request is a directory renaming request,

the determining unit is specifically configured to:

correspondingly, the updating unit is specifically configured to:

Optionally, the user request is a delete file request, and the response module is specifically configured to:

and deleting the preset key value pair corresponding to the target object.

Optionally, the user request is a file inquiry request, and the response module is specifically configured to:

In the functional explanation of each component structure in the processing device of the distributed object provided in this embodiment, please refer to the explanation in the foregoing corresponding method embodiment, and the description of this embodiment is omitted.

In summary, in the processing device for a distributed object provided in this embodiment, when a user request for processing a target object mounted on an operating system by using a distributed object is obtained, based on whether the user request is for processing attribute information of the target object, a metadata database responds to the target request, and processes metadata of the target object, and the process does not need to process data of the target object itself, and does not need to re-store or upload a file object in the distributed database to the distributed database, thereby saving storage space and bandwidth of the distributed object database.

As shown in fig. 13, a schematic structural diagram of an embodiment 2 of a distributed object processing apparatus according to the present application is applied to a second device, where the second device is provided with an operating system, and the apparatus includes the following structures: a second obtaining module 1301, an analyzing module 1302, and a requesting module 1303;

The second obtaining module 1301 is configured to obtain a user request, where the user request is a request for processing a target object that is installed on an operating system with a distributed object;

wherein the analysis module 1302 is configured to analyze the user request to obtain a type of the user request, where the type includes a first type related to metadata of the object and a second type different from the first type;

the request module 1303 is configured to request, if the type of the user request is a first type, a metadata database based on the user request, so that the metadata database processes metadata of the target object in response to the user request, where metadata stored in the metadata database is metadata corresponding to each distributed object in the distributed object database; and if the type of the user request is a second type, requesting the distributed object database based on the user request, so that the distributed object database responds to the user request to process the target object.

In summary, the processing apparatus for a distributed object provided in this embodiment is applied to a second device provided with an operating system, and after obtaining a user request, determines a type of the user request, if the type belongs to a first type related to metadata of an object, requests a metadata database to process metadata of the target object, and if the type belongs to a second type unrelated to metadata of the object, requests the distributed object database to process the target object, thereby implementing different response modes for different types of requests, and reducing access pressure of the distributed object database.

Corresponding to the embodiment of the control method provided by the application, the application also provides the electronic equipment and the readable storage medium corresponding to the control method.

Wherein, this electronic equipment includes: a memory, a processor;

wherein the memory stores a processing program;

the processor is configured to load and execute the processing program stored in the memory, so as to implement the steps of the control method according to any one of the above.

The implementation control method of the electronic equipment is realized by referring to the embodiment of the control method.

Wherein the readable storage medium has stored thereon a computer program which is invoked and executed by a processor to implement the steps of the control method as set forth in any one of the preceding claims.

The computer program stored in the readable storage medium is executed to implement the control method, and the embodiment of the control method is referred to.

In the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, and identical and similar parts between the embodiments are all enough to refer to each other. The device provided in the embodiment corresponds to the method provided in the embodiment, so that the description is simpler, and the relevant points refer to the description of the method.

The previous description of the provided embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features provided herein.

Claims

1. A method of processing a distributed object, applied to a first device provided with a metadata database, the method comprising:

2. The method of claim 1, wherein the obtaining the user request comprises:

3. The method of claim 1, wherein the user request comprises any one of a copy file request, a directory rename request, and wherein processing metadata of the target object in response to the user request comprises:

4. A method according to claim 3, wherein updating the key value of the preset key value pair based on the request content of the user request when the user request is a copy file request comprises:

5. The method of claim 3, wherein when the user request is a directory renaming request, determining a preset key pair based on a storage location of a target object in the user request comprises:

6. The method of claim 1, wherein the user request is a delete file request, and wherein the processing metadata for the target object in response to the user request comprises:

and deleting the preset key value pair corresponding to the target object.

7. The method of claim 1, wherein the user request is a query file request, and wherein the processing metadata for the target object in response to the user request comprises:

8. A method of processing a distributed object, for application to a second device provided with an operating system, the method comprising:

9. A processing apparatus for a distributed object, applied to a first device provided with a metadata database, comprising:

10. A processing apparatus for a distributed object, applied to a second device provided with an operating system, comprising: