CN111966867A - Object deleting method, data processing method and device - Google Patents
Object deleting method, data processing method and device Download PDFInfo
- Publication number
- CN111966867A CN111966867A CN202010831443.0A CN202010831443A CN111966867A CN 111966867 A CN111966867 A CN 111966867A CN 202010831443 A CN202010831443 A CN 202010831443A CN 111966867 A CN111966867 A CN 111966867A
- Authority
- CN
- China
- Prior art keywords
- metadata
- deleted
- deleting
- determining
- data processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 70
- 238000003672 processing method Methods 0.000 title claims abstract description 24
- 238000012545 processing Methods 0.000 claims abstract description 135
- 238000012217 deletion Methods 0.000 claims abstract description 106
- 230000037430 deletion Effects 0.000 claims abstract description 106
- 238000004891 communication Methods 0.000 claims description 24
- 238000004590 computer program Methods 0.000 claims description 16
- 230000008569 process Effects 0.000 claims description 15
- 230000001960 triggered effect Effects 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 230000009471 action Effects 0.000 description 3
- 238000013500 data storage Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 238000007726 management method Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000012512 characterization method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/901—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/907—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Library & Information Science (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention provides an object deleting method, a data processing method and a data processing device. The object deleting method comprises the following steps: determining at least one object to be analyzed in an object storage server; the metadata of the object in the object storage server is stored in a target metadata database, and a designated metadata field for representing whether the object is expired is preset in the target metadata database; for each object, when the object is judged to belong to an object to be deleted based on a set deletion condition, determining the object as the object to be deleted, and setting a field value of the designated metadata field of the object by using designated content; wherein, the designated content is content used for indicating the object is out of date; and deleting the object to be deleted in the object storage server. By the scheme, the accuracy of the processing result of the processing event related to the object statistics can be improved.
Description
Technical Field
The present invention relates to the field of data storage technologies, and in particular, to an object deletion method, a data processing method, and an apparatus.
Background
The object storage is a storage mode with high reliability, high availability, low cost and unlimited expansion, and is suitable for massive unstructured data storage.
Even though the cloud vendor is cheap for object storage, objects often become stale data that is not valuable over time, and therefore a large number of objects need to be deleted to save cost. In order to meet the deletion requirement, the cloud manufacturer provides an interface for setting the deletion condition for the user, so that the user can set the deletion condition according to the requirement of the user, and the object in the object storage server at the cloud end is deleted.
In the prior art, an object deletion method includes: and determining each object to be deleted belonging to the user in the object storage server by using the deletion condition set by the user, and further deleting the determined objects to be deleted in sequence.
However, after each object is determined, there is a time delay for deleting each object, which may have the following problems:
if a processing event related to object statistics occurs during the object deletion process, the processing result of the processing event may be inaccurate. For example: for the object capacity statistics on which charging is stored, the problem that an object which should be deleted originally is possibly counted exists, accuracy of the object capacity statistics is affected, and finally charging is inaccurate.
Disclosure of Invention
The embodiment of the invention aims to provide an object deleting method, a data processing method and a device, so as to improve the accuracy of a processing result of a processing event related to object statistics. The specific technical scheme is as follows:
in a first aspect, an embodiment of the present invention provides an object deleting method, including:
determining at least one object to be analyzed in an object storage server; the metadata of the object in the object storage server is stored in a target metadata database, and a designated metadata field for representing whether the object is expired is preset in the target metadata database;
for each object, when the object is judged to belong to an object to be deleted based on a set deletion condition, determining the object as the object to be deleted, and setting a field value of the designated metadata field of the object by using designated content; wherein, the designated content is content used for indicating the object is out of date;
and deleting the object to be deleted in the object storage server.
Optionally, before determining the object as the object to be deleted, the method further includes:
determining a preset protection condition;
judging whether the object meets the protection condition;
if the object is judged not to meet the protection condition, determining the object to be deleted; if so, the deletion process for the object is ended.
Optionally, the condition content of the deletion condition at least includes a first object range for representing a range to be deleted of the object;
the first object range is represented by a combination of operators and identified content, the operators including a match or a mismatch, the identified content including: prefix and/or label information.
Optionally, the condition content of the protection condition at least includes a second object range for representing a range to be deleted of the object;
the second object range is represented by a combination of operators and identified content, the operators including a match or a mismatch, the identified content including: prefix and/or label information.
Optionally, the determining the object as an object to be deleted includes:
adding the object identification of the object to a target list; the target list is used for recording a list of objects to be deleted;
the deleting the object to be deleted in the object storage server includes:
and reading the object identification from the target list, and deleting the object with the read object identification from the object storage server.
Optionally, the deleting the object to be deleted in the object storage server includes:
deleting the object to be deleted in the object storage server through an asynchronous deletion thread; wherein the asynchronous deleting thread is a thread running in a device with a load amount lower than a predetermined threshold.
In a second aspect, an embodiment of the present invention further provides a data processing method based on the object deleting method provided in the first aspect, where the method includes:
obtaining a data processing instruction; wherein the data processing instruction is a processing instruction of a processing event related to object statistics and related to a target user;
determining, from a target metadata database, that a field value of a specified metadata field is not specified content and an owner is metadata of the target user;
determining a processing result of the data processing instruction based on the determined metadata.
Optionally, the obtaining data processing instructions includes:
obtaining an object capacity statistical instruction;
the determining a processing result of the data processing instruction based on the determined metadata includes:
and calculating the total space occupation amount of the object represented by the determined metadata as a processing result corresponding to the object capacity statistical instruction.
In a third aspect, an embodiment of the present invention further provides an object deleting apparatus, including:
the object determining module is used for determining at least one object to be analyzed in the object storage server; the metadata of the object in the object storage server is stored in a target metadata database, and a designated metadata field for representing whether the object is expired is preset in the target metadata database;
an object processing module, configured to determine, for each object, an object to be deleted when it is determined that the object belongs to an object to be deleted based on a set deletion condition, and set a field value of the designated metadata field of the object with designated content; wherein, the designated content is content used for indicating the object is out of date;
and the object deleting module is used for deleting the object to be deleted in the object storage server.
In a fourth aspect, an embodiment of the present invention further provides a data processing apparatus based on the object deleting apparatus in the third aspect, including:
the instruction obtaining module is used for obtaining a data processing instruction; wherein the data processing instruction is a processing instruction of a processing event related to object statistics and related to a target user;
the metadata determining module is used for determining that the field value of the specified metadata field is not the specified content and the owner is the metadata of the target user from the target metadata base;
and the processing result determining module is used for determining the processing result of the data processing instruction based on the determined metadata.
In a fifth aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, where the processor and the communication interface complete communication between the memory and the processor through the communication bus;
a memory for storing a computer program;
a processor for implementing the method steps provided in the first aspect or the second aspect when executing the program stored in the memory.
In a sixth aspect, the present invention provides a computer-readable storage medium, in which a computer program is stored, and the computer program, when executed by a processor, implements the method steps provided in the first aspect or the second aspect.
The embodiment of the invention has the following beneficial effects:
in the object deleting method provided by the embodiment of the invention, a specified metadata field used for indicating whether an object is expired is added to a target metadata database in advance; thus, after at least one object to be analyzed is determined, for each object, when it is determined that the object belongs to an object to be deleted based on a set deletion condition, the object is determined as the object to be deleted, and a field value of the designated metadata field of the object is set using designated contents; wherein, the designated content is content used for indicating the object is out of date; and then deleting the object to be deleted in the object storage server. In the scheme, when the object is judged to belong to the object to be deleted, the field value of the designated metadata field of the object is set by using designated content, so that whether the object is overdue or not can be reflected in time. In this way, even if a processing event related to object statistics occurs during object deletion, since the target database records the expiration status of an object, the counted objects for the processing event may include only unexpired objects. Therefore, the method and the device can improve the accuracy of the processing result of the processing event related to the object statistics.
In addition, in the data processing method provided by the embodiment of the present invention, after the data processing instruction is obtained, it is determined from the target metadata repository that the field value of the specified metadata field is not the specified content and the owner is the metadata of the target user; further, a processing result of the data processing instruction is determined based on the determined metadata. Since it is determined from the target metadata database that the field value of the specified metadata field is not the metadata of the specified content and the owner is the target user during the data processing, and the processing result is determined by using the determined metadata, the objects counted for the processing event may include only the unexpired objects. Therefore, the accuracy of the processing result of the processing event related to the object statistics can be improved by the data processing method.
Of course, not all of the advantages described above need to be achieved at the same time in the practice of any one product or method of the invention.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other embodiments can be obtained by using the drawings without creative efforts.
Fig. 1 is a flowchart of an object deleting method according to an embodiment of the present invention;
FIG. 2 is another flowchart of an object deleting method according to an embodiment of the present invention;
FIG. 3 is a flow chart of a data processing method according to an embodiment of the present invention;
fig. 4 is a schematic overall schematic diagram of an object deleting method and a data processing method according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of an object deleting device according to an embodiment of the present invention;
FIG. 6 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention;
fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present invention;
fig. 8 is a schematic structural diagram of another electronic device according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In order to improve the accuracy of a processing result of a processing event related to object statistics, embodiments of the present invention provide an object deletion method, a data processing method, and an apparatus.
The following first introduces an object deletion method provided in an embodiment of the present invention.
The execution main body of the object deleting method provided by the embodiment of the invention can be an object deleting device. The object deleting apparatus may be applied to an electronic device, wherein the electronic device may be an object storage server, or a management server for managing the object storage server, or the like. The object storage server is a server for storing the object.
In addition, the objects mentioned in the embodiments of the present invention may be: unstructured data such as pictures, videos, or documents. For any object, it has metadata and an object identification. In particular, metadata of an object refers to the relevant description of the object, such as: the size of the object, the owner of the object, the last modification time of the object, etc.; the object identifier is a globally unique identifier for distinguishing the objects.
In addition, due to the fact that deletion is performed according to the requirements of the target user and the deletion time is uncertain, the object deletion provided by the embodiment of the invention belongs to asynchronous deletion. In the process of deleting the object belonging to the asynchronous deletion, the set deletion condition is used as the criterion for judging whether the object is deleted or not.
Specifically, the condition content of the set deletion condition may include at least two types of information, namely, a deletion mode and a first object range, which is used to indicate a range to be deleted of the object, in the condition content, that is, at least the deletion mode and the first object range are given to the deletion condition. The deleting mode may include a first deleting mode and a second deleting mode, the first deleting mode is a mode of deleting at a specified expiration time, and the second deleting mode is a mode of deleting when a specified expiration time is reached. The first deleting mode specifically refers to deleting at a specific time, and the second deleting mode specifically refers to deleting after a specified time length by taking the latest modification time of the object as the starting time.
As shown in fig. 1, an embodiment of the present invention provides an object deleting method, which may include the following steps:
s101, determining at least one object to be analyzed in an object storage server;
the metadata of the object in the object storage server is stored in a target metadata database, and a specified metadata field for indicating whether the object is expired is preset in the target metadata database.
When the trigger time for deleting the object of the target user is reached, at least one object to be analyzed may be determined from the objects stored in the object storage server, and then whether the at least one object is deleted or not may be analyzed. Wherein the at least one object is an object belonging to a target user.
Wherein the metadata of the object in the object storage server is stored in the target metadata base. Since the metadata of the object includes the owner of the object and the metadata of the object is stored in the target metadata repository, at least one object belonging to the target user stored in the object storage server may be determined as at least one object to be analyzed based on the target metadata repository.
In addition, a specified metadata field for indicating whether the object is expired is added in the target metadata database, and whether the object is expired can be reflected through the specified metadata field. Specifically, the specified metadata field may include an initial content and a specified content, the initial content is a content for indicating that the object is not expired, and the specified content is a content for indicating that the object is expired. Wherein the initial content and the designated content are different contents, illustratively, the initial content is null, and the designated content is non-null; alternatively, the initial content is 0 and the specified content is 1, and so on.
And, there may be various trigger occasions for determining at least one object to be analyzed in the object storage server. For example, if the deletion mode given by the deletion condition preset by the target user is the first deletion mode, the execution of determining at least one object to be analyzed in the object storage server may be triggered when the specified expiration time is reached; if the deletion mode given by the deletion condition preset by the target user is the second deletion mode, the at least one object to be analyzed in the object storage server can be determined periodically, that is, when each period is started, the at least one object to be analyzed in the object storage server can be triggered and determined.
In addition, the number of the preset deletion conditions may be one or more. For the case that the number of preset deletion conditions is multiple, if the deletion modes given by the multiple deletion conditions are all the first deletion mode, when the specified expiration time related to the deletion mode given by each deletion condition is reached, S101 may be triggered to be executed; if there are both the first deletion mode and the second deletion mode in the deletion modes given by the multiple deletion conditions, when the specified expiration time involved in the first deletion mode is reached, S101 may be triggered and executed, and at the same time, S101 may be triggered and executed periodically; and if the deletion modes given by the deletion conditions are all the second deletion mode, the execution of S101 may be triggered periodically.
S102, aiming at each object, when the object is judged to belong to the object to be deleted based on the set deletion condition, determining the object as the object to be deleted, and setting the field value of the designated metadata field of the object by using designated content; wherein, the designated content is content used for indicating the object is out of date;
after determining at least one object to be analyzed, each object may be traversed with the object as a traversal target. When traversing to each object, whether the object belongs to the object to be deleted can be judged based on a preset deleting condition, and when the judgment result is yes, the object is determined as the object to be deleted, and the field value of the designated metadata field of the object is set by using the designated content, namely the content of the field value of the designated metadata field of the object is set as: specifying content; when the judgment result is no, the deletion processing of the object is ended. In this way, the object to be deleted can be known through step S102, and at the same time, which objects are outdated objects are timely reflected in the target metadata database. It is understood that if a field value specifying a metadata field in metadata of an object is set to specify content, the object cannot be accessed, specifically, cannot be queried and downloaded, and when there is a query or download request for the object, the querying party or the downloading party may obtain 404 an error page.
It is understood that, for the deletion condition given the first deletion mode, in the specific implementation process of determining whether the object belongs to the object to be deleted based on the set deletion condition, it may be determined whether the object belongs to the object to be deleted by using the first object range given by the deletion condition, specifically, if the object is located: and if the deleting condition is given within the range of the first object, judging that the object belongs to the object to be deleted. For the deletion condition given the second deletion mode, in the specific implementation process of determining whether the object belongs to the object to be deleted based on the set deletion condition, the deletion condition is given a deletion mode and a first object range to determine whether the object belongs to the object to be deleted, specifically, if the object satisfies the deletion condition, the time length represented by the deletion mode is given, and the object is located: and if the deleting condition is given within the range of the first object, judging that the object belongs to the object to be deleted.
For example, in an implementation manner, determining the object record as the object to be deleted may include:
adding the object identification of the object to a target list; the target list is used for recording a list of objects to be deleted.
In this implementation manner, the object identifier of the object to be deleted is recorded in the target list, so that when the object is subsequently deleted, the object identifier can be read from the target list, and the object with the read object identifier is deleted.
The above-mentioned specific implementation manner of determining the object as the object to be deleted is only an example, and should not be construed as a limitation to the embodiment of the present invention. For example, determining the object as an object to be deleted may also include: and adding a label for indicating the object to be deleted, or setting the object identifier of the object in the target metadata base to be an identifier for indicating the object to be deleted, and the like.
In addition, there are a variety of specific forms of characterizing the range of the first object given in the deletion condition. For clarity of layout and clarity of the scheme, specific characterization of the object range is exemplified below.
S103, deleting the object to be deleted in the object storage server.
After determining each object to be deleted, each object to be deleted in the object storage server may be deleted. When the object is determined as the object to be deleted, the field value of the designated metadata field of the object to be deleted is set with designated content to reflect whether the object is overdue or not in time, so that even if the object is deleted with time delay, if a processing event related to object statistics occurs in the object deletion process, the object counted in the processing event processing process is only the unexpired object, and the accuracy of the processing result can be improved.
It will be appreciated that the specific implementation of deleting the object to be deleted in the object storage server is influenced by the specific implementation of determining the object as the object to be deleted described above.
For example, for a manner of adding the object identifier of the object to the target list, correspondingly, deleting the object to be deleted in the object storage server may include:
and reading the object identification from the target list, and deleting the object with the read object identification from the object storage server.
It is understood that, when the object is deleted, the electronic device may invoke its own deletion process to delete the object. In addition, considering that the load capacity of the electronic device where the object deleting device is located may be large, in order to further improve the object deleting efficiency, other devices with low load capacity may be triggered to delete the object to be deleted. Based on the processing idea, deleting the object to be deleted in the object storage server may include:
deleting the object to be deleted in the object storage server through the asynchronous deletion thread; wherein the asynchronous deleting thread is a thread running in a device with a load amount lower than a predetermined threshold.
The load of any electronic device may be calculated in any manner capable of realizing load calculation, which is not limited in the embodiment of the present invention.
Specifically, when reading each object identifier from the target list and deleting the object having the read object identifier from the object storage server, the method includes:
and reading each object identifier from the target list through an asynchronous deleting thread, and deleting the object with the read object identifier from the object storage server.
In this implementation, after being invoked, the asynchronous delete thread may read each object identifier from the target list, and delete the object with the read object identifier from the object storage server. And after deleting an object, the asynchronous deleting thread can report the object to the object deleting device, so that the object deleting device deletes the object identifier of the object in the target list, thereby avoiding the repeated deletion of the same object.
In addition, if the object deletion fails, a deletion failure record may be generated and saved into a bucket, i.e., a storage area, specified by the target user, so that the target user may analyze the failure cause based on the deletion failure record.
In the scheme, when the object is judged to belong to the object to be deleted, the field value of the designated metadata field of the object is set by using designated content, so that whether the object is overdue or not can be reflected in time. In this way, even if a processing event related to object statistics occurs during object deletion, since the target database records the expiration status of an object, the counted objects for the processing event may include only unexpired objects. Therefore, the method and the device can improve the accuracy of the processing result of the processing event related to the object statistics.
The method comprises the steps that when the requirement of object deletion of a target user is considered, protection requirements also exist for certain objects, namely whether the deletion condition is met or not, the certain objects cannot be deleted, and therefore the object deletion not only meets the deletion requirement of the target user, but also meets the protection requirement of the target user. Based on the processing idea, as shown in fig. 2, an embodiment of the present invention provides an object deleting method, which may include the following steps:
s201, determining at least one object to be analyzed in an object storage server;
in this embodiment, step S201 is the same as step S101 in the above embodiment, and is not described herein again.
S202, aiming at each object, when the object is judged to belong to the object to be deleted based on the set deletion condition, determining a preset protection condition, judging whether the object meets the protection condition, if the object does not meet the protection condition, determining the object as the object to be deleted, and setting the field value of the designated metadata field of the object by using designated content; if yes, finishing the deleting process of the object;
in the present embodiment, step S202 is similar to step S102 in the above-described embodiment, except that: when the object is judged to belong to the object to be deleted based on the set deletion condition, the object is not directly determined as the object to be deleted and the field value of the designated metadata field in the target metadata base is set, but whether the object is the object to be protected is judged based on the set protection condition, if not, the object record is determined as the object to be deleted, and the field value of the designated metadata field of the object is set by using the designated content. It is to be understood that the step of deleting the object is ended if it is determined that the object is an object to be protected.
The condition content of the protection condition at least comprises a second object range used for representing the range of the object to be deleted;
illustratively, in one implementation, the second object range is represented by a combination of operators including matches or mismatches and identifying content including: prefix and/or label information. The description contents about the operators and the identification contents refer to the corresponding contents in the deletion condition.
For ease of understanding, the protection conditions set based on this implementation are illustrated:
protection condition 1: objects that match to the prefix "1234".
S203, deleting the object to be deleted in the object storage server.
In this embodiment, step S203 is the same as step S103 of the above embodiment, and is not described herein again.
In the scheme provided by the embodiment, when the object is judged to belong to the object to be deleted, the field value of the specified metadata field of the object is set by using the specified content, so that whether the object is expired or not can be reflected in time. In this way, even if a processing event related to object statistics occurs during object deletion, since the target database records the expiration status of an object, the counted objects for the processing event may include only unexpired objects. Therefore, the method and the device can improve the accuracy of the processing result of the processing event related to the object statistics. In addition, in the scheme provided by the embodiment, different requirements of the user can be met through the set deleting condition and the set protecting condition.
The following gives examples of specific forms of characterizing the range of the first object given the deletion condition.
Illustratively, to ensure that the first object range can be flexibly set by the user according to the requirement, in one implementation, the first object range can be represented by a combination of operators and identification content, wherein the operators comprise matching or not matching, and the identification content comprises: prefix and/or label information. The tag information is contents such as important contents and backup contents preset for the object.
For ease of understanding, the deletion condition set based on this implementation is exemplified:
deletion condition 1: objects matching the prefix "123" specify deletion on days 2020-12-12;
deletion condition 2: matching the prefix '123' and marking the object with the label information 'backup', and deleting the object in 2020-12-12 days;
deletion condition 3: specifying an object that is deleted on days 2020-12-12, does not match the prefix "123" and is labeled with the "important" information.
For example, in another implementation, the object range may be represented by a storage space, a prefix of the object, or tag information of the object. For ease of understanding, the deletion condition set based on this implementation is exemplified:
deletion condition 1: storing an object with the position A, and designating 2020-12-12 to delete; deletion condition 2: an object with a prefix of '123' is deleted in a designated 2020-12-12 days; deletion condition 3: the label information is a 'backup' object and is deleted in 2020-12-12 days.
The foregoing detailed description of the subject scope is by way of example only and should not be construed as limiting the embodiments of the present invention.
Based on the object deleting method, the embodiment of the invention also provides a data processing method.
The execution subject of the data processing method may be a data processing apparatus. The data processing apparatus may be applied to an electronic device, wherein the electronic device may be an object storage server, or a management server for managing the object storage server, or the like. The object storage server is a server for storing the object.
As shown in fig. 3, a data processing method provided in an embodiment of the present invention may include:
s301, acquiring a data processing instruction; wherein the data processing instruction is a processing instruction of a processing event related to object statistics and related to a target user;
the data processing instruction may be an object query instruction, an object quantity statistical instruction, or an object capacity statistical instruction, but is not limited thereto. The object query instruction is used for querying an object belonging to a target user; the object quantity counting instruction is used for counting the quantity of the objects belonging to the target user; the object capacity statistics instructions are used to count the total capacity of objects belonging to the target user, which can be used for capacity billing.
S302, determining that the field value of the specified metadata field is not the specified content and the owner is the metadata of the target user from the target metadata database;
after the data processing instruction is obtained, due to the delay of object deletion, some objects that should be deleted originally are not deleted, that is, metadata of the objects that should be deleted originally also exist in the target metadata database, so that in order to ensure that only unexpired objects are counted, it is possible to determine, from the target metadata database, that a field value of a specified metadata field is not specified content and that an owner is metadata of the target user, and then perform subsequent processing using the determined metadata.
S303, based on the determined metadata, determines a processing result of the data processing instruction.
Wherein, different data processing instructions have different determined processing results. Specifically, if the data processing instruction is an object query instruction, determining an object represented by the determined metadata as a processing result; if the data processing instruction is an object number counting instruction, counting the number of objects represented by the determined metadata as a processing result; and if the data processing instruction is an object capacity statistical instruction, calculating the total space occupation amount of the object represented by the determined metadata as a processing result corresponding to the object capacity statistical instruction.
Wherein the total capacity of the object can be used to provide billing information to the user, which is accurate because the total capacity of the object is accurate. In this way, the cloud manufacturer can give the user a bill acceptance, that is, an acceptance that the object involved in the charging information is an unexpired object.
In the data processing method provided by the embodiment of the invention, after a data processing instruction is obtained, the field value of the specified metadata field is determined to be metadata which is not specified content and has an owner as a target user from a target metadata database; further, a processing result of the data processing instruction is determined based on the determined metadata. Since the metadata that the field content of the specified metadata field is not the specified value and the owner is the target user is determined from the target metadata database during the data processing, and the determined metadata is further used for determining the processing result, the objects counted for the processing event can only comprise the unexpired objects. Therefore, the accuracy of the processing result of the processing event related to the object statistics can be improved by the data processing method.
To facilitate understanding of the object deletion method and the data processing method described above, fig. 4 is a schematic diagram of the overall principle of the object deletion method and the data processing method. In fig. 4, the object deleting apparatus adds a specified metadata field in the target metadata base, records the identification information of the object in the target list when it is determined that the object belongs to the object to be deleted, and sets a field value of the specified metadata field of the object as specified content; the object deleting device can call an asynchronous deleting thread, and the object identifier in the target list is accessed through the asynchronous deleting thread to delete the object in the disk in the object storage server; also, the asynchronous delete thread may synchronously delete information of the operation to the object deletion apparatus. And when the data processing device obtains the data processing instruction, the data processing device can access the target metadata base so as to determine a response result corresponding to the data processing instruction.
Based on the object deleting method, the embodiment of the invention also provides an object deleting device. As shown in fig. 5, an object deleting apparatus provided in an embodiment of the present invention may include:
an object determination module 510, configured to determine at least one object to be analyzed in the object storage server; the metadata of the object in the object storage server is stored in a target metadata database, and a designated metadata field for representing whether the object is expired is preset in the target metadata database;
an object processing module 520, configured to determine, for each object, when it is determined that the object belongs to an object to be deleted based on a set deletion condition, the object as an object to be deleted, and set a field value of the designated metadata field of the object with designated content; wherein, the designated content is content used for indicating the object is out of date;
and the object deleting module 530 is used for deleting the object to be deleted in the object storage server.
In the object deleting device provided by the embodiment of the invention, a specified metadata field used for indicating whether an object is expired is added to a target metadata database in advance; thus, after at least one object to be analyzed is determined, for each object, when it is determined that the object belongs to an object to be deleted based on a set deletion condition, the object is determined as the object to be deleted, and a field value of the designated metadata field of the object is set using designated contents; wherein, the designated content is content used for indicating the object is out of date; and then deleting the object to be deleted in the object storage server. In the scheme, when the object is judged to belong to the object to be deleted, the field value of the designated metadata field of the object is set by using designated content, so that whether the object is overdue or not can be reflected in time. In this way, even if a processing event related to object statistics occurs during object deletion, since the target database records the expiration status of an object, the counted objects for the processing event may include only unexpired objects. Therefore, the method and the device can improve the accuracy of the processing result of the processing event related to the object statistics.
Optionally, the object processing module 520 is further configured to determine a preset protection condition before determining the object as the object to be deleted; judging whether the object meets the protection condition; if the object is judged not to meet the protection condition, determining the object to be deleted; if so, the deletion process for the object is ended.
Optionally, the condition content of the deletion condition at least includes a first object range for representing a range to be deleted of the object;
the first object range is represented by a combination of operators and identified content, the operators including a match or a mismatch, the identified content including: prefix and/or label information.
Optionally, the condition content of the protection condition at least includes a second object range for representing a range to be deleted of the object;
the second object range is represented by a combination of operators and identified content, the operators including a match or a mismatch, the identified content including: prefix and/or label information.
Optionally, the determining, by the object processing module 520, the object as an object to be deleted includes:
adding the object identification of the object to a target list; the target list is used for recording a list of objects to be deleted;
the object deleting module 530 is specifically configured to:
and reading the object identification from the target list, and deleting the object with the read object identification from the object storage server.
Optionally, the object deleting module 530 is specifically configured to: deleting the object to be deleted in the object storage server through an asynchronous deletion thread; wherein the asynchronous deleting thread is a thread running in a device with a load amount lower than a predetermined threshold.
Corresponding to the object deleting device, the embodiment of the invention also provides a data processing device. As shown in fig. 6, the data processing apparatus may include:
an instruction obtaining module 610, configured to obtain a data processing instruction; wherein the data processing instruction is a processing instruction of a processing event related to object statistics and related to a target user;
a metadata determining module 620, configured to determine, from the target metadata database, that a field value of a specified metadata field is not specified content and an owner is metadata of the target user;
a processing result determining module 630, configured to determine a processing result of the data processing instruction based on the object represented by the determined metadata.
In the data processing apparatus provided in the embodiment of the present invention, after a data processing instruction is obtained, it is determined from a target metadata repository that a field value of a specified metadata field is not specified content and an owner is metadata of a target user; further, a processing result of the data processing instruction is determined based on the determined metadata. Since the metadata that the field content of the specified metadata field is not the specified value and the owner is the target user is determined from the target metadata database during the data processing, and the determined metadata is further used for determining the processing result, the objects counted for the processing event can only comprise the unexpired objects. Therefore, the accuracy of the processing result of the processing event related to the object statistics can be improved by the data processing method.
Optionally, the instruction obtaining module 610 is specifically configured to:
obtaining an object capacity statistical instruction;
the processing result determining module 630 is specifically configured to: and calculating the total space occupation amount of the object represented by the determined metadata as a processing result corresponding to the object capacity statistical instruction.
Based on the object deleting method, an embodiment of the present invention further provides an electronic device, as shown in fig. 7, including a processor 701, a communication interface 702, a memory 703 and a communication bus 704, where the processor 701, the communication interface 702 and the memory 703 complete mutual communication through the communication bus 704,
a memory 703 for storing a computer program;
the processor 701 is configured to implement the steps of the object deleting method provided in the embodiment of the present invention when executing the program stored in the memory 703.
Based on the above data processing method, an embodiment of the present invention further provides an electronic device, as shown in fig. 8, including a processor 801, a communication interface 802, a memory 803, and a communication bus 804, where the processor 801, the communication interface 802, and the memory 803 complete mutual communication via the communication bus 804,
a memory 803 for storing a computer program;
the processor 801 is configured to implement the steps of the data processing method provided by the embodiment of the present invention when executing the program stored in the memory 803.
The communication bus mentioned in the electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
The communication interface is used for communication between the electronic equipment and other equipment.
The Memory may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.
The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components.
In still another embodiment provided by the present invention, a computer-readable storage medium is further provided, in which a computer program is stored, and the computer program, when executed by a processor, implements the steps of the above-mentioned object deleting method.
In yet another embodiment provided by the present invention, a computer-readable storage medium is further provided, in which a computer program is stored, and the computer program realizes the steps of the above data processing method when executed by a processor.
In a further embodiment provided by the present invention, there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform the steps of the object deletion method in the above-described embodiments.
In a further embodiment provided by the present invention, there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform the steps of the data processing method in the above-described embodiment.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, as for the apparatus, device, and storage medium embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and reference may be made to some descriptions of the method embodiments for relevant points.
The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.
Claims (12)
1. An object deletion method, comprising:
determining at least one object to be analyzed in an object storage server; the metadata of the object in the object storage server is stored in a target metadata database, and a designated metadata field for representing whether the object is expired is preset in the target metadata database;
for each object, when the object is judged to belong to an object to be deleted based on a set deletion condition, determining the object as the object to be deleted, and setting a field value of the designated metadata field of the object by using designated content; wherein, the designated content is content used for indicating the object is out of date;
and deleting the object to be deleted in the object storage server.
2. The method of claim 1, wherein before determining the object as the object to be deleted, the method further comprises:
determining a preset protection condition;
judging whether the object meets the protection condition;
if the object is judged not to meet the protection condition, determining the object to be deleted; if so, the deletion process for the object is ended.
3. The method according to claim 1 or 2, wherein the condition content of the deletion condition includes at least a first object range for representing a range in which an object is to be deleted;
the first object range is represented by a combination of operators and identified content, the operators including a match or a mismatch, the identified content including: prefix and/or label information.
4. The method according to claim 2, wherein the condition content of the protection condition comprises at least a second object range for representing a range in which objects are to be deleted;
the second object range is represented by a combination of operators and identified content, the operators including a match or a mismatch, the identified content including: prefix and/or label information.
5. The method according to claim 1 or 2, wherein the determining the object as the object to be deleted comprises:
adding the object identification of the object to a target list; the target list is used for recording a list of objects to be deleted;
the deleting the object to be deleted in the object storage server includes:
and reading the object identification from the target list, and deleting the object with the read object identification from the object storage server.
6. The method according to claim 1 or 2, wherein the deleting the object to be deleted in the object storage server comprises:
deleting the object to be deleted in the object storage server through an asynchronous deletion thread; wherein the asynchronous deleting thread is a thread running in a device with a load amount lower than a predetermined threshold.
7. A data processing method based on the method of any one of claims 1 to 6, comprising:
obtaining a data processing instruction; wherein the data processing instruction is a processing instruction of a processing event related to object statistics and related to a target user;
determining, from a target metadata database, that a field value of a specified metadata field is not specified content and an owner is metadata of the target user;
determining a processing result of the data processing instruction based on the determined metadata.
8. The method of claim 7, wherein obtaining data processing instructions comprises:
obtaining an object capacity statistical instruction;
the determining a processing result of the data processing instruction based on the determined metadata includes:
and calculating the total space occupation amount of the object represented by the determined metadata as a processing result corresponding to the object capacity statistical instruction.
9. An object deletion apparatus, comprising:
the object determining module is used for determining at least one object to be analyzed in the object storage server; the metadata of the object in the object storage server is stored in a target metadata database, and a designated metadata field for representing whether the object is expired is preset in the target metadata database;
an object processing module, configured to determine, for each object, an object to be deleted when it is determined that the object belongs to an object to be deleted based on a set deletion condition, and set a field value of the designated metadata field of the object with designated content; wherein, the designated content is content used for indicating the object is out of date;
and the object deleting module is used for deleting the object to be deleted in the object storage server.
10. A data processing apparatus according to claim 9, comprising:
the instruction obtaining module is used for obtaining a data processing instruction; wherein the data processing instruction is a processing instruction of a processing event related to object statistics and related to a target user;
the metadata determining module is used for determining that the field value of the specified metadata field is not the specified content and the owner is the metadata of the target user from the target metadata base;
and the processing result determining module is used for determining the processing result of the data processing instruction based on the determined metadata.
11. An electronic device is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor and the communication interface are used for realizing mutual communication by the memory through the communication bus;
a memory for storing a computer program;
a processor for implementing the method steps of any one of claims 1 to 6, or the method steps of any one of claims 7 to 8, when executing a program stored in a memory.
12. A computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, which computer program, when being executed by a processor, carries out the method steps of any of the claims 1-6 or the method steps of any of the claims 7-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010831443.0A CN111966867A (en) | 2020-08-18 | 2020-08-18 | Object deleting method, data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010831443.0A CN111966867A (en) | 2020-08-18 | 2020-08-18 | Object deleting method, data processing method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111966867A true CN111966867A (en) | 2020-11-20 |
Family
ID=73388324
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010831443.0A Pending CN111966867A (en) | 2020-08-18 | 2020-08-18 | Object deleting method, data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111966867A (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140201187A1 (en) * | 2011-08-30 | 2014-07-17 | Johan G. Larson | System and Method of Search Indexes Using Key-Value Attributes to Searchable Metadata |
| CN107357808A (en) * | 2017-05-27 | 2017-11-17 | 北京五八信息技术有限公司 | Data managing method, device and equipment |
| WO2018145227A1 (en) * | 2017-02-13 | 2018-08-16 | Pivotal Software, Inc. | Managing data records in object-based cloud storage systems |
| CN108932236A (en) * | 2017-05-22 | 2018-12-04 | 北京金山云网络技术有限公司 | A kind of file management method, scratch file delet method and device |
| CN109213604A (en) * | 2018-10-15 | 2019-01-15 | 北京奇艺世纪科技有限公司 | A kind of management method and device of data source |
| CN109213450A (en) * | 2018-09-10 | 2019-01-15 | 郑州云海信息技术有限公司 | A kind of associated metadata delet method, device and equipment based on flash array |
| CN109815213A (en) * | 2018-12-20 | 2019-05-28 | 武汉璞华大数据技术有限公司 | It is deleted on a kind of Append-Only database and the method and system of modification data |
| CN111124306A (en) * | 2019-12-20 | 2020-05-08 | 浪潮电子信息产业股份有限公司 | Data deleting method, system, equipment and computer readable storage medium |
-
2020
- 2020-08-18 CN CN202010831443.0A patent/CN111966867A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140201187A1 (en) * | 2011-08-30 | 2014-07-17 | Johan G. Larson | System and Method of Search Indexes Using Key-Value Attributes to Searchable Metadata |
| WO2018145227A1 (en) * | 2017-02-13 | 2018-08-16 | Pivotal Software, Inc. | Managing data records in object-based cloud storage systems |
| CN108932236A (en) * | 2017-05-22 | 2018-12-04 | 北京金山云网络技术有限公司 | A kind of file management method, scratch file delet method and device |
| CN107357808A (en) * | 2017-05-27 | 2017-11-17 | 北京五八信息技术有限公司 | Data managing method, device and equipment |
| CN109213450A (en) * | 2018-09-10 | 2019-01-15 | 郑州云海信息技术有限公司 | A kind of associated metadata delet method, device and equipment based on flash array |
| CN109213604A (en) * | 2018-10-15 | 2019-01-15 | 北京奇艺世纪科技有限公司 | A kind of management method and device of data source |
| CN109815213A (en) * | 2018-12-20 | 2019-05-28 | 武汉璞华大数据技术有限公司 | It is deleted on a kind of Append-Only database and the method and system of modification data |
| CN111124306A (en) * | 2019-12-20 | 2020-05-08 | 浪潮电子信息产业股份有限公司 | Data deleting method, system, equipment and computer readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108446407B (en) | Database auditing method and device based on block chain | |
| CN109213604B (en) | Data source management method and device | |
| US8769200B2 (en) | Method for managing hierarchical storage during detection of sensitive information, computer readable storage media and system utilizing same | |
| CN108932286B (en) | Data query method and device | |
| US9514170B1 (en) | Priority queue using two differently-indexed single-index tables | |
| CN111488377A (en) | Data query method and device, electronic equipment and storage medium | |
| CN111831915A (en) | Method, device, electronic equipment and storage medium for responding to data query request | |
| CN115857801A (en) | Data migration method and device, electronic equipment and storage medium | |
| CN111399849A (en) | Method and system for determining application program installation package | |
| CN113010494A (en) | Database auditing method and device and database proxy server | |
| CN112948363A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN111522626A (en) | Virtual machine list generation method and device and electronic equipment | |
| CN111966867A (en) | Object deleting method, data processing method and device | |
| CN110727895A (en) | Sensitive word sending method and device, electronic equipment and storage medium | |
| CN113760854A (en) | Method for identifying data in HDFS memory and related equipment | |
| CN115587247A (en) | Method, device and equipment for monitoring user label and storage medium | |
| CN112148721B (en) | Data checking method and device, electronic equipment and storage medium | |
| CN114896082A (en) | Message processing method and device, electronic equipment and storage medium | |
| CN111399754B (en) | Method and device for releasing storage space and distributed system | |
| CN114356211A (en) | Storage space processing method and device, electronic equipment and storage medium | |
| CN113742378A (en) | Data query and storage method, related equipment and storage medium | |
| CN113821166A (en) | Method, device and equipment for aggregating multi-version small objects | |
| CN114064603A (en) | Data verification method and device, electronic equipment and medium | |
| CN111144848A (en) | Highway engineering quantity standard list management system and method and computer | |
| CN111078643A (en) | Method and device for deleting files in batches and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |