CN109241178A - The method and apparatus for managing memory space - Google Patents

The method and apparatus for managing memory space Download PDF

Info

Publication number
CN109241178A
CN109241178A CN201810804528.2A CN201810804528A CN109241178A CN 109241178 A CN109241178 A CN 109241178A CN 201810804528 A CN201810804528 A CN 201810804528A CN 109241178 A CN109241178 A CN 109241178A
Authority
CN
China
Prior art keywords
time
memory space
objects
metadata
statistical information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201810804528.2A
Other languages
Chinese (zh)
Inventor
薛曜
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Priority to CN201810804528.2A priority Critical patent/CN109241178A/en
Publication of CN109241178A publication Critical patent/CN109241178A/en
Pending legal-status Critical Current

Links

Abstract

Provide a kind of method and apparatus for managing memory space, it is indexed at the first time by introducing, it include all object metadatas of index at the first time using first time search index, and calculate the capacity increment statistical information of each object in one or more objects, the capacity increment statistical information of obtained one or more objects is subjected to polymerization processing using same static fields, obtain polymerization result, it obtains indexing the polymerization result in the time slicing of instruction for same static fields at the first time, finally memory space is managed using polymerization result, help to improve the management performance of memory space.

Description

The method and apparatus for managing memory space
Technical field
This application involves field of storage, and more particularly, to a kind of method and apparatus for managing memory space.
Background technique
Object storage service (Object Storage Service) is that one kind can store document, picture, audio-visual video etc. The cloud storage service of unstructured data can provide magnanimity, safe, highly reliable, inexpensive data storage capacities.Simply deposit Storage service (simple storage service, S3) provides object by simple network service (Web Services) interface Storage service.S3 provides the access of declarative state transfer (representational state transfer, REST) form Interface allows users to easily by accessing object storage service in Web application.
Bucket (bucket), also referred to as memory paragraph, memory space, container as S3 system.User can create multiple buckets, There is no limit for quantity, format (can be file, picture, video etc.) and the size summation for the object stored in each bucket.Currently, S3 system charging mode is divided into detailed charging information and statistics charge information.For example, when user uploads a 5GB file, upper Charging is carried out according to uninterrupted when passing request;If file is not deleted always, the memory space of S3 system is occupied, then is also wanted According to the occupied time statistical information charging of memory space.
In the prior art, statistical information charging is to be written or delete data in user by the synchronous charging in foreground While, S3 system needs the synchronous statistics charge information in foreground.But in big concurrent scene, (for example same user is in same a period of time Between point write-in or delete a large amount of objects) or bottom there are scenes, the foreground synchronized update statistics affairs such as IO is abnormal and just will appear The problems such as statistical information charging is not allowed.Therefore, it needs to propose that a kind of method for managing memory space promotes managerial energy.
Summary of the invention
In view of this, this application provides a kind of method and apparatus for managing memory space to have by introducing time index Help improve the management performance of memory space.
It include multiple clothes in distributed objects storage system in a first aspect, providing a kind of method for managing memory space Business node, the service node are used for storage object, which comprises inquiry includes all object metas indexed at the first time Data, the first time index is used to indicate the period where the time for executing management operation to the object, described right Object metadata is the corresponding object metadata of one or more objects;Calculate the appearance of each object in one or more of objects Accrual accounting information is measured, the capacity increment statistical information includes the size of object, wherein the capacity increment statistical information is By the corresponding object metadata record of object;The capacity increment statistical information of obtained one or more objects is used same Static fields carry out polymerization processing, obtain polymerization result, and the polymerization result includes: when first time index indicates Between the variable quantity of memory space that is occupied in section, according to one or more objects of static fields statistics.
In the embodiment of the present application, statistics node includes the institute indexed the first time using first time search index There is object metadata, the capacity increment statistical information of each object in one or more of objects is then calculated, by what is obtained The capacity increment statistical information of one or more objects carries out polymerization processing using same static fields, obtains polymerization result, institute State polymerization result include: the first time index instruction period in, according to the static fields statistics one or The variable quantity for the memory space that multiple objects occupy finally is carried out the management of memory space using obtained polymerization result, helped In the management performance for improving memory space.Further, compared to the mode of foreground synchronized update charging, counting node can make Asynchronous statistics charging is carried out with above-mentioned polymerization result, reduces the processing affairs of foreground process, helps to improve foreground performance.
In one possible implementation, before the inquiry includes all object metadatas indexed at the first time, The method also includes:
Object Management group request from the user is obtained, the Object Management group request executes management to the first object for requesting Operation, first object is an object in one or more of objects, and the management operation includes in following operation Any one: new Object, upload object, delete object;
The first time index is distributed according to the period where the time when getting Object Management group request;
The metadata of the first object of generation is requested based on the Object Management group, wherein in the metadata of first object It is indexed including the first time;
Store the first time index.
Therefore, after statistics node receives the Object Management group request of user on foreground, by obtaining current time index (ratio As first time indexes), and first time index is stored into disk, in order to subsequent use.
In one possible implementation, the capacity increment statistical information of obtained one or more objects is used same One static fields carry out polymerization processing, including,
For meeting the object of the static fields within the period of first time index instruction:
The memory space that the object that cumulative new Object operation creates occupies;
Cumulative upload object operates the memory space that uploaded object occupies;
Subtract the memory space of deleted object release.
Therefore, whether new Object or upload object, or deletion object, after the embodiment of the present application is based on the period Platform statistical is applicable in.
In one possible implementation, the method also includes:
In the case where obtaining all one or more object metadatas including first time index, alternatively, In the case where the corresponding memory headroom of the accrual accounting information reaches memory threshold, the one or more of object metas of lower wall Data.
Therefore, statistics node is by setting preset condition, so that object metadata lower wall when meeting preset condition, avoids It is to the holding time of memory headroom too long.
In one possible implementation, the method also includes: using the polymerization result manage memory space.
Optionally, memory space is managed using the polymerization result, comprising:
Charging is carried out based on variable quantity of the polymerization result to memory space.
Therefore, service node can be used above-mentioned polymerization result and carry out asynchronous statistics charging, reduce the place of foreground process Director's business, helps to improve foreground performance.
Second aspect provides a kind of device for managing memory space, which includes for executing above-mentioned first aspect Or the method for the management memory space in first aspect in any implementation method.
The third aspect provides a kind of device for managing memory space, which includes: processor and memory;This is deposited For storing instruction, when the apparatus is operative, which executes the instruction of memory storage to reservoir, so that the device is held Management memory space method in any implementation method of the above-mentioned first aspect of row or first aspect.It should be noted that this is deposited Reservoir can integrate in processor, be also possible to independently of except processor.
Fourth aspect provides a kind of computer readable storage medium, which has program, It is above-mentioned in a first aspect, and its any one of possible implementation management that the program executes the device for managing memory space The method of memory space.
5th aspect, provides a kind of chip system, including processor, which is connected with memory, the processing Device from memory for calling and running computer program, so that the computer equipment for being equipped with the chip system executes Either state in either side and its possible embodiment method.The memory can be located inside the chip system, can also To be located at outside the chip system.
6th aspect, provide a kind of program, the program when being executed by processor, for execute in a first aspect, and its The method of any one of possible implementation management memory space.
7th aspect, provides a kind of program product, described program product includes: program code, works as said program code It is managed scan module, computing module, processing module, management module or the place of the device (for example, service node) of memory space Manage device, input/output interface operation when so that described device execute it is above-mentioned in a first aspect, and its in possible implementation The method of any management memory space.
Detailed description of the invention
Fig. 1 is the schematic diagram of synchronized update statistics charge information in the prior art.
Fig. 2 is the schematic diagram indexed according to the service node acquisition time of the embodiment of the present application.
Fig. 3 is the schematic diagram using the period of the embodiment of the present application.
Fig. 4 is the schematic flow chart according to the method for the management memory space of the embodiment of the present application.
Fig. 5 is the schematic diagram according to an example of the background update charging metadata of the embodiment of the present application.
Fig. 6 is the schematic block diagram according to the device of the management memory space of the embodiment of the present application.
Fig. 7 is the schematic block diagram according to the statistics node of the embodiment of the present application.
Specific embodiment
Below in conjunction with attached drawing, the technical solution in the application is described.
The technical solution of the embodiment of the present application can be applied to the storage system for having regulatory requirement to memory space, distribution Object storage system or cloud storage system.Optionally, the technical solution of the embodiment of the present application can be applied to simple storage service In (simple storage service, S3).S3 provides object storage service by simple Web Services interface (Object Storage Service).For example, the technical solution of the embodiment of the present application can be applied to the member in storage system The statistics charge system of data.
Currently, storage system is to update statistics in such a way that foreground is synchronous when updating charge information in the prior art Charge information, for example, user, while being written or deleting data, system can synchronized update statistics charge information.User is logical Cross client to S3 interface send operation requests, which may include one or more in following: upload object, on Biography section, deletion object, deletion section, covering upload object, covering upload section, bucket deletion etc..These operations will lead to customer charge Increase, therefore be related to carrying out charging depending on the user's operation, that is to say, that S3 system is synchronous more for aforesaid operations request Change billing increment information.By taking user requests upload object as an example, it is described here in conjunction with the schematic diagram in Fig. 1.Fig. 1 is existing The schematic diagram of synchronized update statistics charge information in technology.As shown in Figure 1, S3 system, which receives user, passes through client to S3 interface The upload operation of transmission is requested, and then generates corresponding metadata according to the data information of user.Then, S3 system reception comes from The flow data of customer side, and by flow data write-in distributed memory system hard disk.S3 system deposits metadata write-in distribution The hard disk of storage system.Meanwhile S3 system foreground meeting synchronous recording charging metadata, specifically i.e.: generating charging in upload object Billing increment information is placed in waiting list (queue) by increment information, then uses function (for example, poll () function) Current file pointer is suspended to waiting list, the charging metadata of each user of synchronized update, wherein different numbers in Fig. 1 The charging metadata of (including 1,2 and 3) is the charging metadata of different user, has N number of charging in the single bucket under each user Metadata, recording in charging metadata has: the information such as object number, object size for including in corresponding bucket.S3 system is based on this Secondary upload operation requests corresponding increment information, requests corresponding increment information to be added to charging metadata this upload operation In.Finally, lower wall charging metadata (lower wall, which refers to, to be stored charging metadata into non-volatile memory medium from memory, Such as storage is into hard disk), the operation of this upload request is completed.
In big concurrent scene, the mode of above-mentioned synchronized update statistics charge information is simultaneously not suitable for.For example, if same user At same time point, a large amount of objects are written in request, then synchronous statistics bulk information are needed, if synchronized update failure due to time-out Or the failure due to version conflict, it will statistical information is caused to update failure.Therefore, the application is quasi- introduces the general of period It reads, the time index asynchronous management memory space (for example, statistics that backstage carries out charge information) based on the period, to improve system Count the performance of charging.
The method of the management memory space of the embodiment of the present application can be applied in distributed objects storage system, distributed It include multiple service nodes in object storage system, the service node is used for storage object;It can be corresponding to storage object Object metadata (metadata can be referred to as);Optionally, metadata can also be taken by individual one or more metadata Business device storage.Method in the embodiment of the present application can (management node can be point by the management node in distributed memory system Host node in cloth storage system, for managing multiple service nodes) or service node (Service Node) execution, also It can be executed by third party's node in distributed memory system, for the convenience of description, execution the embodiment of the present application method Node is referred to as statistics node.Statistics node is synchronous for use by foreground process in the Object Management group request for receiving user The object write time index of family request operation, in order to which the asynchronous scanning in backstage uses.It should be understood that management node can be with It is one in service node, the function of storage object is provided while possessing management function;It is also possible to dedicated management Node does not provide the store function of object.
The acquisition process that will be detailed below time index in the embodiment of the present application, specifically includes:
Object Management group request from the user is obtained, the Object Management group request executes management to the first object for requesting Operation, the management operation include any one of following operation: new Object, deletes object at upload object;
The first time index is distributed according to the period where the time when getting Object Management group request;
The metadata of the first object of generation is requested based on the Object Management group, wherein in the metadata of first object It is indexed including the first time;
Store the first time index.
Specifically, statistics node is after the Object Management group request for receiving user, acquisition is currently received Object Management group request Time where period corresponding time index (such as index at the first time), and object is written into obtained time index Metadata in.Node is counted in the metadata of the first object of lower wall, needs to store first time index into disk, with It is used convenient for the asynchronous scanning in backstage.
Here the format of the first object is not construed as limiting, for example, the first object can be the files such as video, photo, music, A part of file or other data except file.It should be understood that retouched by taking the first object as an example here State, actually user request management the corresponding metadata of object can in the way of above-mentioned first object write time rope Draw, this is not construed as limiting.
Time index is for identifying the different periods in the embodiment of the present application, and each period, there are corresponding time ropes Draw.Time index may also be referred to as the description that time tag, time slicing value etc. have index function.Wherein, the period refers to One section of duration, may also be referred to as time slicing, for example, 1 point 05 assigns at 1 point can be for 30 seconds 06 minutes if dividing according to the time One period, 1 point can be period ... etc. in 08 minutes at 31 seconds 06 minutes to 1 point.Period corresponding time index Can be incremented by successively sequentially in time since a value, for example, if the period when it is 150 seconds a length of, time index from 0 starts, and increased every 150 seconds and adds 1, specifically i.e.: the corresponding time index of first 150s is 0, when second 150s is corresponding Between index be 1 ... the corresponding time index of n-th 150s is N-1.Optionally, in specific implementation, time index can pass through length Integer long [int] definition, for example, time index can indicate that int value is only increased since 0, and with fixation with int value Time increase, every preset time (such as 150 seconds) plus one.Certainly, time index is intended merely to distinguish the different periods, Therefore there can also be other implementing methods, for example, by using descending integer as time index, 2 system numbers are as time rope Draw, or uses English alphabet, Chinese character as index.Optionally, the naming rule indexed by setting time, Ke Yicong The worth precedence relationship for knowing the period of time index.
By taking service node is executing subject as an example, service node can be obtained at the first time by login mechanism from host node Index, alternatively, the first time index can be generated in service node.Optionally, it if host node breaks down, services Node can also be indexed by the slave node acquisition time of host node, be not construed as limiting to this.For example, as shown in Fig. 2, distributed right As storage system may include multiple service nodes, host node and slave node.When multiple service nodes can be obtained by host node Between index.Also, the content stored in host node can also be backed up in the slave node of host node.In this way, even if main section Failure has occurred in point, and time index can be obtained by nor affecting on service node.
In addition, the difference with service node is that management node receives user if the above method is executed by management node Object Management group request after, can be corresponding from the local period that directly acquire where being currently received the time that Object Management group is requested Time index (for example indexing at the first time), and will be in the metadata of obtained time index write-in object.
It should be noted that statistics node and uses first time rope in the operation for the Object Management group request for executing user Draw query object metadata be it is asynchronous, can by using different processes realize asynchronous operation.For example, statistics node is being held The Object Management group request of row user operates with foreground process, after use when first time search index object metadata Platform process.Wherein, foreground process is usually what user used, needs higher response speed;Background process substantially not with Family interaction, the demand to response speed be not high.Since foreground object management operation and background scanning are asynchronous, in this way, even if It is big concurrent (for example, same user is written or deletes a large amount of objects at same time point) scene, foreground performance will not be influenced.
In order to make it easy to understand, the process of write time index is described by taking the process of upload object as an example below:
Statistics node is after the upload object request for receiving user, available logical timer.Optionally, logical timer is Refer to the current newest time index (or time tag value) obtained from host node.For example, if the period divided with 150s, the 1st A period corresponds to time index 0, and the 2nd period corresponds to time index 1 ..., etc. then the logical timer obtained can be Time index 1.Before statistics node acquisition time index, there may be other statistics nodes to obtain time index Value, therefore the time index obtained at this time is newest time index this moment.Alternatively, logical timer be also possible to it is current newest Logical timer, statistics node are based on the current newest available time index of logical timer.For example, if the period with 150s draw Point, the 1st period is 1s- 150s, and the 2nd period is 151s- 300s ..., etc. if the logical timer obtained It is 90s, then statistics node can learn that the value of time index is 0.Node is counted after obtaining time index, when by this Between index write-in object metadata in.Compared to the process of existing upload object, the time has been increased newly in object metadata here Index increases time index Create Time:X newly for example, the time index can be indicated using X in object metadata.
Similarly, by taking the process for deleting object as an example: statistics node is after receiving the deletion object requests of user, it is also desirable to Newest time index is obtained, and in the corresponding object metadata of object that time index write-in is needed to delete.
In the embodiment of the present application, by time index by the corresponding object of object in distributed memory system (object) Manage request operation (Object Management group request operation may include any one of new Object, upload object, deletion object) Time divides according to time shaft.For example, as shown in figure 3, the corresponding Object Management group request of object in distributed memory system is grasped The time of work can be divided according to the time shaft of N number of period, be specifically included:
Object in period 0 includes video 1 and music 1.Namely: video 1 and music 1 be performed management operation when Between point belong within the scope of the period 0.The time for executing management operation specifically can be the time for starting to execute, at other It can be the time for executing and completing in embodiment, in a further embodiment, be also possible to some equipment (such as S3 management section The service node of point or S3) receive the time for executing the request of management.Wherein, it can be stamped respectively in video 1 and music 1 The label of period 0, for example, the request of the corresponding Object Management group of video 1 is delete operation, then the corresponding time tag of video 1 is Erasing time (delete time) 0, for another example, the request of the corresponding Object Management group of music 1 are newly-built operations, then music 1 is corresponding Time tag is creation time (create time) 0;
Object in period 1 includes photo 2, video 2 and music 1, wherein can be divided in photo 2, video 1 and music 1 The label of period 1 is not stamped, for example, the request of the corresponding Object Management group of photo 2 is delete operation, then corresponding time of photo 2 Label is erasing time (delete time) 1, and for another example, the request of the corresponding Object Management group of video 2 is newly-built operation, then video 2 Corresponding time tag is creation time (create time) 1, and for another example, the request of the corresponding Object Management group of music 1 is newly-built behaviour Make, then the corresponding time tag of music 1 is creation time (create time) 1;
Object in period 2 includes backup 1, for example, the corresponding Object Management group request of backup 1 is upload operation, then it is standby 1 corresponding time tag of part is uplink time (upload time) 2;…;
Object in period N includes backup 2 and backup 3, for example, the corresponding Object Management group request of backup 2 is to upload behaviour Make, then backing up 2 corresponding time tags is uplink time (upload time) N, and for another example, 3 corresponding Object Management groups of backup are asked The newly-built operation of Seeking Truth, then backing up 3 corresponding time tags is creation time (create time) N.
Described above is how the process of write time index, how be described below using time index.
Fig. 4 shows the schematic flow chart of the method 200 of the management memory space according to the embodiment of the present application.The side Method 200 can be executed by the device (virtual bench that physical entity device or software program module form) of management memory space, than Such as, it can be executed by service node or management node.As shown in figure 4, the method 200 includes:
S210, inquiry include all object metadatas of index at the first time, wherein include one in the object metadata The corresponding metadata of a or multiple objects, the first time index were used to indicate where the time for executing management operation to object Period.
If method 200 is executed by service node, it includes the first time that service node can be inquired to management node All object metadatas of index, alternatively, if method 200 is executed by management node, management node can include in local search All object metadatas of the first time index.
By taking first time indexes as an example, statistics node carries out the overall situation to distributed objects storage system by background process and sweeps Retouch, enumerate (list) go out it is all include index at the first time objects, with obtain include one indexed at the first time or Multiple object metadatas.In other words, for the object metadata in each period, time index can be based on by counting node It is scanned.
Optionally, one or more object metadatas including first time index can be user and request to delete Object metadata, be also possible to the metadata that user requests the object uploaded, be also possible to the newly-built object of user's request Metadata, this is not construed as limiting.
S220 calculates the capacity increment statistical information of each object in one or more of objects, the capacity increment Statistical information includes the size of object.
Wherein, the capacity increment statistical information is used for the capacity or memory space that objects of statistics occupies.If object pair The Object Management group request operation answered is to delete object, then capacity increment statistical information statistics is negative increment, that is, needs to delete Memory space;If the corresponding Object Management group request operation of object is upload or new Object, capacity increment statistical information system Meter is positive increment, i.e., object needs the amount of capacity that occupies.The purpose of above-mentioned capacity increment statistical information is objects of statistics The memory space for occupying or deleting.
It optionally, can also include the number of object in the capacity increment statistical information.It should be understood that the number of object is simultaneously It is not necessarily meant to, can be used for the memory space of ancillary statistics object occupancy.
Optionally, statistics node can calculate the appearance of each object after obtaining said one or multiple object metadatas Measure accrual accounting information.For example, if user requests to upload an object to be that 5GB (Gigabyte) (5GB is the object uploaded The amount of capacity for needing to occupy) operation, then capacity increment statistical information can be { 5GB ,+1 }.For another example, if user The operation that an object is 2GB is deleted in request, then capacity increment statistical information can be { 2GB, -1 }.
The capacity increment statistical information of obtained one or more objects is polymerize by S230 using same static fields Processing, obtains polymerization result, and the polymerization result includes: within the period of first time index instruction, according to The variable quantity for the memory space that one or more objects of static fields statistics occupy.
Above-mentioned polymerization processing is specific i.e.: statistics node is right after the capacity increment statistical information for obtaining each object The memory space occupied for the capacity increment statistical information of same static fields is counted, to obtain for same statistics The polymerization result of field.
Wherein, it can be a kind of object meta with specific feature for the capacity increment statistical information of same static fields The capacity increment statistical information of data, for example, belong to it is same user's or same bucket in or same file folder appearance Accrual accounting information is measured, this is not construed as limiting.
Optionally, static fields may include any one of following field: the words such as user, file, priority, bucket Section.In other words, it can be stored according to user, file, priority or bucket as the range counted according to the period The statistics of the occupied dosage in space.
Optionally, statistics node after the capacity increment statistical information for obtaining each object, can will it is described each The capacity increment statistical information of object is added in caching.Specifically, caching can be divided into multiple memory headrooms, for same The capacity increment statistical information of one static fields, can be stored in identical memory headroom.
Optionally, the capacity increment statistical information of the one or more objects that will be obtained using same static fields into Row polymerization processing, including,
For meeting the object of the static fields within the period of first time index instruction:
The memory space that the object that cumulative new Object operation creates occupies;
Cumulative upload object operates the memory space that uploaded object occupies;
Subtract the memory space of deleted object release.
Specifically, counting node for meeting same statistics word within the period of first time index instruction The object of section can subtract the memory space of object release, if right if the corresponding management operation request of object is removal request As corresponding management operation request is upload or request to create, the then memory space that the object that can add up occupies.
With static fields be user for example, same user can upload multiple data in section at the same time, for example, Party A-subscriber uploads video 5GB, music 1GB, picture 10GB, deletes picture 2GB, completes upload operation in period N, So statistics node statistics go out multiple object accrual accounting information, and polymerization result after caching is added are as follows: and {+5GB ,+1 }++ 1GB ,+1 }+{+10GB ,+1 }+{ -2GB, -1 }={ 14GB, 2 }.
Optionally, statistics node manages memory space using the polymerization result.
Specifically, statistics node can be used above-mentioned polymerization result and be managed to memory space, including statistics charging, The operation such as dilatation is carried out based on number of threads.
Optionally, statistics node carries out charging based on variable quantity of the polymerization result to memory space.For example, statistics section Above-mentioned polymerization result can be used in point, and the size of the memory space based on occupancy occupies duration or carries out charging using flow. By taking above-mentioned polymerization result { 14GB, 2 } as an example, if according to capacity charging, it is assumed that the expense unit price of every GB memory space is x, Corresponding charging result is 14GB*x, alternatively, if according to capacity and number charging, it is assumed that the expense unit price of every GB memory space It is a, the expense unit price of number is y, then corresponding charging result is: 14GB*a+2*b.
It in the embodiment of the present application, include the one or more object metadatas indexed at the first time by inquiring, then The capacity increment statistical information for calculating each object in one or more objects, the capacity of obtained one or more objects is increased It measures statistical information and using same static fields carries out polymerization processing, obtain within the period that the first time indexes instruction, According to the polymerization result that same static fields count, the management of memory space is finally carried out using polymerization result, is helped to improve The management performance of memory space.Further, in statistics charge system, compared to the mode of foreground synchronized update charging, originally Application embodiment can be used above-mentioned polymerization result and carry out asynchronous statistics charging, reduces the processing affairs of foreground process, helps In raising foreground performance.
Optionally, the method 200 further include: obtaining all one or more including first time index In the case where object metadata, alternatively, reaching the feelings of memory threshold in the corresponding memory headroom of the capacity increment statistical information Under condition, the one or more of object metadatas of lower wall.
Optionally, the memory threshold can be pre-set.
Specifically, in the case that the caching of memory headroom reaches memory threshold, or about the first time rope After the metadata drawn all has been enumerated, global team is added in the charging metadata indexed about the first time by statistics node Column, asynchronous lower wall, and update the corresponding statistics charging metadata of user.
In order to facilitate the understanding of those skilled in the art, being carried out below with reference to process of the Fig. 5 to background update charging metadata Description.As shown in figure 5, statistics node carries out backstage whole scan to distributed objects storage system, and enumerates and all include All metadata (can be multiple users in the metadata of the period) of time index N, for example, the object being listed is Create Time=N or Delete Time=N etc..Then, storage system according to enumerate come object metadata, meter The capacity increment statistical information of this time operation is calculated, the memory space that objects of statistics occupies obtains capacity increment statistical information, then Capacity increment statistical information is added in caching, wherein the information of the same user is stored in identical memory headroom (ratio Such as, in Fig. 5, number 1 indicates the memory headroom of user 1, and number 2 indicates the memory headroom of user 2, and number 3 indicates user's 3 Memory headroom).Polymerization processing is carried out to all requests of each user in Fig. 5, obtains the corresponding polymerization result of each user.It can Selection of land, whether the memory headroom that can also calculate capacity increment statistical information occupancy reaches memory threshold, if capacity increment is united The memory headroom that meter information occupies reaches memory threshold, alternatively, all scanning through all objects including time index N After metadata, then capacity increment statistical information can be added in asynchronous lower wall queue.Here it is possible to will be held using poll function The file pointer of amount accrual accounting information is added in asynchronous lower wall queue, and line asynchronous of going forward side by side updates, and finally uses each user Polymerization result carry out metadata charging.
The method that the management memory space according to the embodiment of the present application is described in detail above in association with Fig. 1 to Fig. 5.Below will In conjunction with Fig. 6 and Fig. 7 description according to device/equipment of the management memory space of the embodiment of the present application.It should be understood that embodiment of the method institute The technical characteristic of description is equally applicable to following Installation practice.
Fig. 6 shows the schematic block diagram of the device 600 of the management memory space according to the embodiment of the present application, device 600 It can execute and count step performed by node in the above method, therefore be referred to foregoing teachings, be not detailed herein.Distribution It include multiple service nodes in formula object storage system, the service node is used for storage object;The corresponding object meta number of object According to can be stored by service node or metadata node.Described device 600 can be realized in a manner of software and/or hardware, right This is not construed as limiting.As shown in fig. 6, described device 600 includes:
Enquiry module 610 includes all object metadatas indexed at the first time for inquiring, the first time index The period being used to indicate where the time for executing management operation to the object;
Computing module 620, for calculating the capacity increment statistical information of each object in one or more of objects, institute State the size that capacity increment statistical information includes object, wherein the capacity increment statistical information is by the corresponding object of object Metadata record;
Processing module 630, for the capacity increment statistical information of obtained one or more objects to be used same statistics Field carries out polymerization processing, obtains polymerization result, and the polymerization result includes: the period in first time index instruction The variable quantity of the memory space of one or more objects occupancy interior, according to static fields statistics.
In an optional implementation manner, described device 600 further include:
Module is obtained, for obtaining before enquiry module inquiry includes all object metadatas indexed at the first time Object Management group from the user is taken to request, the Object Management group request executes management operation to the first object for requesting, described First object is an object in one or more of objects, and the management operation includes any one of following operation: New Object, deletes object at upload object;
Distribution module, for according to described in the period distribution where the time when getting Object Management group request It indexes at the first time;
Generation module, for requesting the metadata of the first object of generation based on the Object Management group, wherein described first pair It include indexing the first time in the metadata of elephant;
Memory module, for storing the first time index.
In an optional implementation manner, the capacity for one or more objects that the processing module 630 is used to arrive Accrual accounting information carries out polymerization processing using same static fields, specifically includes,
For meeting the object of the static fields within the period of first time index instruction:
The memory space that the object that cumulative new Object operation creates occupies;
Cumulative upload object operates the memory space that uploaded object occupies;
Subtract the memory space of deleted object release.
In an optional implementation manner, described device 600 further include:
Accounting module, for carrying out charging based on variable quantity of the polymerization result to memory space.
It should be understood that can correspond to preceding method embodiment according to the device 600 of the management memory space of the embodiment of the present application (including the method in Fig. 4), and above and other management operation and/or function of the modules in device 600 is respectively The corresponding steps of the aforementioned each method of realization, therefore the beneficial effect in preceding method embodiment also may be implemented, for letter It is clean, it does not repeat here.
Above-mentioned enquiry module 610, computing module 620 and processing module 630 can be realized by the processor 701 in Fig. 7. Optionally, obtaining module can be realized by the input/output interface 703 in Fig. 7, distribution module, accounting module and generation mould Block can also realize that memory module can be realized by the memory 702 in Fig. 7 by above-mentioned processor 701.
Fig. 7 shows the schematic block diagram of the statistics node 700 according to the embodiment of the present application.By executing program generation Code, statistics node 700, which can execute, counts step performed by node in the above method, therefore the present embodiment is not detailed.Such as Shown in Fig. 7, the statistics node 700 includes:
Processor 701, memory 702 and input/output interface 703.
It is communicated with each other, is passed by internal connecting path between processor 701, memory 702 and input/output interface 703 Pass control and/or data-signal.In a possible design, processor 701, memory 702 and input/output interface 703 It can be realized by chip.The memory 702 can store program code, the program that processor 701 calls memory 702 to store Code, to realize the corresponding function of the statistics node.
The processor 701 is used for: by background process using at the first time index to distributed objects storage system into Row whole scan, to inquire all object metadatas for including index at the first time, first time index is used to indicate pair Object executes the period where the time of management operation, and the object metadata is the corresponding object meta of one or more objects Data;Calculate the capacity increment statistical information of each object in one or more of objects, the capacity increment statistical information Size including object, wherein the capacity increment statistical information is recorded by the corresponding object metadata of object;It will obtain The capacity increment statistical informations of one or more objects carry out polymerization processing using same static fields, obtain polymerization result, The polymerization result includes: within the period of first time index instruction, according to one of static fields statistics Or the variable quantity of the memory space of multiple objects occupancy.Optionally, the processor 701 using the polymerization result for being managed Memory space.
The processor 701 is for executing relative program, to realize the revealed method of the embodiment of the present application.It was realizing Cheng Zhong, each step of above method embodiment can pass through the integrated logic circuit of the hardware in processor or software form Instruction is completed.Above-mentioned processor 701 can be general processor, digital signal processor (digital signal Processor, DSP), it is specific integrated circuit (application specific integrated circuit, ASIC), existing At programmable gate array (field programmable gate array, FPGA) or other programmable logic device, discrete Door or transistor logic, discrete hardware components, can also be System on Chip/SoC (system on chip, SoC), can be with It is central processing unit (central processor unit, CPU), can also be network processing unit (network Processor, NP), it can also be digital signal processing circuit (digital signal processor, DSP), can also be Microcontroller (micro controller unit, MCU), can also be programmable controller (programmable logic Device, PLD) or other integrated chips.May be implemented or execute disclosed each method in the embodiment of the present application, step and Logic diagram.General processor can be microprocessor or the processor is also possible to any conventional processor etc..In conjunction with The step of method disclosed in the embodiment of the present application, can be embodied directly in hardware decoding processor and execute completion, or with decoding Hardware and software module combination in processor execute completion.Software module can be located at random access memory, flash memory, read-only storage In the storage medium of this fields such as device, programmable read only memory or electrically erasable programmable memory, register maturation.It should Storage medium can be located at memory 702, and processor 701 reads the information in memory 702, complete the above method in conjunction with its hardware The step of.
The memory 702 is used to store: the program code run for processor 701.Memory 702 can be interior It deposits or solid-state hard disk SSD.It is appreciated that the memory 702 in the embodiment of the present application can be volatile memory or non-easy The property lost memory, or may include both volatile and non-volatile memories.Wherein, nonvolatile memory can be read-only deposit Reservoir (read-only memory, ROM), erasable is compiled programmable read only memory (programmable ROM, PROM) Journey read-only memory (erasable PROM, EPROM), electrically erasable programmable read-only memory (electrically EPROM, EEPROM) or flash memory.Volatile memory can be random access memory (random access memory, RAM), it is used as External Cache.By exemplary but be not restricted explanation, the RAM of many forms is available, such as static Random access memory (static RAM, SRAM), dynamic random access memory (dynamic RAM, DRAM), synchronous dynamic Random access memory (synchronous DRAM, SDRAM), double data speed synchronous dynamic RAM (double data rate SDRAM, DDR SDRAM), enhanced Synchronous Dynamic Random Access Memory (enhanced SDRAM, ESDRAM), synchronized links dynamic random access memory (synchlink DRAM, SLDRAM) and direct rambus Random access memory (direct rambus RAM, DR RAM).It should be noted that the memory of system and method described herein It is intended to include but is not limited to the memory of these and any other suitable type.
In some cases, together with processor 701 can integrate with memory 702.
As another form of the present embodiment, a kind of computer readable storage medium is provided, is stored thereon with instruction, it should Instruction is performed the method for executing and managing memory space in above method embodiment.Computer readable storage medium is for example interior It deposits, hard disk, readable optical disk, flash disk etc..
As another form of the present embodiment, a kind of computer program product comprising instruction is provided, which is held The method that memory space is managed in above method embodiment is executed when row.
The method that above-mentioned the embodiment of the present application discloses can be applied in processor, or be realized by processor.Processor It may be a kind of IC chip, the processing capacity with signal.
Those of ordinary skill in the art may be aware that list described in conjunction with the examples disclosed in the embodiments of the present disclosure Member and algorithm steps can be realized with the combination of electronic hardware or computer software and electronic hardware.These functions are actually It is implemented in hardware or software, the specific application and design constraint depending on technical solution.Professional technician Each specific application can be used different methods to achieve the described function, but this realization is it is not considered that exceed Scope of the present application.
It should be understood that magnitude of the sequence numbers of the above procedures are not meant to execute suitable in the various embodiments of the application Sequence it is successive, the execution of each process sequence should be determined by its function and internal logic, the implementation without coping with the embodiment of the present invention Process constitutes any restriction.
It is apparent to those skilled in the art that for convenience and simplicity of description, the system of foregoing description, The specific work process of device and unit, can refer to corresponding processes in the foregoing method embodiment, and details are not described herein.
In several embodiments provided herein, it should be understood that disclosed systems, devices and methods, it can be with It realizes by another way.For example, the apparatus embodiments described above are merely exemplary, for example, the unit It divides, only a kind of logical function partition, there may be another division manner in actual implementation, such as multiple units or components It can be combined or can be integrated into another system, or some features can be ignored or not executed.Another point, it is shown or The mutual coupling, direct-coupling or communication connection discussed can be through some interfaces, the indirect coupling of device or unit It closes or communicates to connect, can be electrical property, mechanical or other forms.
The unit as illustrated by the separation member may or may not be physically separated, aobvious as unit The component shown may or may not be physical unit, it can and it is in one place, or may be distributed over multiple In network unit.It can select some or all of unit therein according to the actual needs to realize the mesh of this embodiment scheme 's.
It, can also be in addition, each functional unit in each embodiment of the application can integrate in one processing unit It is that each unit physically exists alone, can also be integrated in one unit with two or more units.
It, can be with if the function is realized in the form of SFU software functional unit and when sold or used as an independent product It is stored in a computer readable storage medium.Based on this understanding, the technical solution of the application is substantially in other words The part of the part that contributes to existing technology or the technical solution can be embodied in the form of software products, the meter Calculation machine software product is stored in a storage medium, including some instructions are used so that a computer equipment (can be a People's computer, server or network equipment etc.) execute each embodiment the method for the application all or part of the steps. And storage medium above-mentioned includes: USB flash disk, mobile hard disk, read only memory ROM, random access memory ram, magnetic disk or light The various media that can store program code such as disk.
The above, the only specific embodiment of the application, but the protection scope of the application is not limited thereto, it is any Those familiar with the art within the technical scope of the present application, can easily think of the change or the replacement, and should all contain Lid is within the scope of protection of this application.Therefore, the protection scope of the application should be based on the protection scope of the described claims.

Claims (10)

1. a kind of method for managing memory space, which is characterized in that it include multiple service nodes in distributed objects storage system, The service node is used for storage object, which comprises
Inquiry includes all object metadatas of index at the first time, and the first time index, which is used to indicate, holds the object Period where the time of row management operation, the object metadata is the corresponding object metadata of one or more objects;
Calculate the capacity increment statistical information of each object in one or more of objects, the capacity increment statistical information packet Include the size of object, wherein the capacity increment statistical information is recorded by the corresponding object metadata of object;
The capacity increment statistical information of obtained one or more objects is subjected to polymerization processing using same static fields, is obtained Polymerization result, the polymerization result include: to unite within the period of first time index instruction, according to the static fields The variable quantity for the memory space that one or more objects of meter occupy.
2. the method according to claim 1, wherein including all objects indexed at the first time in the inquiry Before metadata, the method also includes:
Object Management group request from the user is obtained, the Object Management group request executes management behaviour to the first object for requesting Make, first object is an object in one or more of objects, and the management operation includes in following operation Any one: new Object, deletes object at upload object;
The first time index is distributed according to the period where the time when getting Object Management group request;
The metadata of the first object of generation is requested based on the Object Management group, wherein include in the metadata of first object The first time index;
Store the first time index.
3. method according to claim 1 or 2, which is characterized in that the capacity of the one or more objects that will be obtained Accrual accounting information carries out polymerization processing using same static fields, including,
For meeting the object of the static fields within the period of first time index instruction:
The memory space that the object that cumulative new Object operation creates occupies;
Cumulative upload object operates the memory space that uploaded object occupies;
Subtract the memory space of deleted object release.
4. according to the method in any one of claims 1 to 3, which is characterized in that the method also includes:
Charging is carried out based on variable quantity of the polymerization result to memory space.
5. a kind of statistics node, for the space hold of object in statistical cloth object storage system, the statistics node packet Processor, storage medium and interface are included, computer program is stored in the storage medium, the interface connects for correspondence with foreign country It connects, the processor executes following methods by running the computer program:
Inquiry includes all object metadatas of index at the first time, and the first time index, which is used to indicate, holds the object Period where the time of row management operation, the object metadata is the corresponding object metadata of one or more objects;
Calculate the capacity increment statistical information of each object in one or more of objects, the capacity increment statistical information packet Include the amount of capacity of object, wherein the capacity increment statistical information is recorded by the corresponding object metadata of object;
The capacity increment statistical information of obtained one or more objects is subjected to polymerization processing using same static fields, is obtained Polymerization result, the polymerization result include: to unite within the period of first time index instruction, according to the static fields The variable quantity for the memory space that one or more objects of meter occupy.
6. a kind of device for managing memory space, which is characterized in that it include multiple service nodes in distributed objects storage system, The service node is used for storage object, and described device includes:
Enquiry module includes all object metadatas indexed at the first time for inquiring, and the first time index is for referring to Show the period where the time for executing management operation to the object;
Computing module, for calculating the capacity increment statistical information of each object in one or more of objects, the capacity Accrual accounting information includes the size of object, wherein the capacity increment statistical information is by the corresponding object metadata of object Record;
Processing module, for carrying out the capacity increment statistical information of obtained one or more objects using same static fields Polymerization processing, obtains polymerization result, and the polymerization result includes: within the period of first time index instruction, basis The variable quantity for the memory space that one or more objects of the static fields statistics occupy.
7. device according to claim 6, which is characterized in that described device further include:
Module is obtained, for obtaining and before enquiry module inquiry includes all object metadatas indexed at the first time It is requested from the Object Management group of user, the Object Management group request executes management operation to the first object for requesting, and described first Object is an object in one or more of objects, and the management operation includes any one of following operation: newly-built Object, deletes object at upload object;
Distribution module, for according to the period distribution described first where the time when getting Object Management group request Time index;
Generation module, for requesting the metadata of the first object of generation based on the Object Management group, wherein first object It include indexing the first time in metadata;
Memory module, for storing the first time index.
8. device according to claim 6 or 7, which is characterized in that the processing module is used for the one or more that will be arrived The capacity increment statistical information of object carries out polymerization processing using same static fields, specifically includes,
For meeting the object of the static fields within the period of first time index instruction:
The memory space that the object that cumulative new Object operation creates occupies;
Cumulative upload object operates the memory space that uploaded object occupies;
Subtract the memory space of deleted object release.
9. the device according to any one of claim 6 to 8, which is characterized in that described device further include:
Accounting module, for carrying out charging based on variable quantity of the polymerization result to memory space.
10. a kind of computer readable storage medium, which is characterized in that it is stored with program in the computer readable storage medium, The method that described program makes the device for managing memory space execute any one of the claims 1 to 4 management memory space.
CN201810804528.2A 2018-07-20 2018-07-20 The method and apparatus for managing memory space Pending CN109241178A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201810804528.2A CN109241178A (en) 2018-07-20 2018-07-20 The method and apparatus for managing memory space

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201810804528.2A CN109241178A (en) 2018-07-20 2018-07-20 The method and apparatus for managing memory space

Publications (1)

Publication Number Publication Date
CN109241178A true CN109241178A (en) 2019-01-18

Family

ID=65072798

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201810804528.2A Pending CN109241178A (en) 2018-07-20 2018-07-20 The method and apparatus for managing memory space

Country Status (1)

Country Link
CN (1) CN109241178A (en)

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110888872A (en) * 2019-10-16 2020-03-17 浙江大搜车软件技术有限公司 Data storage method and device, computer equipment and storage medium
CN111221824A (en) * 2020-01-02 2020-06-02 广州虎牙科技有限公司 Storage optimization method, device, equipment and medium of storage space
CN111241097A (en) * 2020-01-08 2020-06-05 北京松果电子有限公司 Method of processing object, apparatus for processing object, and storage medium
CN111723202A (en) * 2020-05-19 2020-09-29 支付宝(杭州)信息技术有限公司 Public opinion data processing device, method and system
CN111797174A (en) * 2019-04-08 2020-10-20 华为技术有限公司 Method and apparatus for managing spatiotemporal data
CN111881133A (en) * 2020-07-24 2020-11-03 苏州浪潮智能科技有限公司 Storage bucket management method and device, computer equipment and readable storage medium
CN113064859A (en) * 2021-03-26 2021-07-02 山东英信计算机技术有限公司 Metadata processing method and device, electronic equipment and storage medium
CN113111194A (en) * 2021-04-07 2021-07-13 山东英信计算机技术有限公司 Object metadata aggregation method, reading method, device, equipment and storage medium
CN113177068A (en) * 2021-03-15 2021-07-27 新华三信息安全技术有限公司 Aggregated data query method, equipment and medium
CN116521094A (en) * 2023-07-03 2023-08-01 之江实验室 Metadata storage method and device, computer equipment and storage medium

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN85100285A (en) * 1985-04-01 1985-09-10 北京大学 High resolution chinese character pattern gerenator
CN85103698A (en) * 1984-06-22 1986-11-12 北方电信有限公司 Send digital signal and additional signal
CN103218175B (en) * 2013-04-01 2015-10-28 无锡成电科大科技发展有限公司 The cloud storage platform access control system of many tenants
CN107633045A (en) * 2017-09-15 2018-01-26 龙存(苏州)科技有限公司 The statistical method and its system of tenant data capacity in a kind of cloud storage service

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN85103698A (en) * 1984-06-22 1986-11-12 北方电信有限公司 Send digital signal and additional signal
CN85100285A (en) * 1985-04-01 1985-09-10 北京大学 High resolution chinese character pattern gerenator
CN103218175B (en) * 2013-04-01 2015-10-28 无锡成电科大科技发展有限公司 The cloud storage platform access control system of many tenants
CN107633045A (en) * 2017-09-15 2018-01-26 龙存(苏州)科技有限公司 The statistical method and its system of tenant data capacity in a kind of cloud storage service

Cited By (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111797174A (en) * 2019-04-08 2020-10-20 华为技术有限公司 Method and apparatus for managing spatiotemporal data
CN110888872A (en) * 2019-10-16 2020-03-17 浙江大搜车软件技术有限公司 Data storage method and device, computer equipment and storage medium
CN111221824A (en) * 2020-01-02 2020-06-02 广州虎牙科技有限公司 Storage optimization method, device, equipment and medium of storage space
CN111241097A (en) * 2020-01-08 2020-06-05 北京松果电子有限公司 Method of processing object, apparatus for processing object, and storage medium
CN111241097B (en) * 2020-01-08 2024-03-12 北京小米松果电子有限公司 Method for processing object, device for processing object and storage medium
CN111723202A (en) * 2020-05-19 2020-09-29 支付宝(杭州)信息技术有限公司 Public opinion data processing device, method and system
CN111881133B (en) * 2020-07-24 2022-12-06 苏州浪潮智能科技有限公司 Storage bucket management method and device, computer equipment and readable storage medium
CN111881133A (en) * 2020-07-24 2020-11-03 苏州浪潮智能科技有限公司 Storage bucket management method and device, computer equipment and readable storage medium
CN113177068A (en) * 2021-03-15 2021-07-27 新华三信息安全技术有限公司 Aggregated data query method, equipment and medium
CN113064859A (en) * 2021-03-26 2021-07-02 山东英信计算机技术有限公司 Metadata processing method and device, electronic equipment and storage medium
CN113111194A (en) * 2021-04-07 2021-07-13 山东英信计算机技术有限公司 Object metadata aggregation method, reading method, device, equipment and storage medium
CN116521094A (en) * 2023-07-03 2023-08-01 之江实验室 Metadata storage method and device, computer equipment and storage medium
CN116521094B (en) * 2023-07-03 2023-11-14 之江实验室 Metadata storage method and device, computer equipment and storage medium

Similar Documents

Publication Publication Date Title
CN109241178A (en) The method and apparatus for managing memory space
US11146502B2 (en) Method and apparatus for allocating resource
US9372880B2 (en) Reclamation of empty pages in database tables
CN104067216B (en) System and method for implementing expansible data storage service
CN106294352B (en) A kind of document handling method, device and file system
US9965207B2 (en) Maintenance of cloned computer data
US9348855B2 (en) Supporting big data in enterprise content management systems
US20190026317A1 (en) Memory use in a distributed index and query system
EP3958122A1 (en) Memory management method, apparatus, and system
CN104021451A (en) Mixed enterprise content management method and system based on cloud storage and local storage
CN110287201A (en) Data access method, device, equipment and storage medium
EP3103039B1 (en) System and method for an efficient database storage model based on sparse files
CN110968589A (en) Self-adding ID management method and device in distributed system
CN105354090B (en) The management method and device of virtual unit
US9380126B2 (en) Data collection and distribution management
CN113448946B (en) Data migration method and device and electronic equipment
CN110457307B (en) Metadata management system, user cluster creation method, device, equipment and medium
CN108574718A (en) A kind of cloud host creation method and device
CN109144689B (en) Task scheduling method, device and computer program product
CN109725844B (en) Disk allocation method, device and storage system
US20230063541A1 (en) Determining computer resource usage at multiple levels of a container orchestration system hierarchy
CN108604231A (en) Mirror processing method and computing device
CN111061888B (en) Image acquisition method and system
CN110554915A (en) Information processing method and device, computer storage medium and terminal
CN113890872B (en) Data set uploading method and device, electronic equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination