CN109408571A - Object enumerates method, equipment and computer storage medium - Google Patents
Object enumerates method, equipment and computer storage medium Download PDFInfo
- Publication number
- CN109408571A CN109408571A CN201811146403.1A CN201811146403A CN109408571A CN 109408571 A CN109408571 A CN 109408571A CN 201811146403 A CN201811146403 A CN 201811146403A CN 109408571 A CN109408571 A CN 109408571A
- Authority
- CN
- China
- Prior art keywords
- interval
- interval mark
- enumerates
- bucket
- result
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Abstract
The embodiment of the present application discloses a kind of object and enumerates method, equipment and computer storage medium, wherein object enumerates method and enumerates equipment applied to object, it include: to be sampled at a predetermined interval to the key of the object in bucket to obtain interval mark set, wherein the interval mark set includes multiple interval marks, and the multiple interval mark is the sampling result being sampled to the key of the object in the bucket；It synchronizes and enumerates in the interval mark set any adjacent corresponding object of interval mark two-by-two, multiple enumerate object result to obtain；Finally, merging is the multiple to enumerate object result.Wherein, whole objects of bucket can be divided to multiple sections by multiple interval marks in interval mark set, any adjacent corresponding object of interval mark two-by-two is enumerated in interval mark set by synchronizing, the application can simultaneously enumerate the object in each section in bucket, and then quickly enumerate whole objects in bucket.
Description
Technical field
The present invention relates to interface calling technology fields more particularly to a kind of object to enumerate method, equipment and computer storage
Medium.
Background technique
Bucket is the container that storage object is used in object storage, and in bucket, object is carried out unique by key (key)
Ground mark.Enumerating object (list objects) interface and can enumerate object in bucket in object interface.It is calling
When enumerating object interface, at least need mark value (marker) entering ginseng as enumerate object interface, marker is used to indicate
The initial position once enumerated.
In order to obtain whole objects in bucket, current scheme is that serial loop calling enumerates object interface until column
The return value for lifting object interface is sky.Specifically, " null value is set by marker when calling enumerates object interface for the first time
(null) " marker is set to when, subsequent each calling enumerates interface on the key for the last one object that last time calling returns, directly
It is sky to the return value for enumerating object interface, then calls the object returned to merge for each, just obtained in bucket
Whole objects.
Due to return value of the above scheme when recursive call enumerates object interface dependent on last time calling, and adjust every time
It is definite value with the object number that can at most return, therefore it is very low to enumerate efficiency.As an example it is assumed that interface when calling every time
Time of return be 0.5 second, every time call at most return 1000 objects, need to call then enumerating 10,000,000 objects
10000 interfaces need 10000*0.5=5000 seconds time-consuming.
Summary of the invention
The application provides a kind of object and enumerates method, equipment and computer storage medium, can solve and arranges in the prior art
Enumerate the problem of object in bucket takes a long time.
In a first aspect, this application provides a kind of objects to enumerate method, which enumerates method and includes:
The key of the object in bucket is sampled to obtain interval mark set at a predetermined interval, wherein the section
Tag set includes multiple interval marks, and the multiple interval mark is to be sampled to the key of the object in the bucket
Sampling result；It synchronizes and enumerates in the interval mark set any adjacent corresponding object of interval mark two-by-two, it is more to obtain
It is a to enumerate object result；Finally, merging is the multiple to enumerate object result.
As it can be seen that whole objects of bucket can be divided to several areas by multiple interval marks in interval mark set
Between, any adjacent corresponding object of interval mark two-by-two is enumerated in interval mark set by synchronizing, it can bucket simultaneously
In the object in each section enumerated, and then quickly enumerate whole objects in bucket, shortening is enumerated complete in bucket
Time required for portion's object.
With reference to first aspect, in the first possible implementation of the first aspect, described synchronize enumerates the section
Any adjacent corresponding object of interval mark two-by-two in tag set, comprising:
The multiple interval mark is respectively allocated to multiple processing units, wherein at an interval mark is one corresponding
Manage unit so that the multiple processing unit using interval mark corresponding with itself as enumerate object interface enter join, from
Enumerated in the bucket object until in the object result enumerated include the interval mark set in it is corresponding with itself
The object number for including in the object that the different interval mark of interval mark is identified, or the object result until enumerating
Less than the maximum return value for enumerating object interface.
The possible implementation of with reference to first aspect the first, in second of possible implementation of first aspect
In, first interval mark in the multiple interval mark is null value.
Second of possible implementation of the possible implementation of with reference to first aspect the first and first aspect,
In the third possible implementation of first aspect, the merging is the multiple to be enumerated after object result, and the method is also
It include: that duplicate removal is carried out to the multiple object result of enumerating after merging.
As it can be seen that by implementing above-mentioned optional embodiment, the application is called by multiple processing unit for parallel to be enumerated pair
As interface to enumerate in interval mark set any adjacent corresponding object of interval mark two-by-two, it can quickly enumerate and deposit
Whole objects in storage tank, so as to shorten the time required for whole objects in bucket is enumerated.
Second aspect, this application provides a kind of objects to enumerate equipment, and it includes above-mentioned for executing which, which enumerates equipment,
The module of each possible method implementation of the method and first aspect of first aspect.Specifically, which enumerates equipment packet
It includes:
Sampling module is sampled to obtain interval mark collection for the key at a predetermined interval to the object in bucket
It closes, wherein the interval mark set includes multiple interval marks, the multiple interval mark is to pair in the bucket
The sampling result that the key of elephant is sampled；Enumerate module, for synchronize enumerate in the interval mark set it is any adjacent two-by-two
The corresponding object of interval mark, multiple enumerate object result to obtain；Merging module the multiple enumerates object for merging
As a result.
In conjunction with second aspect, in the first possible implementation of the second aspect, the synchronization module is specifically used for:
The multiple interval mark is respectively allocated to multiple processing units, wherein at an interval mark is one corresponding
Manage unit so that the multiple processing unit using interval mark corresponding with itself as enumerate object interface enter join, from
Enumerated in the bucket object until in the object result enumerated include the interval mark set in it is corresponding with itself
The object number for including in the object that the different interval mark of interval mark is identified, or the object result until enumerating
Less than the maximum return value for enumerating object interface.
In conjunction with the possible implementation of the first of second aspect and second aspect, second in second aspect is possible
In implementation, first interval mark in the multiple interval mark is null value.
It is possible in conjunction with the first possible implementation of second aspect, second aspect and second of second aspect
Implementation, in the third possible implementation of the second aspect, the object enumerate equipment further include: deduplication module is used
Duplicate removal is carried out in enumerating object result to the multiple after merging.
Based on the same inventive concept, enumerating the principle that equipment solves the problems, such as and beneficial effect due to the object may refer to
The method of above-mentioned first aspect and each possible implementation of first aspect and brought beneficial effect, therefore the object
The implementation for enumerating equipment may refer to the method for above-mentioned first aspect and each possible implementation of first aspect, repeat place
It repeats no more.
The third aspect, this application provides a kind of objects to enumerate equipment, and it includes: memory which, which enumerates equipment, is used for
Store one or more programs；Processor, for calling the program being stored in the memory to realize above-mentioned first aspect
Scheme in method design, which enumerates the embodiment that equipment solves the problems, such as and beneficial effect may refer to above-mentioned first
The method of aspect and each possible implementation of first aspect and brought beneficial effect, overlaps will not be repeated.
Fourth aspect, this application provides a kind of computer readable storage medium, which is stored with meter
Calculation machine program, the computer program include computer instruction, which when being executed by a processor hold the processor
The method of the above-mentioned first aspect of row and each possible implementation and the brought beneficial effect of first aspect, repeat place
It repeats no more.
5th aspect, this application provides a kind of computer program product, which includes that computer refers to
Enable, the computer instruction make when being executed by a processor the processor execute above-mentioned first aspect method and first aspect it is each
Possible implementation and brought beneficial effect, overlaps will not be repeated.
Detailed description of the invention
Fig. 1 is the schematic diagram of bucket and object in the embodiment of the present application；
Fig. 2 is the flow diagram that a kind of object provided by the embodiments of the present application enumerates method；
Fig. 3 is the block schematic illustration that a kind of object provided by the embodiments of the present application enumerates equipment；
Fig. 4 is the block schematic illustration that another object provided by the embodiments of the present application enumerates equipment.
Specific embodiment
Below in conjunction with the attached drawing in the embodiment of the present application, the embodiment of the present application is illustrated.
Referring to Figure 1, Fig. 1 is the schematic diagram of bucket and object.Wherein, bucket (bucket) 11 is in object storage
Container for storage object (Object) 12.As shown in Figure 1, each object 12 is stored in a bucket 11.For example,
The object 12 of entitled photos/puppy.jpg is stored in johnsmith bucket 11, URL (Uniform may be used
Resource Locator, uniform resource locator) be http://johnsmith.xx/photos/puppy.jpg link
The object 12 is addressed.Wherein, xx is manufacturer's domain name.
Object 12 is the primary entity of object storage, and object 12 is made of object data and metadata.Wherein, data portion
Opaque to S3, metadata is the name-value pair of one group of description object 12.Metadata may include default meta data (such as last time
Modify the date) and standard HTTP (HyperText Transfer Protocol, hypertext transfer protocol) metadata (such as content-
Type).
In bucket, object 12 will be uniquely identified by key (key) and version number's (revision ID), the key of object 12
That is the title of object.Key refers to the unique identifier of object 12 in bucket 11, and each object 12 in bucket 11 can only
There is a key.Each object 12 can be uniquely identified due to combining bucket 11, key and revision ID, it can will be right
It is mapped as storing the master data being considered as between a kind of " bucket 11+ key+version " and object 12 itself.Network (Web) is serviced
Terminal node, bucket name, key and revision ID (optional) are combined, can uniquely addressing object storage in each object
12。
In object storage, object (list objects) interface is enumerated for enumerating in bucket 11 object 12.Its
In, this, which is enumerated when object interface is called, once at most can return to 1000 objects 12.This enumerates the object that object interface is enumerated
May include in 12 the key of the object 12, size, newest modification time, by MD5 (Message Digest Algorithm 5,
5th edition Message Digest 5) information such as hashed value (i.e. MD5 value) that generate.
Wherein, the suction parameter (hereinafter referred to as entering ginseng) for enumerating object interface includes label (marker).Marker is
The key of one object 12, for specifying the initial position of one query, this enumerates pair of object interface return after marker is arranged
As result will be the later all objects 12 of the key is identified after sorting according to lexicographic order object 12.
The ginseng that enters for enumerating object interface further includes maximum return value (max-keys).Max-keys is enumerated pair for specifying this
As the maximum object count that interface can once return, after setting max-keys, the object, which will enumerate the object result of interface return, to be
According to most preceding max-keys objects 12 of lexicographic order.Wherein, the numberical range of max-keys is [1, n].When not right
When max-keys is configured, the default number of max-keys is n.When the max-keys of setting exceeds n, according to the n of default
It is handled, wherein n is positive integer, it is generally the case that the value of n is 1000.
In embodiments herein, which, which enumerates equipment, to enumerate object interface by serial loop calling first
To enumerate the list of the key of object in the bucket.Serial loop calling is described below and enumerates object interface to enumerate the storage
The method of the list of object key in bucket.
Specifically, when first passage calls this to enumerate object interface, which enumerates equipment and sets empty for marker
It is worth (null), is enumerated since first object in bucket；When in subsequent each calling, this enumerates object interface, the object
Enumerate equipment by marker be set as last time interface call when this enumerate object interface return the last one object key.This is right
As enumerating equipment, serial loop calls this to enumerate object interface always, until enumerating the last one object in the bucket.
When determining that the object enumerates equipment, and to enumerate object interface each by this when having enumerated the last one object in the bucket
The object result of return merges, and obtains whole objects in the bucket.
In embodiments herein, when the object number for including in the object result for enumerating object interface return to this is small
When this enumerates maximum return value (max-keys) of object interface, which, which enumerates equipment, can determine that having enumerated this deposits
The last one object in storage tank.
It is appreciated that in order to maximize the efficiency that object is enumerated, when calling this to enumerate object interface every time, the object column
Lift equipment can set the numerical value of max-keys to not less than n integer or the object enumerate equipment can not be to max-
Keys is configured, so that this, which enumerates object interface, once can at most return to n object.In this case, when by being somebody's turn to do
When first thread detects that this enumerates the object number for including in the object result of object interface return less than n, which is enumerated
Equipment can determine the last one object enumerated in the bucket.
In practice, the operation of the additions and deletions to the object in bucket is usually had, for example, increase object in bucket,
Or the partial objects in deletion bucket.The embodiment of the present invention provides a kind of object and enumerates method, which enumerates method can
It is enumerated in equipment with being applied particularly to object, it is the demand method, apparatus that object is enumerated which, which enumerates equipment, for enumerating storage
The object after additions and deletions is carried out in bucket.Include the following steps: as shown in Fig. 2, the object enumerates method
S21, the key of the object in bucket is sampled at a predetermined interval to obtain interval mark set.Wherein, should
Predetermined space is interval when being sampled to key.The interval mark set includes multiple interval marks, multiple section mark
Note includes the sampling result being sampled to the key of the object in the bucket.
The key of object can enumerate object interface and obtain by serially calling in bucket, and specific method is seen above, and repeat
Place repeats no more.
After the key for enumerating the object in the bucket, which enumerates equipment can be at a predetermined interval to the storage
The key of object in bucket is sampled to obtain interval mark set.
Wherein, which enumerates equipment execution and is at a predetermined interval sampled the key of the object in the bucket to obtain
Interval mark set can specifically include: be sampled to obtain sampling knot at a predetermined interval to the key of the object in the bucket
Fruit, the sampling result are the key of one or more objects；Using the key of the one or more object as interval mark, section is generated
Tag set.
As it can be seen that including the sampling result being sampled to the key of the object in the bucket in the interval mark set.?
It further include null value in the interval mark set in embodiments herein.To include multiple sections in the interval mark set
Label.
In embodiments herein, first interval mark in multiple interval mark is null value, this or more
Sequencing of the key of a object in the interval mark set is identical as its sequencing in the bucket.
Wherein, which is the sampling interval of key.
It for example,, should if the predetermined space is 10000 when the object number for including in the bucket is 20000
Sampling result is the key of the 10000th object in the bucket；If the predetermined space is 5000, which deposits for this
The key of the key of the 5000th object in storage tank, the key of the 10000th object and the 15000th object.
In embodiments herein, which according to the object number for including in the bucket and can be deposited this
Storage tank executes the processing unit number that interval plants are enumerated and determines, to maximize the efficiency that object is enumerated.
For example, the place that interval plants are enumerated is executed when the object number for including in the bucket and to the bucket
When reason number of unit is respectively 20000 and 2, which can be set to 10000 to divide equally the calling for enumerating object interface
To two processing units to maximize the efficiency that object is enumerated.
It is appreciated that whole objects in the bucket are divided into multiple sections by the key of the one or more object, it should
Each of interval mark set interval mark corresponds to a section.
In embodiments herein, when in the interval mark set only include an object key when, null value and should
The key of object can respectively as in the interval mark set first interval mark and second interval mark it is (namely last
One interval mark).When in the interval mark set including the key of multiple objects, null value can be used as the interval mark set
In first interval mark, sequencing of the key of multiple object in the interval mark set can be with it in the storage
Sequencing in bucket is identical.
For example, when in the bucket including 20000 objects, if the sampling result is the in the bucket
The key of 10000 objects, then the key of null value and 10000 objects can be respectively as first in the interval mark set
A interval mark and second interval mark (namely the last one interval mark).
When in the bucket including 20000 objects, if the sampling result is the 5000th object in the bucket
Key, the key of the 10000th object and the key of the 15000th object, then null value, the key of the 5000th object, the 10000th
The key of the key of a object and the 15000th object can respectively as in the interval mark set first interval mark,
Second interval mark, third interval mark and the 4th interval mark (namely the last one interval mark).
To which when in the interval mark set including m interval mark, the corresponding section of first interval mark is (i.e.
First section) it include the object that first object in the bucket is identified to second interval mark, m-th of section mark
Remember that corresponding section (i.e. m-th of section) includes the next object for the object that m-th of interval mark is identified to the storage
The last one object in bucket, the corresponding section (i.e. i-th of section) of i-th (1 < i < m) a interval mark includes i-th of section
The object for marking next object of identified object to be identified to i+1 interval mark.
S22, it synchronizes and enumerates in the interval mark set any adjacent corresponding object of interval mark two-by-two, to obtain
It is multiple to enumerate object result.
When in the interval mark set including m interval mark, if adjacent interval mark is first area two-by-two for this
Between label and second interval mark, then this adjacent corresponding object of interval mark includes first in the bucket two-by-two
The object that object is identified to second interval mark；If adjacent interval mark is m-th of interval mark and first two-by-two for this
A interval mark, then this adjacent corresponding object of interval mark includes object that m-th of interval mark is identified two-by-two
Next object is to the last one object in the bucket；If adjacent interval mark is i-th a section (1 < i < m) two-by-two for this
Label and i+1 interval mark, then this adjacent corresponding object of interval mark refers to that i-th of interval mark is marked two-by-two
The object that next object of the object of knowledge is identified to the i+1 interval mark.
That is, above-mentioned any adjacent corresponding object of interval mark two-by-two is the arbitrarily adjacent section two-by-two
The object that the corresponding section of previous interval mark in label includes.
Appoint as an alternative embodiment, the object enumerates equipment execution and synchronizes to enumerate in the interval mark set
The corresponding object of interval mark adjacent two-by-two of anticipating can specifically include: the multiple interval mark is respectively allocated to multiple places
Manage unit, wherein the corresponding processing unit of an interval mark, so that the multiple processing unit will be corresponding with itself
Interval mark as enumerate object interface enter ginseng, wrapped from object is enumerated in the bucket until in the object result enumerated
Include the object that the different interval mark of interval mark corresponding with itself is identified in the interval mark set, or until
The object number for including in the object result enumerated is less than the maximum return value for enumerating object interface.
It is appreciated that interval mark corresponding with itself is used as and enumerates object interface by the synchronous execution of multiple processing unit
Enter ginseng, the step of enumerating object from the bucket.In embodiments herein, multiple processing unit can be more
A thread or multiple processes.
When multiple processing unit is multiple threads, multiple thread execution will interval mark conduct corresponding with itself
That enumerates object interface enters ginseng, and object is enumerated from the bucket to be specifically included: S221, judging that the interval mark set is
No is empty；S222, when the interval mark set is not sky, from the interval mark set obtain target interval label, and will
Target interval label is deleted from the interval mark set；S223, corresponding section is marked to execute section the target interval
Object enumerates to obtain the target interval and corresponding object is marked to enumerate result.
Wherein, multiple thread is deployed in the object and enumerates in equipment.
In embodiments herein, whether which is empty for showing multiple thread to complete
Portion section performs interval plants and enumerates.
Specifically, if the interval mark set is not sky, show that multiple thread does not execute interval plants to whole sections
It enumerates, then multiple thread can execute step S222~S223；If the interval mark collection is combined into sky, show multiple thread pair
Each section has been performed both by interval plants and has enumerated, then the object, which enumerates equipment, can execute step S23.
When any one thread gets target interval label from the interval mark set, target interval label is
As with the corresponding interval mark of the thread itself.Wherein, target interval label can be appointing in the interval mark set
It anticipates an interval mark.As an alternative embodiment, in the available interval mark set of multiple thread
One interval mark is marked as target interval.
After getting target interval label, which can also be marked from the section and be marked by any one thread
It deletes in note set to update the interval mark set, so that other threads be avoided to mark corresponding section to hold the target interval
Row interval plants are enumerated.
In embodiments herein, multiple thread can call this to enumerate object interface to the target with serial loop
The corresponding section of interval mark, which executes interval plants, to be enumerated to obtain the target interval corresponding object is marked to enumerate result.
Specifically, when calling this to enumerate object interface for the first time, multiple thread can set marker to the target
Interval mark；When in subsequent each recursive call, this enumerates object interface, multiple thread can set last time for marker
The key for enumerating the last one object in the object result of object interface return when interface calls.Multiple thread recycles always
It calls this to enumerate object interface, marks the last one object in corresponding section until enumerating the target interval.Work as determination
When having enumerated the target interval and marking the last one object in corresponding section, show that this interval plants enumerates knot
This is enumerated the object result that object interface returns every time and merged by beam, multiple thread, obtains target interval label pair
The object answered enumerates result.
As an alternative embodiment, multiple thread can detecte the object result for enumerating object interface return
In whether include corresponding with itself interval mark in the interval mark set (i.e. the target interval marks) different section
Mark identified object；If detecting that this is enumerated includes marking not in the object result of object interface return with the target interval
The object that identical interval mark is identified, then multiple thread can determine enumerated the target interval label it is corresponding
The last one object in section.
In a specific embodiment, any adjacent interval mark two-by-two is enumerated in the interval mark set synchronizing
Corresponding object with obtain it is multiple enumerate object result before, which enumerates equipment and can also execute to the interval mark set
Replication processes obtain the first tag set.To which multiple thread executes detection, and this enumerates the object result of object interface return
In whether include corresponding with itself interval mark in the interval mark set (i.e. the target interval marks) different section
Mark identified object that can specifically include: detect in the object result for enumerating object interface return whether include this first
The object for marking different interval mark to be identified with the target interval in interval mark set.
As another optional embodiment, multiple thread may determine that this enumerates the object knot of object interface return
Whether the object number for including in fruit is less than max-keys；Include if judging that this is enumerated in the object result of object interface return
Object number be less than max-keys, then multiple thread can determine that having enumerated the target interval marks corresponding section
In the last one object.
Further, it enumerates to obtain the target area by marking corresponding section to execute interval plants the target interval
Between mark corresponding object to enumerate after result, multiple thread can return to step S221~S223.
When multiple processing unit is multiple processes, multiple process execution will interval mark conduct corresponding with itself
That enumerates object interface enters ginseng, and object is enumerated from the bucket to be specifically included: S224, judging that the interval mark set is
No is empty；S225, when the interval mark set is not sky, from the interval mark set obtain target interval label, and will
Target interval label is deleted from the interval mark set；S226, corresponding section is marked to execute section the target interval
Object enumerates to obtain the target interval and corresponding object is marked to enumerate result；S227, counting is subtracted to the execution of default down counter.
Wherein, multiple process all can be virtual machine process, host processes or container process.
It is enumerated in equipment as an alternative embodiment, multiple process can be deployed in the object.
As another optional embodiment, multiple process can be deployed in other equipment.For example, multiple
Process can be deployed in the computing capability that each equipment is maximally utilized in different equipment respectively.
As another optional embodiment, multiple process can be deployed in the object respectively and enumerate equipment and other
In equipment.It is enumerated in equipment for example, the Partial Process in multiple process can be deployed in the object, other processes can dispose
In other equipment.
Specifically, if the interval mark set is not sky, show that multiple process does not execute interval plants to whole sections
It enumerates, then multiple process can execute step S225~S227；If the interval mark collection is combined into sky, show multiple process pair
Each section has been performed both by interval plants and has enumerated, then the object, which enumerates equipment, can execute step S23.
Wherein, the particular technique details of step S224~S226 can refer to the associated description of step S221~S223, weight
Multiple place repeats no more.
In embodiments herein, after obtaining object and enumerating result, multiple process can also obtain the object
It enumerates the corresponding set of URL of result to close, and the set of URL is closed into storage into default distributed queue.Wherein, it is wrapped in set of URL conjunction
It includes the object and enumerates the corresponding URL of each of result object.
At the end of an interval plants are enumerated, the current value of default down counter is subtracted 1 by multiple process.Wherein,
Initial value is provided in the default down counter.
S23, merge and the multiple enumerate object result.
After performing interval plants to whole sections and enumerating, which, which enumerates equipment, can merge multiple enumerate pair
As result to obtain whole objects in the current bucket.
As an alternative embodiment, merge it is multiple enumerate object result after, which enumerates equipment also
Duplicate removal can be carried out to obtain whole objects in the current bucket to multiple object result of enumerating after merging.
In the case of multiple processing unit is multiple processes, when the current value for presetting down counter is specified number
When value, triggers the object and enumerate equipment and merge and multiple enumerate object result.
Wherein, which is the difference of the initial value and the interval mark number in the interval mark set.It can be with
Understand, when the current value for presetting down counter is the specified numerical value, shows that multiple process holds each section
Interval plants of having gone are enumerated.
In embodiments herein, which, which enumerates equipment, can continue to detect the current number of the default down counter
Value.For example, the object enumerates equipment and can preset down counter to this and is polled.
As an alternative embodiment, the initial value can be set to the interval mark in the interval mark set
Number.Preferably, when the numerical value for presetting down counter is 0, which enumerates equipment can be default from this first
Each set of URL is obtained in distributed queue to close, and is then addressed according to each set of URL conjunction to obtain each object and enumerate knot
Fruit, then result is enumerated to each object of acquisition and is merged, obtain whole objects in the current bucket.
It is appreciated that after obtaining whole objects in the current bucket, the object side of enumerating of the embodiment of the present application
Method can generate new interval mark set according to whole objects in the current bucket, so as to there is object to enumerate need
Step S21~S23 is executed to enumerate the object in the bucket based on new interval mark set in the case of asking.
The object of the embodiment of the present application is enumerated method and is enumerated by the synchronous interval plants that execute of multiple processing units to obtain
Multiple objects are enumerated and obtain whole objects in bucket as a result, being then combined with multiple object and enumerating result.Compared to passing through
One thread serial loop calling enumerates technical solution of the object interface until enumerating whole objects in bucket, the application
Object interface is enumerated by multiple processing unit for parallel calling to enumerate until having executed interval plants to whole sections, it can be quick
Enumerate whole objects in bucket.
It is migrated for object bucket increment, after quickly enumerating whole objects in bucket, the application is implemented
The object of example, which enumerates method, can obtain the increment object set of source bucket by screening the newest modification time of object, and will
The increment object set is synchronized in purpose bucket.If having the object of magnanimity, the embodiment of the present application in the bucket of source
Object enumerate method can greatly shorten obtain increment object set time, so as to shorten the bulk migration of object
Time and cut over break period.
Fig. 3 is referred to, Fig. 3 is the block schematic illustration that a kind of object provided by the embodiments of the present application enumerates equipment 30.Such as Fig. 3
Shown, which, which enumerates equipment 30, may include sampling module 31, enumerates module 32 and merging module 33.
Wherein, the sampling module 31, is sampled for the key at a predetermined interval to the object in bucket to obtain area
Between tag set, wherein the interval mark set includes multiple interval marks, the multiple interval mark is to the storage
The sampling result that the key of object in bucket is sampled.
As an alternative embodiment, first interval mark in the multiple interval mark is null value.
This enumerates module 32, and for synchronizing, to enumerate in the interval mark set any interval mark adjacent two-by-two corresponding
Object, multiple enumerate object result to obtain.
As an alternative embodiment, this enumerates module 32, specifically for dividing the multiple interval mark respectively
It is assigned to multiple processing units, wherein the corresponding processing unit of interval mark, so that the multiple processing unit will be with
Itself corresponding interval mark enters ginseng as enumerate object interface, and object is enumerated from the bucket until pair that enumerates
As pair identified in result including the different interval mark of interval mark corresponding with itself in the interval mark set
As, or the object number for including in the object result until enumerating is less than the maximum return value for enumerating object interface.
The merging module 33 the multiple enumerates object result for merging.
As an alternative embodiment, it can also include deduplication module 34 that the object, which enumerates equipment 30, it to be used for pairing
The multiple object result of enumerating after and carries out duplicate removal.
Based on the same inventive concept, the object provided in the embodiment of the present application enumerate the principle that equipment 30 solves the problems, such as and
Beneficial effect enumerates that embodiment of the method is similar to the application object, therefore the object is enumerated the implementation of equipment 30 and be may refer to such as figure
Object shown in 2 enumerates the implementation of method, and overlaps will not be repeated.
Fig. 4 is referred to, Fig. 4 is the block schematic illustration that another object provided by the embodiments of the present application enumerates equipment 40.Such as
Shown in Fig. 4, it may include: bus 41, processor 42, memory 43 and input/output interface 44 which, which enumerates equipment 40,.Its
In, the processor 42, memory 43 and input/output interface 44 for being connected with each other and making above-mentioned member by the bus 41
Part can be in communication with each other.For the memory 43 for storing one or more computer programs, which includes computer
Instruction.The input/output interface 44 is for controlling the communication connection that the object is enumerated between equipment 40 and other equipment.
Specifically, which is configured for calling the computer instruction to realize that object as shown in Figure 2 is enumerated
Method.
Wherein, which can be CPU (Central Processing Unit, central processing unit).The memory
43 can be any type of memory, such as can be ROM (Read-Only Memory, read-only memory), RAM
(Random Access Memory, random access memory memory body) and nonvolatile RAM etc..
Based on the same inventive concept, the object provided in the embodiment of the present application enumerate the principle that equipment 40 solves the problems, such as and
Beneficial effect enumerates that embodiment of the method is similar to the application object, therefore the object is enumerated the implementation of equipment 40 and be may refer to such as figure
Object shown in 2 enumerates the implementation of method, and overlaps will not be repeated.
Those of ordinary skill in the art will appreciate that realizing all or part of the process in above-described embodiment method, being can be with
Relevant hardware is instructed to complete by computer program, above-mentioned program can be stored in a computer-readable storage medium
In, the program is when being executed, it may include such as the process of the embodiment of above-mentioned each method.Wherein, above-mentioned storage medium can be magnetic
Dish, CD, ROM or RAM.
Claims (10)
1. a kind of object enumerates method characterized by comprising
The key of the object in bucket is sampled to obtain interval mark set at a predetermined interval, wherein the interval mark
Set includes multiple interval marks, and the multiple interval mark is the sampling being sampled to the key of the object in the bucket
As a result；
It synchronizes and enumerates in the interval mark set any adjacent corresponding object of interval mark two-by-two, to obtain multiple enumerate
Object result；
Merge and the multiple enumerates object result.
2. the method according to claim 1, wherein described synchronize is enumerated any two in the interval mark set
The two adjacent corresponding objects of interval mark, comprising:
The multiple interval mark is respectively allocated to multiple processing units, wherein the corresponding processing of an interval mark is single
Member, so that the multiple processing unit enters ginseng using interval mark corresponding with itself as enumerate object interface, from described
Object is enumerated in bucket until including section corresponding with itself in the interval mark set in the object result enumerated
The object number for including in the object for marking different interval mark to be identified, or the object result until enumerating is less than
The maximum return value for enumerating object interface.
3. method according to claim 1 or 2, which is characterized in that first section mark in the multiple interval mark
It is denoted as null value.
4. described in any item methods according to claim 1~3, which is characterized in that the merging is the multiple to enumerate object knot
After fruit, the method also includes:
Duplicate removal is carried out to the multiple object result of enumerating after merging.
5. a kind of object enumerates equipment characterized by comprising
Sampling module is sampled to obtain interval mark set for the key at a predetermined interval to the object in bucket,
Described in interval mark set include multiple interval marks, the multiple interval mark is the key to the object in the bucket
The sampling result being sampled；
Module is enumerated, enumerates in the interval mark set any adjacent corresponding object of interval mark two-by-two for synchronizing,
Multiple object result is enumerated to obtain；
Merging module the multiple enumerates object result for merging.
6. equipment according to claim 5, which is characterized in that the module of enumerating is specifically used for:
The multiple interval mark is respectively allocated to multiple processing units, wherein the corresponding processing of an interval mark is single
Member, so that the multiple processing unit enters ginseng using interval mark corresponding with itself as enumerate object interface, from described
Object is enumerated in bucket until including section corresponding with itself in the interval mark set in the object result enumerated
The object number for including in the object for marking different interval mark to be identified, or the object result until enumerating is less than
The maximum return value for enumerating object interface.
7. equipment according to claim 5 or 6, which is characterized in that first section mark in the multiple interval mark
It is denoted as null value.
8. according to the described in any item equipment of claim 5~7, which is characterized in that the equipment further include:
Deduplication module, for carrying out duplicate removal to the multiple object result of enumerating after merging.
9. a kind of object enumerates equipment, which is characterized in that it includes processor and memory, the processing that the object, which enumerates equipment,
Device and memory are connected with each other, wherein the processor executes the computer instruction in the memory, realizes such as claim
1~4 described in any item methods.
10. a kind of computer storage medium, which is characterized in that the computer storage medium is stored with computer program, described
Computer program includes computer instruction, and the computer instruction makes the processor execute such as right when being executed by a processor
It is required that 1~4 described in any item methods.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811146403.1A CN109408571B (en) | 2018-09-28 | 2018-09-28 | Object enumeration method, device and computer storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811146403.1A CN109408571B (en) | 2018-09-28 | 2018-09-28 | Object enumeration method, device and computer storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109408571A true CN109408571A (en) | 2019-03-01 |
| CN109408571B CN109408571B (en) | 2022-03-29 |
Family
ID=65466568
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811146403.1A Active CN109408571B (en) | 2018-09-28 | 2018-09-28 | Object enumeration method, device and computer storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109408571B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116150093A (en) * | 2023-03-04 | 2023-05-23 | 北京大道云行科技有限公司 | Method for realizing object storage enumeration of objects and electronic equipment |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050138319A1 (en) * | 2003-12-23 | 2005-06-23 | International Business Machines Corporation | Parallel memory compaction |
| CN104462324A (en) * | 2014-12-03 | 2015-03-25 | 浪潮电子信息产业股份有限公司 | HDFS multithreaded parallel downloading method |
| CN104994177A (en) * | 2015-08-06 | 2015-10-21 | 上海爱数软件有限公司 | Synchronized method of network disk system, terminal equipment and network disk system |
-
2018
- 2018-09-28 CN CN201811146403.1A patent/CN109408571B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050138319A1 (en) * | 2003-12-23 | 2005-06-23 | International Business Machines Corporation | Parallel memory compaction |
| CN104462324A (en) * | 2014-12-03 | 2015-03-25 | 浪潮电子信息产业股份有限公司 | HDFS multithreaded parallel downloading method |
| CN104994177A (en) * | 2015-08-06 | 2015-10-21 | 上海爱数软件有限公司 | Synchronized method of network disk system, terminal equipment and network disk system |
Non-Patent Citations (1)
| Title |
|---|
| TOM: "《S3 HowTo Dealwith ListObjec》", 《HTTPS://WORK-JLSUN.GITHUB.IO/2016/11/19/S3-HOWTO-DEALWITH-LISTOBJECT.HTML》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116150093A (en) * | 2023-03-04 | 2023-05-23 | 北京大道云行科技有限公司 | Method for realizing object storage enumeration of objects and electronic equipment |
| CN116150093B (en) * | 2023-03-04 | 2023-11-03 | 北京大道云行科技有限公司 | Method for realizing object storage enumeration of objects and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109408571B (en) | 2022-03-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104423960B (en) | A kind of method and system of project continuous integrating | |
| CN107302450B (en) | Data acquisition task changing method and device | |
| CN104317843A (en) | Data synchronism ETL (Extract Transform Load) system | |
| CN107357883A (en) | Data migration method and device | |
| CN110413441A (en) | Active and standby storage volume synchrodata method of calibration, device, equipment and storage medium | |
| CN106940677A (en) | One kind application daily record data alarm method and device | |
| CN105677465B (en) | The data processing method and device of batch processing are run applied to bank | |
| US20140156598A1 (en) | Replication control using eventually consistent meta-data | |
| CN101442591B (en) | Method and system for self-checking and updating performance business model discrepancy | |
| CN105847378B (en) | A kind of method and system for realizing that big data is synchronous | |
| CN108123851A (en) | The lifetime detection method and device of main and subordinate node synchronization link in distributed system | |
| CN106503158A (en) | Method of data synchronization and device | |
| CN106874189A (en) | A kind of implementation method of the automatization test system of real-time data of power grid storehouse system | |
| CN107391634A (en) | Data migration method and device | |
| CN109783258A (en) | A kind of message treatment method, device and server | |
| CN109388677A (en) | Method of data synchronization, device, equipment and its storage medium between cluster | |
| CN110290166A (en) | Across company-data exchange method, system, device and readable storage medium storing program for executing | |
| CN104462342B (en) | database snapshot synchronization processing method and device | |
| CN112115200A (en) | Data synchronization method and device, electronic equipment and readable storage medium | |
| CN109408571A (en) | Object enumerates method, equipment and computer storage medium | |
| CN108520043A (en) | Data object acquisition method, apparatus and system, computer readable storage medium | |
| CN106407233A (en) | A data processing method and apparatus | |
| CN106874047A (en) | Multiprocessor scheduling method, device and distributed system | |
| CN114401207A (en) | Positioning method and device for communication abnormal terminal equipment and electronic equipment | |
| CN112035444B (en) | Method and device for transferring image data among heterogeneous systems without stopping |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220217 Address after: 550025 Huawei cloud data center, jiaoxinggong Road, Qianzhong Avenue, Gui'an New District, Guiyang City, Guizhou Province Applicant after: Huawei Cloud Computing Technology Co.,Ltd. Address before: 518129 Bantian HUAWEI headquarters office building, Longgang District, Guangdong, Shenzhen Applicant before: HUAWEI TECHNOLOGIES Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |