CN110413588A - Distributed objects storage method, device, computer equipment and storage medium - Google Patents
Distributed objects storage method, device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN110413588A CN110413588A CN201910693969.4A CN201910693969A CN110413588A CN 110413588 A CN110413588 A CN 110413588A CN 201910693969 A CN201910693969 A CN 201910693969A CN 110413588 A CN110413588 A CN 110413588A
- Authority
- CN
- China
- Prior art keywords
- file
- type
- type file
- uploaded
- distributed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/178—Techniques for file synchronisation in file systems
- G06F16/1794—Details of file format conversion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
This application provides a kind of distributed objects storage method, device, computer equipment and storage mediums, wherein this method comprises: receiving data upload requests, carries first kind file in data upload requests；In response to data upload requests, first kind file is converted into Second Type file, wherein the difference between the byte number of Second Type file and the byte number of first kind file is greater than preset threshold；Second Type file is uploaded to distributed objects storage system and carries out object storage.Above-mentioned distributed objects storage method effectively improves the flexibility and scalability of distributed objects storage system so that distributed objects storage system is suitable for storing the file of arbitrary size.
Description
Technical field
This application involves field of computer technology, in particular to a kind of distributed objects storage method, device, computer are set
Standby and storage medium.
Background technique
Object storage is the technology being commonly used in internet, and object storage is different from file storage, and object is not with catalogue
The structure of layering carries out tissue.Each object is all located at the same level in the referred to as flat space of storage pool, every level-one
Each element has unique mark within the storage system, and user accesses container or object by this mark.In object
In storage, flat data organizational structure is usually used and has abandoned nested file, it is huge so as to avoid safeguarding
Directory tree.
Currently, object storage generallys use the mode of distributed storage.However, distributed objects storage there are it is some not
Foot is unsuitable for storing the file of arbitrary size, and flexibility and scalability is poor.
In view of the above-mentioned problems, currently no effective solution has been proposed.
Summary of the invention
The embodiment of the present application provides a kind of distributed objects storage method, device, computer equipment and storage medium, with
Solve the problems, such as that distributed objects storage in the prior art is unsuitable for storing arbitrary size file.
The embodiment of the present application provides a kind of distributed objects storage method, comprising: data upload requests is received, in data
It passes in request and carries first kind file；In response to data upload requests, first kind file is converted into Second Type text
Part, wherein the difference between the byte number of Second Type file and the byte number of first kind file is greater than preset threshold；By
Two type files are uploaded to distributed objects storage system and carry out object storage.
In one embodiment, multiple first kind files and multiple first kind files are carried in data upload requests
In each first kind file metadata, the byte number of first kind file is less than the first predetermined word joint number；Correspondingly, by first
Type file is converted into Second Type file, comprising: obtains the metadata of each first kind file in multiple first kind files；
By multiple first kind file mergencess at Second Type file, and records each first kind file in multiple first kind files and exist
Location information in Second Type file；Index file is generated according to the metadata of each first kind file and location information.
In one embodiment, Second Type file is uploaded to distributed objects storage system and carries out object storage, packet
It includes: Second Type file is uploaded to progress object storage in the distributed file system of distributed objects storage system；By rope
Quotation part is uploaded in the distributed data base of distributed objects storage system and is stored.
In one embodiment, the member of third predetermined word joint number and first kind file is also carried in data upload requests
The byte number of data, first kind file is greater than the second predetermined word joint number；Correspondingly, first kind file is converted into the second class
Type file, comprising: according to third predetermined word joint number by first kind file division at multiple Second Type files, and record multiple
The offset of each Second Type file in Second Type file, wherein the byte number of Second Type file is third preset byte
Number；The index text of each Second Type file is generated according to the metadata of the offset of each Second Type file and first kind file
Part.
In one embodiment, Second Type file is uploaded to distributed objects storage system and carries out object storage, packet
It includes: multiple Second Type files is uploaded to progress object storage in the distributed file system of distributed objects storage system,
The data service metadata generated in record upload procedure；The index file of each Second Type file is uploaded to distributed objects storage
It is stored in the distributed data base of system；According to the metadata of first kind file, data service metadata, each Second Type text
The attribute information of part generates control object, wherein the attribute information of Second Type file includes of multiple Second Type files
Several and third predetermined word joint number；The distributed file system that control object is uploaded to distributed objects storage system is subjected to object
Storage.
In one embodiment, multiple Second Type files are uploaded to the distributed document of distributed objects storage system
Object storage is carried out in system, comprising: generate Universally Unique Identifier at random；According to Universally Unique Identifier and each Second Type
The offset of file generates the key assignments of each Second Type file；By Second Type file each in multiple Second Type files, each
The key assignments of two type files and the metadata of first kind file are uploaded to distributed objects storage system and carry out object storage.
In one embodiment, according to the metadata of first kind file, data service metadata, each Second Type file category
Property information generate control object, comprising: target metadata is generated according to the metadata of first kind file and data service metadata；Root
Control object is generated according to the attribute information of target metadata, general exclusive identification code and Second Type file.
In one embodiment, multiple Second Type files are uploaded to the distributed document of distributed objects storage system
Object storage is carried out in system, further includes: when uploading error, generate error message, wherein carry in error message general
Unique identifier；The Second Type file uploaded is verified according to Universally Unique Identifier, uploads failure to determine
Second Type file；The Second Type file for uploading failure is uploaded to progress object storage in distributed file system again.
The embodiment of the present application also provides a kind of distributed objects storage devices, comprising: receiving module, for receiving data
Upload request carries first kind file in data upload requests；Conversion module is used in response to data upload requests, will
First kind file is converted into Second Type file, wherein the byte number of Second Type file and the byte of first kind file
Difference between number is greater than preset threshold；Uploading module, for Second Type file to be uploaded to distributed objects storage system
Carry out object storage.
The embodiment of the present application also provides a kind of computer equipment, including processor and refers to for storage processor to be executable
The memory of order, the processor realize distributed objects storage side described in above-mentioned any embodiment when executing described instruction
The step of method.
The embodiment of the present application also provides a kind of computer readable storage medium, is stored thereon with computer instruction, the finger
Order is performed the step of realizing distributed objects storage method described in above-mentioned any embodiment.
In the embodiment of the present application, a kind of distributed objects storage method is provided, data upload requests are received, by data
First kind file is converted into Second Type file in upload request, and the Second Type file of generation is uploaded to it is distributed right
As storage system carries out object storage.In aforesaid way, by will be not suitable for directly being uploaded to distributed objects storage system into
The first kind file of row object storage is converted into the Second Type file for being suitble to directly be uploaded to distributed objects storage system,
Then Second Type file is uploaded to distributed objects storage system and carries out object storage, so that distributed objects storage system
The file that arbitrary size can be stored effectively improves the flexibility and scalability of distributed objects storage system.By above-mentioned
Scheme solves existing distributed objects storage system and is unsuitable for the technical issues of storing arbitrary size file, has reached realization
Store arbitrary size file, and the technical effect of effectively lifting system flexibility and scalability.
Detailed description of the invention
The drawings described herein are used to provide a further understanding of the present application, constitutes part of this application, not
Constitute the restriction to the application.In the accompanying drawings:
Fig. 1 shows a kind of schematic diagram of application scenarios of the distributed objects storage method in one embodiment of the application；
Fig. 2 shows the flow charts of the distributed objects storage method in one embodiment of the application；
Fig. 3 shows control object and Second Type text in the distributed objects storage method in one embodiment of the application
The structural schematic diagram of part；
Fig. 4 shows the schematic diagram of the distributed objects storage device in one embodiment of the application；
Fig. 5 shows the schematic diagram of the computer equipment in one embodiment of the application.
Specific embodiment
The principle and spirit of the application are described below with reference to several illustrative embodiments.It should be appreciated that providing this
A little embodiments are used for the purpose of making those skilled in the art can better understand that realizing the application in turn, and be not with any
Mode limits scope of the present application.On the contrary, it is more thorough and complete to these embodiments are provided so that the application discloses, and
And the scope of the present disclosure can be completely communicated to those skilled in the art.
It will be apparent to one skilled in the art that presently filed embodiment can be implemented as a kind of system, appliance arrangement, method
Or computer program product.Therefore, disclose can be with specific implementation is as follows by the application, it may be assumed that complete hardware, complete soft
The form that part (including firmware, resident software, microcode etc.) or hardware and software combine.
In view of existing distributed objects storage is unsuitable for storing arbitrary size file, inventors discovered through research that,
It can be by the way that the first kind file for being not suitable for directly being uploaded to the progress object storage of distributed objects storage system be converted into
It is suitble to directly be uploaded to the Second Type file of distributed objects storage system, Second Type file is then uploaded to distribution
Object storage system carries out object storage, and distributed objects storage system is enabled to store the file of arbitrary size.
Based on this, the embodiment of the present application provides a kind of distributed objects storage method, and Fig. 1 shows the implementation of the application one
A kind of schematic diagram of application scenarios of the distributed objects storage method provided in example.As shown in Figure 1, user end to server is sent out
Send data upload requests, wherein the first kind file to be uploaded is carried in data upload requests.Server receives data
After upload request, the first kind file in data upload requests is converted into Second Type file, and by the second of generation
Type file is uploaded to distributed objects storage system and carries out object storage.
Wherein, above-mentioned client can be desktop computer, notebook, mobile phone terminal, PDA etc., as long as can be to user
Or business personnel carries out content displaying and is ok to the equipment that server sends data upload requests, what it is for client is in
It is now formed, the application is not construed as limiting.Above-mentioned server can be single server, be also possible to server cluster, either
Cloud Server etc. is ok, and specific group is shaped to the application and is also not construed as limiting.Above-mentioned distributed objects storage system can be
Hadoop object storage system or other distributed objects storage systems.
Wherein, the difference between the byte number of first kind file and the byte number of Second Type file is greater than default threshold
Value, that is, the size of first kind file differs larger with the size of Second Type file.Second Type file, which can be, to be suitble to directly
Connect the file for reaching distributed objects storage system.First kind file can be small documents, if small documents individually deposited
Storage will greatly waste hardware resource in distributed file system HDFS, therefore, can be by multiple first kind file mergencess
At Second Type file.First kind file is also possible to super large file, and distributed objects storage system can not store super large
File.Therefore, first kind file can be converted into multiple Second Type files, then upload multiple Second Type files
Object storage is carried out to distributed objects storage system.
Fig. 2 shows the flow charts of distributed objects storage method in one embodiment of the application.Although this application provides
As the following examples or method operating procedure shown in the drawings or apparatus structure, but based on conventional or be not necessarily to creative labor
It may include more or less operating procedure or modular unit in the method or device.There is no must in logicality
In the step of wanting causality or structure, the execution sequence of these steps or the modular structure of device are not limited to the embodiment of the present application
Description and execution shown in the drawings sequence or modular structure.The device in practice or terminal of the method or modular structure
When products application, carry out sequence execution or parallel can be connected according to embodiment or method shown in the drawings or modular structure
It executes (such as environment or even distributed processing environment of parallel processor or multiple threads).
Specifically, as shown in Fig. 2, the distributed objects storage method that a kind of embodiment of the application provides may include following
Step:
Step S201 receives data upload requests, carries first kind file in data upload requests.
Wherein, server can receive the data upload requests of client transmission.Wherein, it is carried in data upload requests
First kind file.Wherein, first kind file can be the file for being not suitable for directly uploading distributed objects storage.
First kind file is converted into Second Type file in response to data upload requests by step S202, wherein the
Difference between the byte number of two type files and the byte number of first kind file is greater than preset threshold.
After receiving data upload requests, server can be in response to data upload requests, by first kind file
It is converted into the Second Type file for being suitble to directly be uploaded to distributed objects storage system.Wherein, the byte of first kind file
Several differences with the byte number of Second Type file are greater than preset threshold, that is, size and the Second Type text of first kind file
The size difference of part is larger.
Second Type file is uploaded to distributed objects storage system and carries out object storage by step S203.
After first kind file is converted into Second Type file, Second Type file can be uploaded to distribution
Object storage system carries out object storage.
In aforesaid way, the first of the progress object storage of distributed objects storage system is directly uploaded to by that will be not suitable for
Type file is converted into the Second Type file for being suitble to directly be uploaded to distributed objects storage system, then by Second Type text
Part is uploaded to distributed objects storage system and carries out object storage, and distributed objects storage system is enabled to store arbitrary size
File, effectively improve the flexibility and scalability of distributed objects storage system.It solves through the above scheme existing
Distributed objects storage system is unsuitable for the technical issues of storing arbitrary size file, has reached realization storage arbitrary size text
Part, and the technical effect of effectively lifting system flexibility and scalability.
Multiple first kind files and multiple can be carried in some embodiments of the application, in data upload requests
The metadata of each first kind file in one type file, the byte number of first kind file is less than the first predetermined word joint number.Phase
First kind file is converted into Second Type file by Ying Di, may include: to obtain each first kind in multiple first kind files
The metadata of type file；By multiple first kind file mergencess at Second Type file, and record in multiple first kind files
Location information of each first kind file in Second Type file；According to the metadata and location information of each first kind file
Generate index file.
Wherein, the first predetermined word joint number can be determined according to system parameter and actual demand.In first kind file
In the case that byte number is less than the first predetermined word joint number, it can determine that first kind file is small documents.In first kind file
It, can be by multiple first kind file mergencess at Second Type file, later by Second Type file in the case where for small documents
It is uploaded to distributed objects storage system and carries out object storage.Wherein, multiple first kind texts are carried in data upload requests
The metadata of each first kind file in part and multiple first kind files.Wherein, metadata is some parameters about file
Information, for example, metadata may include the information such as the creation time of file, file size.By multiple first kind file mergencess
At Second Type file, and record positioning letter of each first kind file in Second Type file in multiple first kind files
Breath.Wherein, location information can be the information such as offset of the first kind file in Second Type file.According to the first kind
The metadata and location information of file generate index file.It is suitable on directly by the above-mentioned means, small documents can be merged into
Object storage is carried out after reaching the Second Type file of distributed objects storage system, hardware resource can be saved, improves resource
Utilization rate.
Further, in some embodiments of the application, Second Type file is uploaded to distributed objects storage system
Object storage is carried out, may include: the distributed file system that Second Type file is uploaded to distributed objects storage system
Middle progress object storage；Index file is uploaded in the distributed data base of distributed objects storage system and is stored.
Specifically, distributed objects storage system may include distributed file system and distributed data base.Wherein, divide
Cloth file system may include HDFS (Hadoop Distributed File System, Hadoop distributed field system
System).HDFS is the distributed file system of an Error Tolerance, is suitble to be deployed on cheap machine, can provide height and handle up
The data access of amount, the file storage being very suitable on large-scale dataset.Distributed data base may include Hbase
(Hadoop Database, Hadoop database).Hbase is a high reliability, high-performance, towards column, telescopic distribution
Formula storage system can support ultra-large random access in real time.By multiple small first kind file mergencess at the second class
Type file is uploaded to progress object storage in distributed file system.By the metadata and location information of multiple first kind files
The index file of formation is stored in distributed data base.User, can be first from HBase when reading first kind file
Information in middle reading index file reads first kind text from the corresponding position in HDFS further according to the information in index file
Part simultaneously returns to user.By the above-mentioned means, can not only save memory space, but also reading time-consuming can be reduced.
In some embodiments of the application, third predetermined word joint number and the first kind can also be carried in data upload requests
The byte number of the metadata of type file, first kind file is greater than the second predetermined word joint number.Correspondingly, first kind file is turned
It changes Second Type file into, may include: by first kind file division according to third predetermined word joint number into multiple Second Types
File, and record the offset of each Second Type file in multiple Second Type files, wherein the byte number of Second Type file
For third predetermined word joint number；Each second class is generated according to the offset of each Second Type file and the metadata of first kind file
The index file of type file.
Wherein, the second predetermined word joint number can be determined according to system parameter and actual demand.In first kind file
In the case that byte number is greater than the second predetermined word joint number, first kind file can be determined as to super large file.In the first kind
It further include the first kind in data upload requests other than including first kind file in the case that file is super large file
The metadata and third predetermined word joint number of file.Wherein, metadata is some parameter informations about file, for example, metadata
It may include the information such as creation time, the file size of file.Wherein, third predetermined word joint number is less than the second predetermined word joint number,
It is adapted for directly uploading the file size that distributed memory system carries out object storage.It can be according to third predetermined word joint number by
One type file division records the offset of each Second Type file of multiple Second Type files at multiple Second Type files
Amount.The byte number of Second Type file, that is, third predetermined word joint number.It can be according to the offset and first of each Second Type file
The metadata of type file generates the index file of each Second Type file.By the above-mentioned means, super large file can be cut
At multiple Second Type files for being suitble to direct upload distributed objects storage system to be stored and generate each Second Type text
The index file of part, so that distributed objects storage system can store super large file.
Further, in some embodiments of the application, Second Type file is uploaded to distributed objects storage system
Object storage is carried out, may include: the distributed document that multiple Second Type files are uploaded to distributed objects storage system
Object storage is carried out in system, records the data service metadata generated in upload procedure；By the index file of each Second Type file
It is uploaded in the distributed data base of distributed objects storage system and is stored；According to the metadata of first kind file, industry
Be engaged in metadata, each Second Type file attribute information generate control object, wherein the attribute information of Second Type file includes
The number and third predetermined word joint number of multiple Second Type files；Control object is uploaded to point of distributed objects storage system
Cloth file system carries out object storage.
Specifically, distributed objects storage system may include distributed file system and distributed data base.Wherein, divide
Cloth file system may include HDFS.HDFS is the distributed file system of an Error Tolerance, is suitble to be deployed in cheap
Machine on, the data access of high-throughput can be provided, be very suitable on large-scale dataset file storage.Distributed data
Library may include Hbase.Hbase is a high reliability, high-performance, towards column, telescopic distributed data base, Ke Yizhi
Hold ultra-large random access in real time.The first kind file division of super large is uploaded to distribution at multiple Second Type files
Object storage is carried out in formula file system.The index file storage that the metadata of multiple Second Type files and offset are formed
In distributed data base.During multiple Second Type files are uploaded to distributed objects storage system, record life
At data service metadata.Wherein, data service metadata may include some information subsidiary when uploading, for example, uploading author and list
The information such as position.It can be generated according to the attribute information of the metadata of first kind file, data service metadata, each Second Type file
Control object.Wherein, the attribute information of Second Type file includes the total number and third predetermined word of multiple Second Type files
Joint number (that is, size of Second Type file).Control object is uploaded to the distributed field system of distributed objects storage system
System carries out object storage.User can first read control when reading first kind file from distributed file system
Then object lists the information of all Second Type files according to control object.According to the start-stop position in downloading request, determine
Which Second Type file is the content to be downloaded be included in, these Second Types text is obtained from distributed file system
Part, and these Second Type files are assembled into inlet flow.Inlet flow is back to client, client can be from inlet flow
It reads data and is stored in local.By the above-mentioned means, the upload and downloading of super large file may be implemented, distribution is effectively improved
The flexibility and scalability of storage system reduce and read time-consuming.
Further, in some embodiments of the application, multiple Second Type files are uploaded to distributed objects storage
Object storage is carried out in the distributed file system of system, may include: random generation Universally Unique Identifier；According to it is general only
One identification code and the offset of each Second Type file generate the key assignments of each Second Type file；It will be in multiple Second Type files
The metadata of each Second Type file, the key assignments of each Second Type file and first kind file is uploaded to distributed objects storage
System carries out object storage.
When multiple Second Type files being uploaded to distributed objects storage system carrying out object storage, need each the
Key assignments, metadata and Second Type file of two type files itself carry out object storage.Wherein, the member of Second Type file
Data can be the metadata of first kind file.The key assignments of each Second Type file can be according to the inclined of each Second Type file
Shifting amount determines.Illustratively, it is first random generate Universally Unique Identifier (Universally Unique Identifier,
UUID), the key assignments of each Second Type file is then generated according to the offset of Universally Unique Identifier and each Second Type file.
By the above-mentioned means, can be convenient the key assignments for quickly generating each Second Type file, and each Second Type file is carried out pair
As storage.
Further, in some embodiments of the application, according to the metadata of first kind file, data service metadata, each
The attribute information of Second Type file generates control object, may include: metadata and the business member according to first kind file
Data generate target metadata；It is generated according to the attribute information of target metadata, general exclusive identification code and Second Type file
Control object.
Specifically, mesh can be generated according to the data service metadata generated in the metadata and upload procedure of first kind file
Metadata is marked, control pair is then generated according to the attribute information of target metadata, general exclusive identification code and Second Type file
As.After generating control object, control object is uploaded to distributed file system and carries out object storage.Pass through above-mentioned side
The control object for being stored with the relevant information of each Second Type file can be generated in formula, is convenient for subsequent reading.
As shown in figure 3, the key assignments key of control object is equal to the key assignments KEY of first kind file.It is stored in control object
The attribute information of metadata, UUID and each Second Type file.Wherein, each Second Type file in multiple Second Type files
Key assignments key be UUID and each Second Type file offset scope_i.
In the case where first kind file is super large file, first kind file is divided into many Second Type texts
Part may malfunction in multiple Second Type file upload procedures.In the case where uploading error, server be can be generated out
Wrong information, wherein UUID is carried in error message, and the Second Type file uploaded is verified according to UUID, it can be with
The Second Type file for uploading failure is found, and is uploaded again.Therefore, in some embodiments of the application, by multiple
Two type files are uploaded to progress object storage in the distributed file system of distributed objects storage system, can also include:
When uploading error, error message is generated, wherein Universally Unique Identifier is carried in error message；According to general unique knowledge
Other code verifies the Second Type file uploaded, to determine the Second Type file for uploading failure；Failure will be uploaded
Second Type file is uploaded to progress object storage in distributed file system again.
When malfunctioning during multiple Second Type files upload, error message is generated.Wherein, it is carried in error message
There is UUID.According to UUID, the Second Type file uploaded is verified, finds the Second Type file for uploading failure.It will
The Second Type file for uploading failure is uploaded in distributed file system progress object storage again, after verify all the
Two type files are sent completely order, and more new metadata after confirmation is errorless, update control object.It illustratively, can be with
Do not allow commonly to upload interface with breakpoint transmission interface substitution, i.e., do not occur to upload abnormal but directly uses breakpoint transmission interface, clothes
Business device can control this.But when breakpoint transmission interface scans find that all Second Type files are in fact uploaded into
Function is also not considered as that this is mistake, is only to skip Second Type file and uploads process, and embodies this feelings in Boolean return
Condition.By the above-mentioned means, breakpoint transmission can be supported, the accuracy and efficiency of big file uploading is improved.
The above method is illustrated below with reference to two specific embodiments, it should be noted, however, that the specific implementation
Example does not constitute an undue limitation on the present application merely to the application is better described.
In a specific embodiment, distributed objects storage method the following steps are included:
Step 1, user end to server sends data upload requests, wherein multiple the are carried in data upload requests
The metadata of one type file and each first kind file, wherein the byte number of first kind file is less than the first preset byte
Number is small documents；
Step 2, server obtains the metadata of each first kind file in response to data upload requests, by multiple first
Type file is merged into Second Type file, and it is literary in Second Type to record each first kind file in multiple first kind files
Location information in part generates index file according to the metadata of each first kind file and location information；
Step 3, Second Type file is uploaded in the distributed file system of distributed objects storage system by server
Object storage is carried out, and index file is uploaded in the distributed data base of distributed objects storage system and is stored.
In another specific embodiment, distributed objects storage method the following steps are included:
Step 1, user end to server sends data upload requests, wherein the first kind is carried in data upload requests
Type file, the metadata of first kind file, third predetermined word joint number, wherein the byte number of first kind file is greater than second
Predetermined word joint number is super large file；
Step 2, server is in response to the request that receives, according to third predetermined word joint number by first kind file division at
Multiple Second Type files, and the offset of each Second Type file in multiple Second Type files is recorded, according to each second class
The offset of type file and the metadata of first kind file generate the index file of each Second Type file；
Step 3, multiple Second Type files are uploaded to the distributed field system of distributed objects storage system by server
Object storage is carried out in system, is recorded the data service metadata generated in upload procedure and is specifically generated general unique identification at random
Code generates the key assignments of each Second Type file according to the offset of Universally Unique Identifier and each Second Type file, will be multiple
The metadata of each Second Type file, the key assignments of each Second Type file and first kind file is uploaded in Second Type file
Distributed objects storage system carries out object storage；
Step 4, the index file of each Second Type file is uploaded to the distribution of distributed objects storage system by server
It is stored in formula database；
Step 5, server generates target metadata according to the metadata and data service metadata of first kind file, according to mesh
The attribute information for marking metadata, general exclusive identification code and Second Type file generates control object, wherein each Second Type text
The attribute information of part includes the size of each Second Type file and the total number of Second Type file；
Step 6, server carries out the distributed file system that control object is uploaded to distributed objects storage system pair
As storage.
Distributed objects storage method in above-mentioned two embodiment would prove unsuitable for directly uploading distributed objects respectively and deposit
The super large file of object storage is carried out in storage system and small documents are converted into being suitable for directly uploading in distributed objects storage system
The Second Type file of object storage is carried out, then Second Type file is uploaded and is stored, so that distributed objects storage system
The file that arbitrary size can be stored effectively improves the flexibility and scalability of distributed objects storage system.By above-mentioned
Scheme solves existing distributed objects storage system and is unsuitable for the technical issues of storing arbitrary size file, has reached realization
Store arbitrary size file, and the technical effect of effectively lifting system flexibility and scalability.
Based on the same inventive concept, a kind of distributed objects storage device is additionally provided in the embodiment of the present application, it is such as following
Embodiment described in.Since the principle that distributed objects storage device solves the problems, such as is similar to distributed objects storage method, because
The implementation of this distributed objects storage device may refer to the implementation of distributed objects storage method, and overlaps will not be repeated.
Used below, the combination of the software and/or hardware of predetermined function may be implemented in term " unit " or " module ".Although with
Device described in lower embodiment is preferably realized with software, but the combined realization of hardware or software and hardware
It may and be contemplated.Fig. 4 is a kind of structural block diagram of the distributed objects storage device of the embodiment of the present application, such as Fig. 4 institute
Show, comprising: receiving module 401, conversion module 402 and uploading module 403 are below illustrated the structure.
The upload request for receiving data of receiving module 401 carries first kind file in data upload requests.
Conversion module 402 is used to that first kind file to be converted into Second Type file in response to data upload requests,
In, the difference between the byte number of Second Type file and the byte number of first kind file is greater than preset threshold.
Uploading module 403, which is used to for Second Type file to be uploaded to distributed objects storage system, carries out object storage.
Multiple first kind files and multiple can be carried in some embodiments of the application, in data upload requests
The metadata of each first kind file in one type file, the byte number of first kind file is less than the first predetermined word joint number；Phase
Ying Di, conversion module can be specifically used for: obtain the metadata of each first kind file in multiple first kind files；It will be multiple
First kind file mergences records in multiple first kind files each first kind file in the second class at Second Type file
Location information in type file；Index file is generated according to the metadata of each first kind file and location information.
In some embodiments of the application, uploading module can be specifically used for: Second Type file is uploaded to distribution
Object storage is carried out in the distributed file system of object storage system；Index file is uploaded to distributed objects storage system
Distributed data base in stored.
In some embodiments of the application, third predetermined word joint number and the first kind can also be carried in data upload requests
The byte number of the metadata of type file, first kind file is greater than the second predetermined word joint number；Correspondingly, conversion module can be specific
For: according to third predetermined word joint number by first kind file division at multiple Second Type files, and record multiple second classes
The offset of each Second Type file in type file, wherein the byte number of Second Type file is third predetermined word joint number；According to
The offset of each Second Type file and the metadata of first kind file generate the index file of each Second Type file.
In some embodiments of the application, uploading module can be specifically used for: multiple Second Type files are uploaded to point
Object storage is carried out in the distributed file system of cloth object storage system, records the business member number generated in upload procedure
According to；The index file of each Second Type file is uploaded in the distributed data base of distributed objects storage system and is deposited
Storage；Control object is generated according to the attribute information of the metadata of first kind file, data service metadata, each Second Type file,
Wherein, the attribute information of Second Type file includes the number and third predetermined word joint number of multiple Second Type files；It will control
The distributed file system that object is uploaded to distributed objects storage system carries out object storage.
In some embodiments of the application, multiple Second Type files are uploaded to the distribution of distributed objects storage system
Object storage is carried out in formula file system, may include: random generation Universally Unique Identifier；According to Universally Unique Identifier with
The offset of each Second Type file generates the key assignments of each Second Type file；By each Second Type in multiple Second Type files
The metadata of file, the key assignments of each Second Type file and first kind file is uploaded to distributed objects storage system and carries out pair
As storage.
In some embodiments of the application, according to the metadata of first kind file, data service metadata, each Second Type text
The attribute information of part generates control object, may include: to generate mesh according to the metadata and data service metadata of first kind file
Mark metadata；Control object is generated according to the attribute information of target metadata, general exclusive identification code and Second Type file.
In some embodiments of the application, multiple Second Type files are uploaded to the distribution of distributed objects storage system
Object storage is carried out in formula file system, can also include: to generate error message, wherein in error message when uploading error
Carry Universally Unique Identifier；The Second Type file uploaded is verified according to Universally Unique Identifier, with determination
Upload the Second Type file of failure；The Second Type file for uploading failure is uploaded to again in distributed file system and is carried out
Object storage.
It can be seen from the above description that the embodiment of the present application realizes following technical effect: straight by that will be not suitable for
Connect reach distributed objects storage system carry out object storage first kind file be converted into be suitble to directly be uploaded to distribution
Then Second Type file is uploaded to distributed objects storage system and carried out pair by the Second Type file of formula object storage system
As storage, distributed objects storage system is enabled to store the file of arbitrary size, effectively improves distributed objects storage system
The flexibility and scalability of system.It solves existing distributed objects storage system through the above scheme and is unsuitable for storage arbitrarily
The technical issues of big small documents, has reached and has realized storage arbitrary size file, and effective lifting system flexibility and scalability
Technical effect.
Embodiment further provides a kind of computer equipments by the application, specifically can be based on the application refering to shown in fig. 5
The computer equipment composed structure schematic diagram for the distributed objects storage method that embodiment provides, the computer equipment specifically may be used
To include input equipment 51, processor 52, memory 53.Wherein, the memory 53 is used for storage processor executable instruction.
The step of processor 52 realizes distributed objects storage method described in above-mentioned any embodiment when executing described instruction.
In the present embodiment, the input equipment, which specifically can be, carries out information exchange between user and computer system
One of main device.The input equipment may include keyboard, mouse, camera, scanner, light pen, writing input board, language
Sound input unit etc.；Input equipment is used to initial data be input in computer with the programs for handling these numbers.The input
Equipment, which can also obtain, receives the data that other modules, unit, equipment transmit.The processor can be by any appropriate
Mode is realized.For example, processor can take such as microprocessor or processor and storage that can be executed by (micro-) processor
Computer readable program code (such as software or firmware) computer-readable medium, logic gate, switch, specific integrated circuit
(Application Specific Integrated Circuit, ASIC), programmable logic controller (PLC) and insertion microcontroller
Form etc..The storage implement body can be in modern information technologies for protecting stored memory device.The storage
Device may include many levels, in digital display circuit, as long as can save binary data can be memory；In integrated electricity
The circuit with store function of Lu Zhong, a not no physical form are also memory, such as RAM, FIFO；In systems, have
There is the storage equipment of physical form to be also memory, such as memory bar, TF card.
In the present embodiment, the function and effect of computer equipment specific implementation, can be with other embodiment pair
According to explanation, details are not described herein.
A kind of computer storage medium based on distributed objects storage method, institute are additionally provided in the application embodiment
It states computer storage medium and is stored with computer program instructions, it is above-mentioned any to be performed realization in the computer program instructions
The step of distributed objects storage method described in embodiment.
In the present embodiment, above-mentioned storage medium includes but is not limited to random access memory (Random Access
Memory, RAM), read-only memory (Read-Only Memory, ROM), caching (Cache), hard disk (Hard Disk
Drive, HDD) or storage card (Memory Card).The memory can be used for storing computer program instructions.Network is logical
Letter unit can be according to standard setting as defined in communication protocol, for carrying out the interface of network connection communication.
In the present embodiment, the function and effect of the program instruction specific implementation of computer storage medium storage, can
To compare explanation with other embodiment, details are not described herein.
Obviously, those skilled in the art should be understood that each module of above-mentioned the embodiment of the present application or each step can be with
It is realized with general computing device, they can be concentrated on a single computing device, or be distributed in multiple computing devices
On composed network, optionally, they can be realized with the program code that computing device can perform, it is thus possible to by it
Store and be performed by computing device in the storage device, and in some cases, can be held with the sequence for being different from herein
The shown or described step of row, perhaps they are fabricated to each integrated circuit modules or will be multiple in them
Module or step are fabricated to single integrated circuit module to realize.In this way, the embodiment of the present application be not limited to it is any specific hard
Part and software combine.
It should be understood that above description is to illustrate rather than to be limited.By reading above-mentioned retouch
It states, many embodiments and many applications except provided example all will be aobvious and easy for a person skilled in the art
See.Therefore, scope of the present application should not be determined referring to foregoing description, but should referring to preceding claims and this
The full scope of the equivalent that a little claims are possessed determines.
The foregoing is merely preferred embodiment of the present application, are not intended to limit this application, for the skill of this field
For art personnel, the embodiment of the present application can have various modifications and variations.Within the spirit and principles of this application, made
Any modification, equivalent substitution, improvement and etc. should be included within the scope of protection of this application.
Claims (11)
1. a kind of distributed objects storage method characterized by comprising
Receive data upload requests, wherein first kind file is carried in the data upload requests；
In response to the data upload requests, the first kind file is converted into Second Type file, wherein described second
Difference between the byte number of type file and the byte number of the first kind file is greater than preset threshold；
The Second Type file is uploaded to distributed objects storage system and carries out object storage.
2. the method according to claim 1, wherein carrying multiple first kind in the data upload requests
The metadata of each first kind file, the byte number of the first kind file are small in file and the multiple first kind file
In the first predetermined word joint number；
Correspondingly, the first kind file is converted into Second Type file, comprising:
Obtain the metadata of each first kind file in the multiple first kind file；
By the multiple first kind file mergences at Second Type file, and record each in the multiple first kind file
Location information of one type file in the Second Type file；
Index file is generated according to the metadata of each first kind file and location information.
3. according to the method described in claim 2, being deposited it is characterized in that, the Second Type file is uploaded to distributed objects
Storage system carries out object storage, comprising:
The Second Type file is uploaded in the distributed file system of the distributed objects storage system and carries out object
Storage；
The index file is uploaded in the distributed data base of the distributed objects storage system and is stored.
4. the method according to claim 1, wherein also carrying third predetermined word in the data upload requests
The byte number of the metadata of joint number and the first kind file, the first kind file is greater than the second predetermined word joint number；
Correspondingly, the first kind file is converted into Second Type file, comprising:
According to the third predetermined word joint number by the first kind file division at multiple Second Type files, and described in record
The offset of each Second Type file in multiple Second Type files, wherein the byte number of the Second Type file is described
Third predetermined word joint number；
Each second class is generated according to the offset of each Second Type file and the metadata of the first kind file
The index file of type file.
5. according to the method described in claim 4, being deposited it is characterized in that, the Second Type file is uploaded to distributed objects
Storage system carries out object storage, comprising:
The multiple Second Type file is uploaded in the distributed file system of the distributed objects storage system and is carried out
Object storage, records the data service metadata generated in upload procedure；
The index file of each Second Type file is uploaded to the distributed data base of the distributed objects storage system
In stored；
According to the metadata of the first kind file, the data service metadata, each Second Type file attribute information
Generate control object, wherein the attribute information of the Second Type file include the multiple Second Type file number and
The third predetermined word joint number；
The distributed file system that the control object is uploaded to the distributed objects storage system is subjected to object storage.
6. according to the method described in claim 5, it is characterized in that, the multiple Second Type file is uploaded to the distribution
Object storage is carried out in the distributed file system of formula object storage system, comprising:
It is random to generate Universally Unique Identifier；
Each Second Type file is generated according to the Universally Unique Identifier and the offset of each Second Type file
Key assignments；
By Second Type file each in the multiple Second Type file, the key assignments and described first of each Second Type file
The metadata of type file is uploaded to the distributed objects storage system and carries out object storage.
7. according to the method described in claim 6, it is characterized in that, according to the metadata of the first kind file, the industry
Business metadata, the attribute information generation control object of each Second Type file, comprising:
Target metadata is generated according to the metadata of the first kind file and the data service metadata；
According to the generation of the attribute information of the target metadata, the general exclusive identification code and the Second Type file
Control object.
8. according to the method described in claim 6, it is characterized in that, the multiple Second Type file is uploaded to the distribution
Object storage is carried out in the distributed file system of formula object storage system, further includes:
When uploading error, error message is generated, wherein the Universally Unique Identifier is carried in the error message；
The Second Type file uploaded is verified according to the Universally Unique Identifier, to determine upload failure second
Type file；
The Second Type file for uploading failure is uploaded to progress object storage in the distributed file system again.
9. a kind of distributed objects storage device characterized by comprising
Receiving module, for receiving data upload request carry first kind file in the data upload requests；
Conversion module, in response to the data upload requests, the first kind file to be converted into Second Type file,
Wherein, the difference between the byte number of the Second Type file and the byte number of the first kind file is greater than default threshold
Value；
Uploading module carries out object storage for the Second Type file to be uploaded to distributed objects storage system.
10. a kind of computer equipment, which is characterized in that including processor and for the storage of storage processor executable instruction
The step of device, the processor realizes any one of claims 1 to 8 the method when executing described instruction.
11. a kind of computer readable storage medium, is stored thereon with computer instruction, which is characterized in that described instruction is performed
The step of Shi Shixian any one of claims 1 to 8 the method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910693969.4A CN110413588B (en) | 2019-07-30 | 2019-07-30 | Distributed object storage method and device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910693969.4A CN110413588B (en) | 2019-07-30 | 2019-07-30 | Distributed object storage method and device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110413588A true CN110413588A (en) | 2019-11-05 |
| CN110413588B CN110413588B (en) | 2022-05-17 |
Family
ID=68364091
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910693969.4A Active CN110413588B (en) | 2019-07-30 | 2019-07-30 | Distributed object storage method and device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110413588B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110888837A (en) * | 2019-11-15 | 2020-03-17 | 星辰天合（北京）数据科技有限公司 | Object storage small file merging method and device |
| CN111143366A (en) * | 2019-12-27 | 2020-05-12 | 焦点科技股份有限公司 | High-efficiency storage method for massive large object data |
| CN113285816A (en) * | 2020-02-19 | 2021-08-20 | 华为技术有限公司 | Control request sending method, device and system based on key value configuration |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130311520A1 (en) * | 2010-10-22 | 2013-11-21 | Intec Inc. | Storage-service-provision apparatus,system, service-provision method, and service-provision program |
| CN106909651A (en) * | 2017-02-23 | 2017-06-30 | 郑州云海信息技术有限公司 | A kind of method for being write based on HDFS small documents and being read |
| CN109634916A (en) * | 2018-12-10 | 2019-04-16 | 平安科技（深圳）有限公司 | File storage and method for down loading, device and storage medium |
-
2019
- 2019-07-30 CN CN201910693969.4A patent/CN110413588B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130311520A1 (en) * | 2010-10-22 | 2013-11-21 | Intec Inc. | Storage-service-provision apparatus,system, service-provision method, and service-provision program |
| CN106909651A (en) * | 2017-02-23 | 2017-06-30 | 郑州云海信息技术有限公司 | A kind of method for being write based on HDFS small documents and being read |
| CN109634916A (en) * | 2018-12-10 | 2019-04-16 | 平安科技（深圳）有限公司 | File storage and method for down loading, device and storage medium |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110888837A (en) * | 2019-11-15 | 2020-03-17 | 星辰天合（北京）数据科技有限公司 | Object storage small file merging method and device |
| CN111143366A (en) * | 2019-12-27 | 2020-05-12 | 焦点科技股份有限公司 | High-efficiency storage method for massive large object data |
| CN111143366B (en) * | 2019-12-27 | 2020-12-01 | 焦点科技股份有限公司 | High-efficiency storage method for massive large object data |
| CN113285816A (en) * | 2020-02-19 | 2021-08-20 | 华为技术有限公司 | Control request sending method, device and system based on key value configuration |
| WO2021164330A1 (en) * | 2020-02-19 | 2021-08-26 | 华为技术有限公司 | Key value configuration-based control request sending method, apparatus and system |
| CN113285816B (en) * | 2020-02-19 | 2022-10-28 | 华为技术有限公司 | Control request sending method, device and system based on key value configuration |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110413588B (en) | 2022-05-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| AU2018395919B2 (en) | Efficiently propagating diff values | |
| CN109254733B (en) | Method, device and system for storing data | |
| US9460185B2 (en) | Storage device selection for database partition replicas | |
| US9311326B2 (en) | Virtual file system for automated data replication and review | |
| US11296940B2 (en) | Centralized configuration data in a distributed file system | |
| US9641598B1 (en) | Contextually unique identifier generation service | |
| CN110413588A (en) | Distributed objects storage method, device, computer equipment and storage medium | |
| JP5375972B2 (en) | Distributed file system, data selection method thereof, and program | |
| WO2016187452A1 (en) | Topology aware distributed storage system | |
| CN102142032B (en) | Method and system for reading and writing data of distributed file system | |
| CN103617199A (en) | Data operating method and data operating system | |
| CN111209120A (en) | Data synchronization method and device for microservice and computer readable storage medium | |
| US10970193B2 (en) | Debugging a client synchronization service | |
| CN109597903A (en) | Image file processing apparatus and method, document storage system and storage medium | |
| CN104951475B (en) | Distributed file system and implementation method | |
| US10313284B1 (en) | Upload and share files to a sharing service using a messaging client | |
| Cardellini et al. | Joint operator replication and placement optimization for distributed streaming applications | |
| US10015248B1 (en) | Syncronizing changes to stored data among multiple client devices | |
| US11132401B1 (en) | Distributed hash table based logging service | |
| CN114928604B (en) | File distribution method and device | |
| US11386115B1 (en) | Selectable storage endpoints for a transactional data storage engine | |
| CN103399876A (en) | Distributed file system and file positioning method thereof | |
| CN203911979U (en) | Hadoop cluster storage system | |
| Noor et al. | Benchmarking erasure coding schemes in openStack swift | |
| JP2014191478A (en) | Information processor, information processing method and program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |