CN107220388A - A kind of method and apparatus of upper transmitting file - Google Patents
A kind of method and apparatus of upper transmitting file Download PDFInfo
- Publication number
- CN107220388A CN107220388A CN201710522460.4A CN201710522460A CN107220388A CN 107220388 A CN107220388 A CN 107220388A CN 201710522460 A CN201710522460 A CN 201710522460A CN 107220388 A CN107220388 A CN 107220388A
- Authority
- CN
- China
- Prior art keywords
- file
- file destination
- amazon
- destination
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
- G06F16/164—File meta data generation
- G06F16/166—File name conversion
Abstract
The embodiment of the invention discloses a kind of method and apparatus of upper transmitting file, the data in the file destination are written to temporary file by amazon s3 object storage systems after the file destination that client is sent is received；Wherein, the file destination has its corresponding file name；After the write-in of the data is completed, using rename mechanism, by the corresponding file name of file destination described in the temporary file RNTO, and the related metadata informations of amazon s3 are updated.The operating process re-writes new file without reading data from temporary file again, but directly changes the title of temporary file, updates the mode of metadata information, realizes the upload of file destination, improve the efficiency of transmitting file.
Description
Technical field
The present invention relates to technical field of data processing, more particularly to a kind of method and apparatus of upper transmitting file.
Background technology
Distributed system basic framework (Hadoop) is a distributed system basis developed by Apache funds club
Framework.User can develop distributed program, make full use of the power of cluster in the case where not knowing about distributed low-level details
Carry out high-speed computation and storage.Hadoop architecture mainly realizes distribution by distributed memory system (HDFS)
The basic-level support of storage, realizes that the program that distribution or parallel task are handled is supported by MapReduce.
Amazon object storage system (amazon s3) is serviced disclosed in one, and web application developer can be with
Digital asset, including picture, video, music and document are stored using it.Amazon s3 provide a RESTful API to compile
Journey mode is realized to be interacted with the service.
Hadoop+amazon s3 are docked, by amazon s3 object storage systems high stability, high scalability, highly redundant
Property storage characteristics and Hadoop in the powerful parallel task disposal abilities of MapReduce blend, build more efficiently, it is fast
Prompt, easy-to-use big data platform.
The flow of upper transmitting file is under current Hadoop+amazon s3 docking frameworks：1st, hadoop is called in put orders
Local file is passed to the temporary file in amazon s3 object storage systems correspondence bucket.2nd, after file is uploaded successfully,
Amazon s3 call rename mechanism, and data are read from temporary file, and is written to again of the same name with transmitting file on hadoop
File in.It document flow is uploaded more than can be seen that and upload a successfully file and actually to read file data twice,
Efficiency is not naturally high.
It can be seen that, the speed of transmitting file how is lifted, is those skilled in the art's urgent problem to be solved.
The content of the invention
The purpose of the embodiment of the present invention is to provide a kind of method and apparatus of upper transmitting file, can lift the speed of transmitting file
Rate.
In order to solve the above technical problems, the embodiment of the present invention provides a kind of method of upper transmitting file, including：
Amazon s3 object storage systems receive the file destination that client is sent；The file destination has its corresponding
File name；
Data in the file destination are written to temporary file；
After the write-in of the data is completed, using rename mechanism, by target described in the temporary file RNTO
The corresponding file name of file, and update the related metadata informations of amazon s3.
Optionally, in addition to：
The amazon s3 object storage systems send to the client and upload successful response message.
Optionally, in addition to：
Judge whether the file destination received is complete；
If it is not, then sending the solicited message for obtaining the file destination to the client.
The embodiment of the present invention additionally provides a kind of device of upper transmitting file, including receiving unit, writing unit and renaming
Unit,
The receiving unit, the file destination for receiving client transmission；The file destination has its corresponding file
Title；
Said write unit, for the data in the file destination to be written into temporary file；
After the write-in of the data is completed, then the renaming unit, the renaming unit, for utilizing are triggered
Rename mechanism, by the corresponding file name of file destination described in the temporary file RNTO, and updates amazon s3 phases
The metadata information of pass.
Optionally, in addition to transmitting element,
The transmitting element, successful response message is uploaded for being sent to the client.
Optionally, in addition to judging unit,
Whether the judging unit, the file destination for judging to receive is complete；
If it is not, then triggering the transmitting element, the transmitting element, which is additionally operable to send to the client, obtains the mesh
Mark the solicited message of file.
Amazon s3 object storage systems are receiving the target text that client is sent it can be seen from above-mentioned technical proposal
After part, the data in the file destination are written to temporary file；Wherein, the file destination has its corresponding filename
Claim；After the write-in of the data is completed, using rename mechanism, by file destination pair described in the temporary file RNTO
The file name answered, and update the related metadata informations of amazon s3.The operating process from temporary file without reading number again
According to re-writing new file, but by directly changing the title of temporary file, the mode of metadata information is updated, target is realized
The upload of file, improves the efficiency of transmitting file.
Brief description of the drawings
In order to illustrate the embodiments of the present invention more clearly, the required accompanying drawing used in embodiment will be done simply below
Introduce, it should be apparent that, drawings in the following description are only some embodiments of the present invention, for ordinary skill people
For member, on the premise of not paying creative work, other accompanying drawings can also be obtained according to these accompanying drawings.
Fig. 1 is a kind of flow chart of the method for upper transmitting file provided in an embodiment of the present invention；
Fig. 2 is a kind of structural representation of the device of upper transmitting file provided in an embodiment of the present invention.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is carried out clear, complete
Site preparation is described, it is clear that described embodiment is only a part of embodiment of the invention, rather than whole embodiments.Based on this
Embodiment in invention, those of ordinary skill in the art are not under the premise of creative work is made, and what is obtained is every other
Embodiment, belongs to the scope of the present invention.
In order that those skilled in the art more fully understand the present invention program, with reference to the accompanying drawings and detailed description
The present invention is described in further detail.
In traditional approach, under Hadoop+amazon s3 docking frameworks during upper transmitting file, amazon s3 object storage systems
The data of the local file received are written in temporary file first, then again by the digital independent in temporary file to one
In individual new file, the new file is a file of the same name with local file, after the write-in of data is completed, i.e., by temporary file
In data be completely written to after new file, the temporary file can be deleted.The process needs to read file data twice, causes
It is less efficient that file is uploaded.
Therefore, the embodiments of the invention provide a kind of method and apparatus of upper transmitting file, amazon s3 object storage systems
The file destination that client is sent is received, the data in the file destination are written to temporary file.When the completion data
After write-in, using rename mechanism, by the corresponding file name of file destination described in the temporary file RNTO, and update
Metadata information related amazon s3.By directly changing the title of temporary file, the mode of metadata information is updated, it is real
The upload of existing file destination, improves the efficiency of transmitting file.
Next, a kind of method for upper transmitting file that the embodiment of the present invention is provided is discussed in detail.Fig. 1 is implemented for the present invention
A kind of flow chart of the method for upper transmitting file that example is provided, this method includes：
S101：Amazon s3 object storage systems receive the file destination that client is sent.
Client is received after the hadoop put orders of user's input, corresponding file destination can be uploaded into amazon
S3 object storage systems.
Wherein, the file destination has its corresponding file name.
S102：Data in the file destination are written to temporary file.
Client is when to amazon s3 object storage system transmission objectives files, amazon s3 object storage system meetings
Data in file destination are written in temporary file.
Wherein, the title of the temporary file and the title of file destination are differed.In the specific implementation, often with target
The title of file is plus title of the specific suffix as temporary file, for example, entitled " File " of file destination, accordingly,
The name nominating of temporary file can be " File.copying " by amazon s3 object storage systems.
S103：After the write-in of the data is completed, using rename mechanism, by described in the temporary file RNTO
The corresponding file name of file destination, and update the related metadata informations of amazon s3.
After the data in file destination are completely written to temporary file, amazon s3 object storage systems will interim text
The name modifications of part are into file destination corresponding file name originally.And system can automatically update corresponding metadata information,
So as to complete the renaming of temporary file.
Amazon s3 object storage systems are receiving the target text that client is sent it can be seen from above-mentioned technical proposal
After part, the data in the file destination are written to temporary file；Wherein, the file destination has its corresponding filename
Claim；After the write-in of the data is completed, using rename mechanism, by file destination pair described in the temporary file RNTO
The file name answered, and update the related metadata informations of amazon s3.The operating process from temporary file without reading number again
According to re-writing new file, but the title of temporary file is directly changed, update the mode of metadata information, realize file destination
Upload, improve the efficiency of transmitting file.
After amazon s3 object storage systems complete the renaming to temporary file, then the file destination is illustrated
Amazon s3 object storage systems are successfully uploaded to, now, the amazon s3 object storage systems can be to the client
End, which is sent, uploads successful response message.In order to know that file destination has succeeded in time using the user of the client
Upload.
In view of being influenceed by factors such as networks, there may be the target text of amazon s3 object storage systems reception
Number of packages is according to incomplete phenomenon.The occurrence of in order to avoid this kind, amazon s3 object storage systems may determine that reception
Whether the file destination is complete；If it is not, then sending the solicited message for obtaining the file destination to the client.By this
Deterministic process can effectively ensure being smoothed out for file upload.
Fig. 2 be a kind of structural representation of the device of upper transmitting file provided in an embodiment of the present invention, including receiving unit 21,
Writing unit 22 and renaming unit 23,
The receiving unit 21, the file destination for receiving client transmission；The file destination has its corresponding text
Part title；
Said write unit 22, for the data in the file destination to be written into temporary file；
After the write-in of the data is completed, then the renaming unit 23, the renaming unit 23, for profit are triggered
Rename mechanism is used, by the corresponding file name of file destination described in the temporary file RNTO, and amazon s3 are updated
Related metadata information.
Optionally, in addition to transmitting element,
The transmitting element, successful response message is uploaded for being sent to the client.
Optionally, in addition to judging unit,
Whether the judging unit, the file destination for judging to receive is complete；
If it is not, then triggering the transmitting element, the transmitting element, which is additionally operable to send to the client, obtains the mesh
Mark the solicited message of file.
The explanation of feature may refer to the related description of embodiment corresponding to Fig. 1 in embodiment corresponding to Fig. 2, here no longer
Repeat one by one.
Amazon s3 object storage systems are receiving the target text that client is sent it can be seen from above-mentioned technical proposal
After part, the data in the file destination are written to temporary file；Wherein, the file destination has its corresponding filename
Claim；After the write-in of the data is completed, using rename mechanism, by file destination pair described in the temporary file RNTO
The file name answered, and update the related metadata informations of amazon s3.The operating process from temporary file without reading number again
According to re-writing new file, but the title of temporary file is directly changed, update the mode of metadata information, realize file destination
Upload, improve the efficiency of transmitting file.
A kind of method and apparatus of the upper transmitting file provided above the embodiment of the present invention is described in detail.Explanation
The embodiment of each in book is described by the way of progressive, what each embodiment was stressed be it is different from other embodiment it
Place, between each embodiment identical similar portion mutually referring to.For device disclosed in embodiment, due to itself and reality
Apply that method disclosed in example is corresponding, so description is fairly simple, related part is referring to method part illustration.It should refer to
Go out, for those skilled in the art, under the premise without departing from the principles of the invention, can also be to the present invention
Some improvement and modification are carried out, these are improved and modification is also fallen into the protection domain of the claims in the present invention.
Professional further appreciates that, with reference to the unit of each example of the embodiments described herein description
And algorithm steps, can be realized with electronic hardware, computer software or the combination of the two, in order to clearly demonstrate hardware and
The interchangeability of software, generally describes the composition and step of each example according to function in the above description.These
Function is performed with hardware or software mode actually, depending on the application-specific and design constraint of technical scheme.Specialty
Technical staff can realize described function to each specific application using distinct methods, but this realization should not
Think beyond the scope of this invention.
Directly it can be held with reference to the step of the method or algorithm that the embodiments described herein is described with hardware, processor
Capable software module, or the two combination are implemented.Software module can be placed in random access memory (RAM), internal memory, read-only deposit
Reservoir (ROM), electrically programmable ROM, electrically erasable ROM, register, hard disk, moveable magnetic disc, CD-ROM or technology
In any other form of storage medium well known in field.
Claims (6)
1. a kind of method of upper transmitting file, it is characterised in that including：
Amazon s3 object storage systems receive the file destination that client is sent；The file destination has its corresponding file
Title；
Data in the file destination are written to temporary file；
After the write-in of the data is completed, using rename mechanism, by file destination described in the temporary file RNTO
Corresponding file name, and update the related metadata informations of amazon s3.
2. according to the method described in claim 1, it is characterised in that also include：
The amazon s3 object storage systems send to the client and upload successful response message.
3. method according to claim 1 or 2, it is characterised in that also include：
Judge whether the file destination received is complete；
If it is not, then sending the solicited message for obtaining the file destination to the client.
4. a kind of device of upper transmitting file, it is characterised in that including receiving unit, writing unit and renaming unit,
The receiving unit, the file destination for receiving client transmission；The file destination has its corresponding file name；
Said write unit, for the data in the file destination to be written into temporary file；
After the write-in of the data is completed, then the renaming unit, the renaming unit, for utilizing rename are triggered
Mechanism, by the corresponding file name of file destination described in the temporary file RNTO, and updates the related members of amazon s3
Data message.
5. device according to claim 4, it is characterised in that also including transmitting element,
The transmitting element, successful response message is uploaded for being sent to the client.
6. the device according to claim 4 or 5, it is characterised in that also including judging unit,
Whether the judging unit, the file destination for judging to receive is complete；
If it is not, then triggering the transmitting element, the transmitting element, which is additionally operable to send to the client, obtains the target text
The solicited message of part.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710522460.4A CN107220388A (en) | 2017-06-30 | 2017-06-30 | A kind of method and apparatus of upper transmitting file |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710522460.4A CN107220388A (en) | 2017-06-30 | 2017-06-30 | A kind of method and apparatus of upper transmitting file |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107220388A true CN107220388A (en) | 2017-09-29 |
Family
ID=59950667
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710522460.4A Pending CN107220388A (en) | 2017-06-30 | 2017-06-30 | A kind of method and apparatus of upper transmitting file |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107220388A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109756484A (en) * | 2018-12-12 | 2019-05-14 | 杭州数梦工场科技有限公司 | Control method, control device, gateway and the medium of gateway based on object storage |
| CN110990348A (en) * | 2018-09-30 | 2020-04-10 | 北京国双科技有限公司 | Message notification method, device and system |
| CN112631514A (en) * | 2020-12-17 | 2021-04-09 | 龙存科技（北京）股份有限公司 | File duplicate removal method and system applied to cloud disk system |
| CN112866409A (en) * | 2021-02-08 | 2021-05-28 | 深圳市杉岩数据技术有限公司 | Object transmission and collection method based on object storage fragment uploading and terminal equipment |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102946440A (en) * | 2012-12-04 | 2013-02-27 | 乐视致新电子科技（天津）有限公司 | Automatic cloud synchronization and recovery system based on cloud storage technology |
| CN105095418A (en) * | 2015-07-13 | 2015-11-25 | 浪潮(北京)电子信息产业有限公司 | Method and apparatus for processing write request |
| CN105630900A (en) * | 2015-12-18 | 2016-06-01 | 浪潮(北京)电子信息产业有限公司 | Data storage method and system |
| CN105812427A (en) * | 2014-12-31 | 2016-07-27 | 中国电信股份有限公司 | Methods and devices for file uploading and downloading, and file server |
-
2017
- 2017-06-30 CN CN201710522460.4A patent/CN107220388A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102946440A (en) * | 2012-12-04 | 2013-02-27 | 乐视致新电子科技（天津）有限公司 | Automatic cloud synchronization and recovery system based on cloud storage technology |
| CN105812427A (en) * | 2014-12-31 | 2016-07-27 | 中国电信股份有限公司 | Methods and devices for file uploading and downloading, and file server |
| CN105095418A (en) * | 2015-07-13 | 2015-11-25 | 浪潮(北京)电子信息产业有限公司 | Method and apparatus for processing write request |
| CN105630900A (en) * | 2015-12-18 | 2016-06-01 | 浪潮(北京)电子信息产业有限公司 | Data storage method and system |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110990348A (en) * | 2018-09-30 | 2020-04-10 | 北京国双科技有限公司 | Message notification method, device and system |
| CN110990348B (en) * | 2018-09-30 | 2023-04-18 | 北京国双科技有限公司 | Message notification method, device and system |
| CN109756484A (en) * | 2018-12-12 | 2019-05-14 | 杭州数梦工场科技有限公司 | Control method, control device, gateway and the medium of gateway based on object storage |
| CN109756484B (en) * | 2018-12-12 | 2021-10-15 | 杭州数梦工场科技有限公司 | Object storage-based gateway control method, control device, gateway and medium |
| CN112631514A (en) * | 2020-12-17 | 2021-04-09 | 龙存科技（北京）股份有限公司 | File duplicate removal method and system applied to cloud disk system |
| CN112866409A (en) * | 2021-02-08 | 2021-05-28 | 深圳市杉岩数据技术有限公司 | Object transmission and collection method based on object storage fragment uploading and terminal equipment |
| CN112866409B (en) * | 2021-02-08 | 2022-10-11 | 深圳市杉岩数据技术有限公司 | Object transmission and collection method based on object storage fragment uploading and terminal equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11734125B2 (en) | Tiered cloud storage for different availability and performance requirements | |
| CN103765423B (en) | Gathering transaction data associated with locally stored data files | |
| US8230185B2 (en) | Method for optimizing cleaning of maps in FlashCopy cascades containing incremental maps | |
| TW505852B (en) | Computer, data sharing system, and method for maintaining replica consistency | |
| CN107220388A (en) | A kind of method and apparatus of upper transmitting file | |
| US11188499B2 (en) | Storing and retrieving restricted datasets to and from a cloud network with non-restricted datasets | |
| US9930130B2 (en) | Processing hybrid data using a single web client | |
| CN106446159B (en) | A kind of method of storage file, the first virtual machine and name node | |
| CN106161523A (en) | A kind of data processing method and equipment | |
| CN106933747B (en) | Data-storage system and date storage method based on multithread | |
| CN107577420A (en) | Document handling method and device, server | |
| US10747458B2 (en) | Methods and systems for improving efficiency in cloud-as-backup tier | |
| CN105653209A (en) | Object storage data transmitting method and device | |
| CN108255989A (en) | Picture storage method, device, terminal device and computer storage media | |
| US20120311375A1 (en) | Redirecting requests to secondary location during temporary outage | |
| CN102045369A (en) | Information push system and method for realizing cloud copy | |
| US20130339307A1 (en) | Managing system image backup | |
| US20180248947A1 (en) | Metered Network Synchronization | |
| CN103595808A (en) | Method and device for pushing update information of file | |
| CN106599323A (en) | Method and apparatus for realizing distributed pipeline in distributed file system | |
| TW201638799A (en) | Distributed storage of software images in computing systems | |
| US10860184B2 (en) | Sharing groups for capturing digital media | |
| CN104166571B (en) | The installation control method and device of a kind of software | |
| CN111880831A (en) | Method and device for synchronously updating server, computer equipment and storage medium | |
| CN112905541A (en) | Mirror image warehouse garbage cleaning method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170929 |