CN111367865A - AWS 3-based file fragment uploading implementation method - Google Patents
AWS 3-based file fragment uploading implementation method Download PDFInfo
- Publication number
- CN111367865A CN111367865A CN202010161252.8A CN202010161252A CN111367865A CN 111367865 A CN111367865 A CN 111367865A CN 202010161252 A CN202010161252 A CN 202010161252A CN 111367865 A CN111367865 A CN 111367865A
- Authority
- CN
- China
- Prior art keywords
- file
- uploading
- uploaded
- information
- fragment
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 239000012634 fragment Substances 0.000 title claims abstract description 54
- 238000000034 method Methods 0.000 title claims abstract description 34
- 230000005540 biological transmission Effects 0.000 claims abstract description 32
- 238000001514 detection method Methods 0.000 abstract description 2
- 238000013467 fragmentation Methods 0.000 description 4
- 238000006062 fragmentation reaction Methods 0.000 description 4
- 238000013500 data storage Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 239000000344 soap Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/604—Tools and structures for managing or administering access control systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2221/00—Indexing scheme relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F2221/21—Indexing scheme relating to G06F21/00 and subgroups addressing additional information or applications relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F2221/2141—Access rights, e.g. capability lists, access control lists, access tables, access matrices
Abstract
The invention discloses a method for realizing file fragment uploading based on AWS 3, relating to the technical field of file transmission; starting a file uploading task by using an AWS 3, setting parameters of file information uploading, fragmenting an uploaded single file according to corresponding parameters, marking a fragment number, uploading the fragments according to the fragment number, reestablishing a transmission request if the task is suspended, continuing uploading, acquiring uploaded file fragment information, starting breakpoint continuous transmission until a complete file is successfully uploaded, acquiring ETag information, and storing the uploaded file; the problem of the server pressure is too big because of greatly reduced whole upload, has promoted the efficiency of file upload again greatly, through practical detection, can reduce server pressure between several times to several ten times, efficiency promotes more than several times, and the probability of uploading failure reduces greatly after adopting the burst to upload simultaneously.
Description
Technical Field
The invention discloses a method for realizing file uploading, relates to the technical field of file transmission, and particularly relates to a method for realizing file fragment uploading based on AWS 3.
Background
The AWS 3 is Amazon Simple Storage Service, which is an Internet Storage solution. The uploading is a process of transmitting a file to a server through a client, that is, a process of transmitting the file to a background through a front end. The AWS 3 can support users at any time anywhere on the Web, storing and retrieving data of any size, and the AWS 3 itself provides a simple and intuitive management console to handle these tasks. Amazon S3 stores the data as objects in the storage area. An object consists of a file and optionally any metadata describing the file.
In the process of data uploading, if the content to be transmitted is too large, various problems such as large memory consumption, IIS timeout, and the need of re-uploading when problems occur are easily encountered in the transmission process.
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides a method for realizing file fragment uploading based on AWS 3, and the method solves the problems of large memory consumption, IIS overtime, problem of re-uploading and the like when a large file is uploaded by using a fragment uploading mode.
The specific scheme provided by the invention is as follows:
a realization method for uploading files in a fragmentation way based on AWS 3 is characterized in that an AWS 3 is utilized to start a file uploading task, parameters of uploading file information are set, an uploaded single file is fragmented according to corresponding parameters, the number of the fragments is marked,
uploading the fragments according to the fragment numbers, if the task is suspended, reestablishing the transmission request, continuing uploading, obtaining the uploaded file fragment information, starting breakpoint continuous transmission until the complete file is uploaded successfully, obtaining the ETag information, and storing the uploaded file.
In the file uploading process, the file uploading progress is updated, and the file uploading progress comprises file uploading information and the number of bytes of the current fragment uploading completion.
In the implementation method, if the file uploading fails, the parameters of the file information uploading are reset, and the single file uploading is carried out again according to the reset parameters.
When the file uploading task is started in the implementation method, a timer is set, and the file uploading task is scheduled.
An AWS 3-based file fragment uploading implementation system comprises a setting module and a transmission module,
the setting module starts a file uploading task by using the AWS 3, sets parameters of uploading file information, divides the uploaded single file according to the corresponding parameters, marks the number of the divided file,
and the transmission module uploads the fragments according to the fragment number, reestablishes the transmission request if the task is suspended, then continues uploading, acquires the uploaded file fragment information, starts breakpoint continuous transmission until the complete file is uploaded successfully, acquires the ETag information and stores the uploaded file.
In the process of uploading the file, the transmission module updates the file uploading progress, and the file uploading progress comprises file uploading information and the number of bytes of the current fragment uploading completion.
If the file uploading fails in the implementation system, the setting module resets the parameters of the file information uploading, and the transmission module uploads the single file again according to the reset parameters.
When the setting module in the implementation system starts a file uploading task, a timer is set, and the file uploading task is scheduled.
The invention has the advantages that:
the invention provides a realization method for uploading files in a fragmentation way based on AWS 3, wherein large files can be uploaded to a server after being fragmented at a client by AWS 3, so that the problem of overlarge server pressure caused by integral uploading is greatly reduced, the efficiency of uploading files is greatly improved, the pressure of the server can be reduced by several times to several ten times through actual detection, the efficiency is improved by more than several times, and meanwhile, the probability of uploading failure is greatly reduced after the files are uploaded in the fragmentation way.
Drawings
FIG. 1 is a schematic process diagram of the process of the present invention.
Detailed Description
The present invention is further described below in conjunction with the following figures and specific examples so that those skilled in the art may better understand the present invention and practice it, but the examples are not intended to limit the present invention.
The invention provides a realization method for uploading files in a fragmentation way based on AWS 3, which comprises the steps of starting a file uploading task by using AWS 3, setting parameters of file information uploading, fragmenting the uploaded single file according to the corresponding parameters, marking the number of fragments,
uploading the fragments according to the fragment numbers, if the task is suspended, reestablishing the transmission request, continuing uploading, obtaining the uploaded file fragment information, starting breakpoint continuous transmission until the complete file is uploaded successfully, obtaining the ETag information, and storing the uploaded file.
The method of the invention is used for realizing the file fragment uploading based on the AWS 3, and the specific process is as follows:
sending a request to call service to a server, starting a file upload task timer based on AWS 3, scheduling a file upload task, setting parameters of uploaded file information,
starting a file uploading task, starting to upload a single file,
updating the file uploading progress in the uploading process, including the contents of file uploading information and the number of bytes of the current fragment uploading completion,
the uploaded single file is fragmented according to corresponding parameters, the fragment number is marked, the fragment uploading is started,
uploading the fragments, preparing the partNumber of the next fragment to be uploaded,
if the task is suspended, deleting the current fragment, checking the pause lock to prevent repeated operation, waiting until the actual transmission request is established, releasing the pause lock,
when the user clicks to continue uploading again, the uploaded file fragment information is acquired, the breakpoint continuous transmission operation is started, when the uploading fails, the uploaded file information parameter is reset, the uploading is restarted,
after all files are uploaded successfully, extracting an ETag information list, taking the situation that a storage bucket does not set ETag in Exposed-heads set in a cross-domain into consideration, acquiring ETag information from a server,
the timer is cleared and the parameters are reset.
And saving the uploaded file information.
In the above process, the method of the present invention stores files in buckets, which are the base containers for data storage in Amazon S3, where an unlimited amount of data may be stored, and developers may upload the required number of objects to Amazon S3 buckets. Each object is stored and retrieved using a unique key value assigned by the developer. Other personnel using the amazon s3 bucket to upload or download data may grant or deny access. Three types of users are granted permission to upload and download. The authentication mechanism may help to secure data against unauthorized access.
The method of the invention can also be matched with any Internet development kit for use through standard REST and SOAP interfaces.
Meanwhile, the invention also provides a system for realizing file fragment uploading based on the AWS 3, which comprises a setting module and a transmission module,
the setting module starts a file uploading task by using the AWS 3, sets parameters of uploading file information, divides the uploaded single file according to the corresponding parameters, marks the number of the divided file,
and the transmission module uploads the fragments according to the fragment number, reestablishes the transmission request if the task is suspended, then continues uploading, acquires the uploaded file fragment information, starts breakpoint continuous transmission until the complete file is uploaded successfully, acquires the ETag information and stores the uploaded file.
The system can be established at a client, and realizes file fragment uploading based on AWS 3, and the specific process is as follows:
confirming the system login state of the invention, sending a request to the server to call service, starting a file uploading task timer by a setting module based on AWS 3, scheduling a file uploading task, setting parameters of file information uploading, fragmenting the uploaded single file according to the corresponding parameters, marking the fragment number,
the transmission module starts a file uploading task, starts to upload a single file,
updating the file uploading progress in the uploading process, including the contents of file uploading information and the number of bytes of the current fragment uploading completion,
slice uploading is performed starting according to the slice number,
uploading the fragments, preparing the partNumber of the next fragment to be uploaded,
if the task is suspended, deleting the current fragment, checking the pause lock to prevent repeated operation, waiting until the actual transmission request is established, releasing the pause lock,
when the user clicks to continue uploading again, the uploaded file fragment information is acquired, the breakpoint continuous transmission operation is started, when the uploading fails, the uploaded file information parameter is reset, the uploading is restarted,
after all files are uploaded successfully, extracting an ETag information list, taking the situation that a storage bucket does not set ETag in Exposed-heads set in a cross-domain into consideration, acquiring ETag information from a server,
the timer is cleared and the parameters are reset.
And saving the uploaded file information.
In the above process, the present system stores files in buckets, which are the base containers for data storage in Amazon S3, where an unlimited amount of data may be stored, and developers may upload the required number of objects to Amazon S3 buckets. Each object is stored and retrieved using a unique key value assigned by the developer. Other personnel using the amazon s3 bucket to upload or download data may grant or deny access. Three types of users are granted permission to upload and download. The authentication mechanism may help to secure data against unauthorized access.
The above-mentioned embodiments are merely preferred embodiments for fully illustrating the present invention, and the scope of the present invention is not limited thereto. The equivalent substitution or change made by the technical personnel in the technical field on the basis of the invention is all within the protection scope of the invention. The protection scope of the invention is subject to the claims.
Claims (8)
1. A realization method for uploading file fragments based on AWS 3 is characterized in that an AWS 3 is utilized to start a file uploading task, parameters of uploading file information are set, the uploaded single file is fragmented according to the corresponding parameters, the number of the fragments is marked,
uploading the fragments according to the fragment numbers, if the task is suspended, reestablishing the transmission request, continuing uploading, obtaining the uploaded file fragment information, starting breakpoint continuous transmission until the complete file is uploaded successfully, obtaining the ETag information, and storing the uploaded file.
2. The implementation method of claim 1, wherein during the process of uploading the file, the file uploading progress is updated, and the file uploading progress comprises file uploading information and the number of bytes of uploading completion of the current fragment.
3. The method as claimed in claim 1 or 2, wherein if the file uploading fails, the parameters of the file information uploading are reset, and the single file uploading is performed again according to the reset parameters.
4. The method of claim 3, wherein when the file upload task is started, a timer is set to schedule the file upload task.
5. An AWS 3-based file fragment uploading implementation system is characterized by comprising a setting module and a transmission module,
the setting module starts a file uploading task by using the AWS 3, sets parameters of uploading file information, divides the uploaded single file according to the corresponding parameters, marks the number of the divided file,
and the transmission module uploads the fragments according to the fragment number, reestablishes the transmission request if the task is suspended, then continues uploading, acquires the uploaded file fragment information, starts breakpoint continuous transmission until the complete file is uploaded successfully, acquires the ETag information and stores the uploaded file.
6. The system of claim 5, wherein the transmission module updates a file uploading schedule during the file uploading process, wherein the file uploading schedule comprises file uploading information and the number of bytes of the current fragment uploading completion.
7. The system of claim 5 or 6, wherein if the file uploading fails, the setting module resets the parameters of the file information uploading, and the transmission module uploads the single file again according to the reset parameters.
8. The system of claim 7, wherein the setting module sets a timer to schedule the file upload task when the setting module starts the file upload task.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010161252.8A CN111367865A (en) | 2020-03-10 | 2020-03-10 | AWS 3-based file fragment uploading implementation method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010161252.8A CN111367865A (en) | 2020-03-10 | 2020-03-10 | AWS 3-based file fragment uploading implementation method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111367865A true CN111367865A (en) | 2020-07-03 |
Family
ID=71210474
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010161252.8A Pending CN111367865A (en) | 2020-03-10 | 2020-03-10 | AWS 3-based file fragment uploading implementation method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111367865A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106021381A (en) * | 2016-05-11 | 2016-10-12 | 北京搜狐新媒体信息技术有限公司 | Data access/storage method and device for cloud storage service system |
| CN108011967A (en) * | 2017-12-15 | 2018-05-08 | 四川长虹电器股份有限公司 | Picture upload method based on AWS micro services |
| CN109067733A (en) * | 2018-07-27 | 2018-12-21 | 成都华为技术有限公司 | The method and apparatus for sending data, and the method and apparatus for receiving data |
| CN109714330A (en) * | 2018-12-24 | 2019-05-03 | 武汉烽火众智数字技术有限责任公司 | A kind of breaker point continuous transmission method and system of across a network |
| CN110401725A (en) * | 2019-08-23 | 2019-11-01 | 深圳市鹰硕技术有限公司 | File uploading method, device, terminal, server, system and storage medium |
-
2020
- 2020-03-10 CN CN202010161252.8A patent/CN111367865A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106021381A (en) * | 2016-05-11 | 2016-10-12 | 北京搜狐新媒体信息技术有限公司 | Data access/storage method and device for cloud storage service system |
| CN108011967A (en) * | 2017-12-15 | 2018-05-08 | 四川长虹电器股份有限公司 | Picture upload method based on AWS micro services |
| CN109067733A (en) * | 2018-07-27 | 2018-12-21 | 成都华为技术有限公司 | The method and apparatus for sending data, and the method and apparatus for receiving data |
| CN109714330A (en) * | 2018-12-24 | 2019-05-03 | 武汉烽火众智数字技术有限责任公司 | A kind of breaker point continuous transmission method and system of across a network |
| CN110401725A (en) * | 2019-08-23 | 2019-11-01 | 深圳市鹰硕技术有限公司 | File uploading method, device, terminal, server, system and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 吴尚高: "《基于AWS的文件转换服务的设计和实现》", 《中国优秀博硕士学位论文全文数据库(硕士)信息科技辑》 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8549327B2 (en) | Background service process for local collection of data in an electronic discovery system | |
| US10268471B2 (en) | Method for upgrading terminal system, terminal, and system | |
| EP3179701B1 (en) | File upload and download methods and associated server | |
| CN1852534A (en) | Data backing-up and recovering method and system | |
| CN105095022B (en) | Data backup method and device | |
| US20070038680A1 (en) | Management of mobile-device data | |
| CN109309631A (en) | A kind of method and device based on universal network file system write-in data | |
| US20080208928A1 (en) | Device Management System and Method for Managing Device Management Object | |
| JP2010081380A (en) | Image processor, and network system | |
| CN1224218C (en) | Remote control system and method for domestic network | |
| US11652788B2 (en) | Communication system including first mediation apparatus and second mediation apparatus for transmitting command received from management server | |
| JP2010182260A (en) | Thin client system, management server, thin client terminal, and thin client server | |
| JP7131044B2 (en) | Program and communication system | |
| CN109697112B (en) | Distributed intensive one-stop operating system and implementation method | |
| CN103036933A (en) | System and method for cloud scrapbook | |
| JP6042243B2 (en) | Monitoring system, monitoring server, and monitoring method | |
| CN111367865A (en) | AWS 3-based file fragment uploading implementation method | |
| US20120209897A1 (en) | Disc image file management system and disc image file management method | |
| CN111209054A (en) | Configuration method of terminal equipment, terminal management equipment and electronic equipment | |
| JP2008108046A (en) | Transaction processing system, transaction processing method and program therefor | |
| EP1608124A2 (en) | Device and methods for downloading data | |
| JP2016118837A (en) | Thin client system, and document file control method of thin client | |
| JP2015228177A (en) | Image forming apparatus, and log transfer system provided with the same | |
| CN112699084B (en) | File cleaning method and device, storage medium and electronic device | |
| JP2008270997A5 (en) |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200703 |
|
| RJ01 | Rejection of invention patent application after publication |