CN108011967A - Picture upload method based on AWS micro services - Google Patents
Picture upload method based on AWS micro services Download PDFInfo
- Publication number
- CN108011967A CN108011967A CN201711345873.6A CN201711345873A CN108011967A CN 108011967 A CN108011967 A CN 108011967A CN 201711345873 A CN201711345873 A CN 201711345873A CN 108011967 A CN108011967 A CN 108011967A
- Authority
- CN
- China
- Prior art keywords
- picture
- upload
- aws
- micro services
- dynamodb
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/34—Network arrangements or protocols for supporting network services or applications involving the movement of software or configuration parameters
Abstract
The invention belongs to technical field of the computer network, it discloses a kind of picture upload method based on AWS micro services, solves to utilize the bottleneck problem of the performance risk uploaded present in storage service of FastDFS progress document pictures, high availability and elastic dilatation ability in conventional art.This method comprises the following steps：A. upload service split into upload, download two micro services；B. using AWS service API, correlation technique is called to obtain the access key information provided in environment；C. upload request is handled, picture metamessage is write in the corresponding table of DynamoDB databases；The UUID generated when d. according to write-in DynamoDB databases, by object picture renaming and is uploaded to S3 servers relevant position.
Description
Technical field
The invention belongs to technical field of the computer network, and in particular to a kind of picture upload method based on AWS micro services.
Background technology
The scanning plug-in application of general cloud platform service system, employs the Web layer architectures of B/S：
1) DynamSoft plug-in units are installed on client browsers, associate scanner, the scanning of papery document is uploaded into Web clothes
Business device.
2) electronic documents are uploaded directly into Web server by client browser.
3) .Nginx Web servers, back-end processing use Java, are built in Tomcat Servlet.In addition,
The installation kit of DynamSoft plug-in units is placed on Nginx Web servers.
4) documents storage use FastDFS distributed memory systems, include tracking server (tracker server) and
Storage server (storage server)：Tracking server, mainly does traffic control, plays the effect of equilibrium；Storage service
Device, it is main that capacity and backup services are provided.
5) documents metamessage stores, and includes document ID, title, document storage address etc., using MySQL database, is placed in
In one management table, its deployment uses leader follower replication pattern.
6) other application of cloud platforms service system can be by document ID or storage address, to read document complete information.
As it can be seen that in traditional cloud platform service system, document/picture is carried out using FastDFS and is stored, there are following for it
Defect：Do not confirm that metamessage whether there is in MySQL when document is downloaded first, and access and be directly entered FastDFS storage ends, exist
Performance risk；
The elastic dilatation of FastDFS, there are the risk of management and maintenance；In addition, according to Changhong practical experience, near the end of month
Document upload amount can be especially big, will be by Web server due to uploading every time, its high availability and elastic dilatation ability will
As bottleneck；
Other application can directly access MySQL database and FastDFS storages obtain the complete information of document, without external
The API of inquiry, is directly exposed to database and FastDFS storages outside, becomes directly relying on for other application, this can produce number
The problem of according to inconsistent and access performance bottleneck with interface.
The content of the invention
The technical problems to be solved by the invention are：It is proposed a kind of picture upload method based on AWS micro services, solve to pass
The performance risk, high availability and the elasticity that upload present in storage service of document picture are carried out in system technology using FastDFS
The bottleneck problem of dilatation ability.
The present invention solves the technical solution that above-mentioned technical problem uses：
Based on the picture upload method of AWS micro services, comprise the following steps：
A. upload service split into upload, download two micro services；
B. using AWS service API, correlation technique is called to obtain the access key information provided in environment；
C. upload request is handled, picture metamessage is write in the corresponding table of DynamoDB databases；
The UUID generated when d. according to write-in DynamoDB databases, by object picture renaming and is uploaded to S3 servers
Relevant position.
As further optimization, step a is specifically included：
Split according to SpringMVC frames, former upload project is retained to Model layers, respective upload is downloaded
Method is put into Controller layers respective respectively；Utilize DynamoDBMapper pairs of the high-level programming interface (HLPI) of DynamoDB
Model layers of class object is transformed：Addition is corresponding to explain, and the attribute of the attribute of class and table is mapped.
As further optimization, step b is specifically included：
By the configuration access ID in configuration file and private key is accessed, by spring frames
PropertyPlaceholderConfigurer classes are come parsing Java Properties configuration attributes and read.
Optimize as further, in step c, injected in picture metamessage storing process using the constructed fuction of Spring
AWS DynamoDBClient examples, are then realized using database persistence technology and stored.
Optimize as further, in step d, allocation methods uploaded by multipart in object picture upload procedure,
Realize the request for handling multiple pictures；In the upload procedure for handling each picture, generated by pre-sign URL technologies pre-
Sign URL, then uploading pictures are to specified URL；According to the object information of return, the UUID of picture is obtained, by image renamed as
UUID and it is stored on S3 servers, creates the example of operation S3, use TransferManager example tm, calls
Upload () method passes to object picture under the catalogue that S3 server users specify.
As further optimization, this method further includes step：
E. whenever download pictures, inquire about it first with the UUID of the picture unique id generated and whether there is DynamoDB
In, asked if it does not exist, then not sent to S3 servers.
The beneficial effects of the invention are as follows：
1) in picture upload procedure, by pre-sign URL technologies generate pre-signature URL, then uploading pictures to specify
URL；Bucket title, object key and expired are provided by creating the example of GeneratePresignedUrlRequest classes
On the date, so as to be controlled to Accessor Access's object, solve storage URL unprotects, avoids attacker and bypasses application clothes
Business device, distributes request arbituarily and abuses this URL.
2) by increasing document query function, generation is explained by DynamoDBGeneratedUuid using when uploading
Unique identifier, goes inquiry DynamoDB to have not corresponding picture metamessage, if do not found, directly whenever downloading with this UUID
The document being not present is taken back, is avoided every time to the extra pressure caused by S3 storage server requests.
3) upload of document is downloaded peels off into two micro services from Web server；Uploaded by document and download separation, can root
According to needing to respective service dilatation.
Embodiment
The present invention is directed to propose a kind of picture upload method based on AWS micro services, solves to utilize in conventional art
The bottle of performance risk, high availability and elastic dilatation ability present in the upload storage service of FastDFS progress document pictures
Neck problem.
The present invention solves to use using safe, expansible, High Availabitity the characteristic of AWS the cloud service S3 and DynamoDB provided
Family real time access amount brings greatly the pressure of system, operates the reality of S3 and DynamoDB by api interface using Java correlation techniques
Relevant business is showed.
In specific implementation, the present invention uses the design of AWS micro services, replaces FastDFS storage pictures etc. big using S3
Document object, DynamoDB are used for storage object metamessage instead of MySQL.
Implementation steps are as follows：
1st, original upload service split into upload, download two micro services：
Split according to SpringMVC frame theories, by former upload project retain Model layers, will be respective on descend
Support method is put into Controller layers respective respectively.Utilize DynamoDBMapper pairs of the high-level programming interface (HLPI) of DynamoDB
Model layers of class object is transformed, and mainly addition is corresponding explains, and the attribute of the attribute of class and table is mapped, such as@
Class is mapped to corresponding table by DynamoDBTable (tableName=).
2nd, API is serviced using AWS, calls correlation technique to obtain the access key information provided in environment；
Configuration access ID aws_access_key_id and access private key, aws_secret_access_ in configuration file
Key, is matched somebody with somebody by the PropertyPlaceholderConfigurer classes in spring frames to parse Java Properties
Put attribute and read.
3rd, upload request is handled, picture metamessage is write in the corresponding table of DynamoDB databases；
AWS DynamoDBClient examples are injected using the constructed fuction of Spring in picture metamessage storing process, so
Afterwards storage is realized using database persistence technology.The request uploaded by handling picture, obtains uploading pictures name, picture storage
The attributes such as bucket, example is generated using database persistence technology, that is, by the constructed fuction injection of Spring；Use
The additions and deletions that DynamoDBMapper object instances are carried out to database, which change, looks into operation.Specifically when stored, mapper=new
DynamoDBMapper(client)；Mapper.save () is realized writes DynamoDB by object metamessage.Finally need
It will be inserted into successfully to record to return after database and, for subsequent treatment.
4th, the UUID generated according to write-in DynamoDB databases, by object picture renaming and is uploaded to S3 server phases
Answer position.
By this upload allocation methods of multipart in upload procedure, the request for handling multiple pictures is realized；Locating
In the upload procedure for managing each picture, pre-signature URL, then uploading pictures are generated to specified URL by pre-sign URL technologies.
According to the object information of return, the UUID of picture is obtained, by image renamed as UUID and is stored on S3 servers, created
The example of S3 is operated, using TransferManager example tm, calls upload () method that object picture is passed to S3 services
Under the catalogue that device user specifies.
Claims (6)
1. the picture upload method based on AWS micro services, it is characterised in that comprise the following steps：
A. upload service split into upload, download two micro services；
B. using AWS service API, correlation technique is called to obtain the access key information provided in environment；
C. upload request is handled, picture metamessage is write in the corresponding table of DynamoDB databases；
D. according to the UUID that generates during write-in DynamoDB databases, by object picture renaming and to be uploaded to S3 servers corresponding
Position.
2. the picture upload method as claimed in claim 1 based on AWS micro services, it is characterised in that
Step a is specifically included：
Split according to SpringMVC frames, former upload project is retained to Model layers, by respective upload method for down loading
It is put into respectively Controller layers respective；Using the high-level programming interface (HLPI) DynamoDBMapper of DynamoDB to Model layers
Class object transformed：Addition is corresponding to explain, and the attribute of the attribute of class and table is mapped.
3. the picture upload method as claimed in claim 1 based on AWS micro services, it is characterised in that
Step b is specifically included：
By the configuration access ID in configuration file and private key is accessed, by spring frames
PropertyPlaceholderConfigurer classes are come parsing Java Properties configuration attributes and read.
4. the picture upload method as claimed in claim 1 based on AWS micro services, it is characterised in that
It is real using the constructed fuction injection AWS DynamoDBClient of Spring in picture metamessage storing process in step c
Example, is then realized using database persistence technology and stored.
5. the picture upload method as claimed in claim 1 based on AWS micro services, it is characterised in that
In step d, allocation methods are uploaded by multipart in object picture upload procedure, realizes and handles multiple pictures
Request；In the upload procedure for handling each picture, pre-signature URL, then uploading pictures are generated by pre-sign URL technologies
To specified URL；According to the object information of return, the UUID of picture is obtained, by image renamed as UUID and is stored in S3 clothes
It is engaged on device, creates the example of operation S3, using TransferManager example tm, calls upload () method by object picture
Pass under the catalogue that S3 server users specify.
6. the picture upload method as claimed in claim 5 based on AWS micro services, it is characterised in that
This method further includes step：
E. whenever download pictures, inquire about it first with the UUID of the picture unique id generated and whether there is in DynamoDB,
Asked if it does not exist, then not sent to S3 servers.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711345873.6A CN108011967A (en) | 2017-12-15 | 2017-12-15 | Picture upload method based on AWS micro services |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711345873.6A CN108011967A (en) | 2017-12-15 | 2017-12-15 | Picture upload method based on AWS micro services |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108011967A true CN108011967A (en) | 2018-05-08 |
Family
ID=62058999
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711345873.6A Pending CN108011967A (en) | 2017-12-15 | 2017-12-15 | Picture upload method based on AWS micro services |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108011967A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108989462A (en) * | 2018-08-24 | 2018-12-11 | 四川长虹电器股份有限公司 | Picture method for down loading based on AWS cloud service |
| CN109783447A (en) * | 2019-01-23 | 2019-05-21 | 北京邮电大学 | Electronics license deposits card method, acquisition methods, device, system and electronic equipment |
| CN111367865A (en) * | 2020-03-10 | 2020-07-03 | 山东汇贸电子口岸有限公司 | AWS 3-based file fragment uploading implementation method |
| CN112256351A (en) * | 2020-10-26 | 2021-01-22 | 卫宁健康科技集团股份有限公司 | Implementation method of Feign component, micro-service calling method and device |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102103518A (en) * | 2011-02-23 | 2011-06-22 | 运软网络科技(上海)有限公司 | System for managing resources in virtual environment and implementation method thereof |
| CN103177329A (en) * | 2011-12-22 | 2013-06-26 | Sap股份公司 | Rule-based determination and validation in business object processing |
| CN103561057A (en) * | 2013-10-15 | 2014-02-05 | 深圳清华大学研究院 | Data storage method based on distributed hash table and erasure codes |
| US20140047413A1 (en) * | 2012-08-09 | 2014-02-13 | Modit, Inc. | Developing, Modifying, and Using Applications |
| CN105302920A (en) * | 2015-11-23 | 2016-02-03 | 上海爱数信息技术股份有限公司 | Optimal management method and system for cloud storage data |
-
2017
- 2017-12-15 CN CN201711345873.6A patent/CN108011967A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102103518A (en) * | 2011-02-23 | 2011-06-22 | 运软网络科技(上海)有限公司 | System for managing resources in virtual environment and implementation method thereof |
| CN103177329A (en) * | 2011-12-22 | 2013-06-26 | Sap股份公司 | Rule-based determination and validation in business object processing |
| US20140047413A1 (en) * | 2012-08-09 | 2014-02-13 | Modit, Inc. | Developing, Modifying, and Using Applications |
| CN103561057A (en) * | 2013-10-15 | 2014-02-05 | 深圳清华大学研究院 | Data storage method based on distributed hash table and erasure codes |
| CN105302920A (en) * | 2015-11-23 | 2016-02-03 | 上海爱数信息技术股份有限公司 | Optimal management method and system for cloud storage data |
Non-Patent Citations (1)
| Title |
|---|
| 亚马逊AWS: ""利用Amazon DynamoDB助力游戏应用"", 《HTTPS://BLOG.CSDN.NET/AWSCHINA/ARTICLE/DETAILS/38562221》 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108989462A (en) * | 2018-08-24 | 2018-12-11 | 四川长虹电器股份有限公司 | Picture method for down loading based on AWS cloud service |
| CN109783447A (en) * | 2019-01-23 | 2019-05-21 | 北京邮电大学 | Electronics license deposits card method, acquisition methods, device, system and electronic equipment |
| CN111367865A (en) * | 2020-03-10 | 2020-07-03 | 山东汇贸电子口岸有限公司 | AWS 3-based file fragment uploading implementation method |
| CN112256351A (en) * | 2020-10-26 | 2021-01-22 | 卫宁健康科技集团股份有限公司 | Implementation method of Feign component, micro-service calling method and device |
| CN112256351B (en) * | 2020-10-26 | 2023-11-17 | 卫宁健康科技集团股份有限公司 | Method for realizing Feign component, method and device for calling micro-service |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8341118B2 (en) | Method and system for dynamically replicating data within a distributed storage system | |
| CN108011967A (en) | Picture upload method based on AWS micro services | |
| US10887279B2 (en) | Scalable cloud hosted metadata service | |
| CN103179185B (en) | Method and system for creating files in cache of distributed file system client | |
| CN103731483B (en) | Virtual file system based on cloud computing | |
| CN110268380A (en) | File system hierarchy structure mirror image across cloud data repository | |
| US20070255763A1 (en) | Database replication method and system | |
| US11785232B2 (en) | Media storage | |
| CN104050216B (en) | For customizing the file system manager of resource allocation | |
| CN101442558B (en) | Method and system for providing index service for P2SP network | |
| CN104079983B (en) | System and method for HLS program recording and service | |
| CN106021327B (en) | A kind of Dropbox system for supporting multi-user to read while write and its read document method, written document method | |
| CN101635734A (en) | Method and device for storing and managing downloaded data on non-volatile storage medium | |
| US20140067990A1 (en) | Method for accessing a content item in a cloud storage system, and a corresponding cloud broker, cloud cache agent and client application | |
| CN106648903A (en) | Method and system for calling distributed file system | |
| US20120221514A1 (en) | Method for improving the responsiveness of a client device | |
| EP3044682B1 (en) | Transaction query engine | |
| US9870422B2 (en) | Natural language search | |
| CN101615206A (en) | The system and method for metadata is provided | |
| CN103034669A (en) | System for providing user data storage environment using network-based file system | |
| EP2237170A1 (en) | Data sorage system | |
| US11714858B2 (en) | Methods and systems for providing file data from a file system | |
| CN105677579A (en) | Data access method and system in cache system | |
| US20090327306A1 (en) | Arrangement for anonymous api downloaded resources for advanced content | |
| EP2237144A1 (en) | Method of remotely storing data and related data storage system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180508 |