CN115499425A - Cloud platform file export method - Google Patents
Cloud platform file export method Download PDFInfo
- Publication number
- CN115499425A CN115499425A CN202210882793.9A CN202210882793A CN115499425A CN 115499425 A CN115499425 A CN 115499425A CN 202210882793 A CN202210882793 A CN 202210882793A CN 115499425 A CN115499425 A CN 115499425A
- Authority
- CN
- China
- Prior art keywords
- file
- export
- task
- service
- user
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a cloud platform file export method, which comprises the following steps: s10: the platform user sends out a task exporting request; s20, gateway service judges whether the platform user request is valid or not, and prevents the system from being illegally called; meanwhile, the user request is forwarded to the downstream service in a round-robin manner; and S30, file service, namely saving export task information of a file export request of a user and executing an asynchronous export task in a mode of remotely calling service. And S40, business service, inquiring a business service database through a user export request, writing data needing to be exported into a file, and storing the data into a cloud storage server. The method comprises the following steps: not only the waiting time of the user is reduced, but also the time for the user to download the file is reduced; meanwhile, the use of the memory of the program service is reduced, the program operation is more stable, and the request pressure to the database is reduced to a certain extent.
Description
Technical Field
The invention relates to the technical field of product information management, in particular to a cloud platform file exporting method.
Background
In a business system, a large amount of file data (often hundreds of thousands of millions of file data) needs to be exported by a platform user (such as a high-level and low-level business cloud platform), but according to a traditional export mode (direct export), the export request of the platform user is too long, the internal memory of a server is too large, the pressure of a database is too large, and the adverse effect on the user and the platform service is caused. Moreover, if the amount of exported data is too large, the memory usage of the program is too large, which causes memory leakage of the service, and thus makes the program unusable.
In order to improve the use experience of platform users, reduce the waiting time for downloading files by the platform users and simultaneously ensure the usability of platform services, a technical solution for exporting a large amount of file data is urgently needed.
Disclosure of Invention
The invention mainly aims to provide a cloud platform file export method to solve the defects in the prior art.
In order to realize the purpose, the technical scheme adopted by the invention is as follows: the method for exporting the cloud platform file specifically comprises the following steps:
s10: the platform user sends out a task exporting request;
s20, gateway service judges whether the platform user request is valid or not, and prevents the system from being illegally called; meanwhile, the user request is forwarded to the downstream service in a round-robin manner;
s30, file service, namely saving export task information of a user export file request, and executing an asynchronous export task in a remote service calling mode; when the platform user perceives that the export state of the export task is finished, the exported file content can be downloaded through the corresponding cloud storage server address; when the service type is consistent with the request parameter, which indicates that the file to be exported is consistent with the last time, the export task is not added, so that the execution pressure of the service is reduced;
and S40, business service, inquiring a business service database through a user export request, writing data needing to be exported into a file, and storing the data into a cloud storage server.
Preferably, in S30, the method includes the following steps:
s301, a user adds a file export task through a service type and a request parameter, and if the task already exists, the task is not added;
s302, remotely calling a service by the file service to execute a file export task;
s303, the platform user inquires the task state of the file export task, if the task state is successful, the task state is returned to the cloud storage server address, if the task state is export, the platform user needs to wait for the export completion, and if the task state is fail, the problem of the service is explained;
and S304, the platform user downloads the file according to the cloud storage server address.
Preferably, in S30, the export task information includes a service type of the export task, a task state, a file address of the cloud storage server, and a request parameter.
Preferably, in S40, a streaming query mode is used to acquire data from the database, and package the acquired data and write the packaged data into a file, instead of putting a large amount of data into a memory for processing.
Preferably, in the above steps, the connection needs to be kept not closed in the whole streaming processing process, and in the cloud platform, an inquiry read-only transaction is started for operation, and the connection is closed only when the transaction is submitted after execution is completed.
Preferably, the cloud storage server adopts Tencent cloud cos: the method has the advantages that a large amount of file data generated in the cloud platform is stored, the use of system broadband is reduced, more system flow is enabled to transmit the request data of platform users, and meanwhile, the cost of a server for storing a large amount of files is reduced.
The present invention also provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the cloud platform file export method.
The invention discloses a cloud platform file export method which comprises the following steps:
1. when the task is exported, the waiting time of the user is reduced;
2. when the export file is stored, the time for downloading the file by the user is reduced;
3. when the file is exported, the use of the memory of the program service is reduced, so that the program operation is more stable, and the request pressure on the database is reduced to a certain extent.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the structures shown in the drawings without creative efforts.
Fig. 1 is a flowchart of a cloud platform file export method according to an embodiment of the present invention;
fig. 2 is a schematic diagram of a cloud platform file export method according to an embodiment of the present invention;
fig. 3 is a flowchart of asynchronous export of Excel according to an embodiment of the present invention;
fig. 4 is a diagram of an Excel asynchronous export interface according to an embodiment of the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further described with reference to the accompanying drawings.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that all the directional indicators (such as upper, lower, left, right, front, and rear … …) in the embodiments of the present invention are only used to explain the relative position relationship between the components, the motion situation, etc. in a specific posture (as shown in the drawings), and if the specific posture is changed, the directional indicator is changed accordingly.
In addition, the descriptions related to "first", "second", etc. in the present invention are only used for descriptive purposes and are not to be construed as indicating or implying relative importance or implicit to the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In addition, technical solutions between various embodiments may be combined with each other, but must be realized by a person skilled in the art, and when the technical solutions are contradictory or cannot be realized, such a combination should not be considered to exist, and is not within the protection scope of the present invention.
In the embodiment of the present invention, referring to fig. 1, the method for exporting a cloud platform file includes the following steps:
s10: the platform user sends out a task exporting request;
s20, gateway service judges whether the platform user request is valid or not, and prevents the system from being illegally called; meanwhile, the method also has the function of load balancing, and forwards the user request to the downstream service in a round-robin manner;
s30, file service, namely saving export task information of a user export file request (wherein the export task information mainly comprises a business type, a task state, a cloud storage server file address and a request parameter of the export task), and executing an asynchronous export task in a mode of remotely calling the business service; when the platform user perceives that the export state of the export task is finished, the exported file content can be downloaded through the corresponding cloud storage server address; when the service type is consistent with the request parameter, which indicates that the file needing to be exported is consistent with the last time, an export task is not added, so that the execution pressure of the service is reduced. The method specifically comprises the following steps:
s301, a user adds a file export task through a service type and a request parameter, and if the task already exists, the task is not added;
s302, remotely calling a service by the file service to execute a file export task;
s303, the platform user inquires the task state of the file export task, if the task state is successful, the task state is returned to the cloud storage server address, if the task state is export, the platform user needs to wait for the export completion, and if the task state is fail, the problem of the service is explained;
and S304, the platform user downloads the file according to the cloud storage server address.
Generally, in daily operation, the export can be performed by using ordered pages, but the pages have the following disadvantages:
(1) The number of pages is not well controlled, too small query times are too many, and too large query times can cause the problem that the program occupies too large memory;
(2) Too many query times are carried out on the database, so that too many database scanning times are caused, and the pressure on the database is too large.
In the embodiment, the export task is executed asynchronously, so that the waiting time of the user is reduced. Asynchronous output is one of the two basic modes of operation for any particular output stream (synchronous and asynchronous output). In asynchronous mode, the natural output transactions do not have to be synchronized or interlocked with the output state of a program.
Taking Excel asynchronous export as an example, referring to fig. 3, the steps for realizing the file asynchronous export function are as follows:
1. the front end triggers a download function button, and AJAX sends a request for downloading to a background;
2. after receiving the request, the background puts the data processing flow into a thread pool, and returns success to inform that the request is successful (returns fail if abnormal);
3. if the AJAX return value is successful, continuously requesting a download state to a background (the background needs to provide an interface for acquiring the state);
4. when the download state is success, initiating an export file (a background provides an interface, the file is obtained from a server and the Excel is exported in a file stream form, the export is successful, and the update state is finish);
5. when the download status is finish or other error status, the request for the acquisition status from the background is stopped. The interface is shown in fig. 4.
In this embodiment, the cloud storage server adopts an Tencent cloud cos: the method has the advantages that a large amount of file data generated in the cloud platform is stored, the use of system broadband is reduced, more system flow is used for transmitting request data of platform users, and meanwhile, the cost of servers for storing a large amount of files is reduced.
In order to prevent the situation that the Tencent cloud cos deadlock happens accidentally and the user cannot use the export function all the time, each task Tencent cloud cos object has an expiration time which is set to be 30 minutes. That is, regardless of whether the export task execution is complete, after 30 minutes the task will be abandoned, the user may click the export button again.
S40, business service, namely inquiring a business service database through a user export request, writing data needing to be exported into a file, and storing the data into a cloud storage server;
in this embodiment, a streaming query manner is used to acquire data from a database, package the acquired data, and write the packaged data into a file, instead of putting a large amount of data into a memory for processing.
Furthermore, in the whole streaming processing process, the connection needs to be kept not closed, in the cloud platform, an inquiry read-only transaction is started for operation, and the connection is closed only when the transaction is submitted after the execution is finished.
MyBatis provides an interface class called org, apache, ibatis, cur sor, for streaming queries, which inherits the java.
1. Cursor is closable;
2. cursor is traversable.
In addition, cursor provides three methods:
1. isOpen (): for determining whether the Cursor object is in an open state before fetching data. Only when turned on, cursor can fetch data;
2. isconsummed (): and the method is used for judging whether the query result is completely taken.
3. getcurrentndex (): returning how many pieces of data have been acquired
Example (c):
the invention discloses a cloud platform file export method which comprises the following steps:
1. the asynchronous execution of the export task is adopted, so that the waiting time of a user is reduced;
2. the Tencent cloud cos is adopted to store the exported file, so that the time for a user to download the file is reduced;
3. and by adopting stream-type export, the use of the memory of the program service is reduced, the program operation is more stable, and the request pressure on the database is reduced to a certain extent.
The present embodiment also provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the steps of the cloud platform file export method described above.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications and equivalents of the present invention, which are made by the contents of the present specification and the accompanying drawings, or directly/indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (8)
1. A cloud platform file export method is characterized by comprising the following steps:
s10: the platform user sends out a task exporting request;
s20, gateway service judges whether the platform user request is valid or not, and prevents the system from being illegally called; meanwhile, the user request is forwarded to the downstream service in a round-robin manner;
s30, file service, namely saving export task information of a user export file request, and executing an asynchronous export task in a remote service calling mode;
and S40, business service, inquiring a business service database through a user export request, writing data needing to be exported into a file, and storing the data into a cloud storage server.
2. The cloud platform file export method according to claim 1, wherein in S30, the following steps are included:
s301, a user adds a file export task through a service type and a request parameter, and if the task already exists, the task is not added;
s302, remotely calling a service by the file service to execute a file export task;
s303, the platform user inquires a task state of the file export task, if the task state is successful, the address of the cloud storage server is returned, if the task state is exported, the platform user is required to wait for exporting, and if the task state is failed, the problem occurs in the service;
and S304, the platform user downloads the file according to the cloud storage server address.
3. The cloud platform file export method according to claim 1, wherein in S30, exporting task information includes exporting a service type of a task, a task state, a file address of a cloud storage server, and a request parameter.
4. The method for exporting the cloud platform file according to claim 1, wherein in S30, when the platform user perceives that the export status of the export task is complete, the exported file content may be downloaded through the corresponding address of the cloud storage server; when the service type is consistent with the request parameter, which indicates that the file needing to be exported is consistent with the last time, the export task is not added, and the execution pressure of the service is reduced.
5. The cloud platform file export method according to claim 1, wherein in S40, a streaming query manner is used to obtain data from a database, and the obtained data is packaged and written into a file, instead of putting a large amount of data into a memory for processing.
6. The method for exporting the cloud platform file according to claim 5, wherein the connection is not required to be kept closed in the whole streaming process, and in the cloud platform, the query read-only transaction is started for operation, and the connection is closed only when the transaction is submitted after the execution is completed.
7. The method according to claim 1, wherein the cloud storage server uses a Tencent cloud cos: the method has the advantages that a large amount of file data generated in the cloud platform is stored, the use of system broadband is reduced, more system flow is used for transmitting request data of platform users, and meanwhile, the cost of servers for storing a large amount of files is reduced.
8. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the cloud platform file export method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210882793.9A CN115499425A (en) | 2022-07-25 | 2022-07-25 | Cloud platform file export method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210882793.9A CN115499425A (en) | 2022-07-25 | 2022-07-25 | Cloud platform file export method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115499425A true CN115499425A (en) | 2022-12-20 |
Family
ID=84465982
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210882793.9A Pending CN115499425A (en) | 2022-07-25 | 2022-07-25 | Cloud platform file export method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115499425A (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103257964A (en) * | 2012-02-16 | 2013-08-21 | 环达电脑(上海)有限公司 | Asynchronous exporting method of mass data |
| US20130339490A1 (en) * | 2012-06-19 | 2013-12-19 | Salesforce.Com, Inc. | Method and system for semi-synchronously exporting data |
| CN113656357A (en) * | 2021-09-01 | 2021-11-16 | 京东科技信息技术有限公司 | File management method, device, system and storage medium |
-
2022
- 2022-07-25 CN CN202210882793.9A patent/CN115499425A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103257964A (en) * | 2012-02-16 | 2013-08-21 | 环达电脑(上海)有限公司 | Asynchronous exporting method of mass data |
| US20130339490A1 (en) * | 2012-06-19 | 2013-12-19 | Salesforce.Com, Inc. | Method and system for semi-synchronously exporting data |
| CN113656357A (en) * | 2021-09-01 | 2021-11-16 | 京东科技信息技术有限公司 | File management method, device, system and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10248415B2 (en) | Dynamic code generation and memory management for component object model data constructs | |
| US8156137B2 (en) | Data processing systems and methods | |
| US6769048B2 (en) | Cache synchronization method, system and apparatus for a distributed application and an object located in a client cache | |
| KR100210208B1 (en) | Method and apparatus for enabling persistent attributes in an object oriented environment | |
| US20090300017A1 (en) | Transaction Parallel Control Method, and Database Managemet System | |
| US9967370B2 (en) | OData enabled mobile software applications | |
| JPH0997204A (en) | Method and apparatus for provision of transparent invariancein distributed object operating environment | |
| EP2797014A1 (en) | Database update execution according to power management schemes | |
| US20050203960A1 (en) | Systems and methods that synchronize data with representations of the data | |
| US9158555B2 (en) | Efficient serialization of mutable objects | |
| US20080141260A1 (en) | User mode file system serialization and reliability | |
| US8499298B2 (en) | Multiprocessing transaction recovery manager | |
| US8862613B2 (en) | Extensibility of business process and application logic | |
| US8539490B2 (en) | System and method for providing dynamic transaction optimizations | |
| CN104423982A (en) | Request processing method and device | |
| US10318520B2 (en) | System and method for reducing communications overhead in a distributed transactions environment by modifying implementation of the transaction end function | |
| CN112597486A (en) | Method for preventing repeated access to Restful API based on Spring | |
| WO2021237190A1 (en) | Distributed transaction execution in distributed databases | |
| CN115499425A (en) | Cloud platform file export method | |
| KR101888131B1 (en) | Method for Performing Real-Time Changed Data Publish Service of DDS-DBMS Integration Tool | |
| US9984096B2 (en) | System and method for reducing communications overhead in a distributed transactions environment by modifying implementation of the transaction start function | |
| US7624108B1 (en) | Method and apparatus for formally specifying application-specific read/write consistency | |
| CN115543485B (en) | Data conversion configuration generation method, device, computer equipment and medium | |
| De Porre et al. | Squirrel: an extensible distributed key-value store | |
| US20050289212A1 (en) | Non-transactional resource reference |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |