CN104866485A - Method for downloading mass data on data platform - Google Patents
Method for downloading mass data on data platform Download PDFInfo
- Publication number
- CN104866485A CN104866485A CN201410060855.3A CN201410060855A CN104866485A CN 104866485 A CN104866485 A CN 104866485A CN 201410060855 A CN201410060855 A CN 201410060855A CN 104866485 A CN104866485 A CN 104866485A
- Authority
- CN
- China
- Prior art keywords
- data
- system server
- txt file
- output stream
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method for downloading mass data on a data platform, which is suitable for downloading the mass data. The method comprises the following steps of a data query step that a system server queries a number of data items which conform to the conditions in a database; a TXT file computing step that the system server pages the data items and determines a file format of TXT files according to the total number of the data items and the number of the TXT files corresponding to each page; a TXT file generation step that the TXT files are generated on a local WEB application server according to the file format of TXT files, the system server writes the data items into the TXT files and releases the memory resources of the system server after writing; an output flow writing step that an output stream is formed in an internal memory of the system server, the output stream is data flow, the data items in the TXT files are circularly read from the local WEB application server and the output stream is written; and a data export step that the data items are exported from the output stream, and the internal memory resources of the system server are released after exporting.
Description
Technical field
The present invention relates to network data transmission technology, particularly relate to the big data quantity method for down loading on a kind of data platform.
Background technology
For large-scale enterprise or the network platform, usually have a large amount of data statistical tables and reports to need to transmit, especially apply the enterprise of the data platforms such as such as ERP, user usually needs a large amount of data sheets to download in local to carry out subsequent treatment.
The transmission mode of current employing is that user initiates request from the client of oneself, system server deposits middle process request within it, according to requesting query database, be buffered in the internal memory of system server by inquiring about the data item that obtains, wait to be checked all complete after from the internal memory of system server, send the client of user again to.In the process, all work all completes in the internal memory of system server, when data volume is very large, needs to create a large amount of object and carry out these data item of buffer memory in internal memory.At this moment free memory can be caused to reduce, other work of system server are affected, cause when data volume is excessive internal memory to overflow in addition and cause system crash.A solution configures larger internal memory, but this will cause the rising of cost, and, along with the continuous increase of data volume, only rely on the mode increasing memory size can not solve the problem of big data quantity download.
Summary of the invention
The present invention is intended to propose a kind of method being applicable to carry out big data quantity download on data platform.
According to one embodiment of the invention, propose the big data quantity method for down loading on a kind of data platform, for big data quantity is downloaded to local WEB application server by database, comprise following step:
Data query step, system server inquires about the number of qualified data item in a database according to the querying condition that user inputs.
TXT file calculation procedure, system server is by qualified data item paging, and system server is according to the total number of qualified data item, and the TXT file number that often one page is corresponding determines the file layout of TXT file.
TXT file generated step, the file layout according to TXT file generates TXT file on the WEB application server of this locality, and data item writes in TXT file by system server, has write the memory source of rear delivery system server.
Output stream write step, forms output stream in the internal memory of system server, and output stream is data stream, reads the data item TXT file from the WEB application server cocycle of this locality and writes output stream.
Statistical conversion step, derived data item from output stream, has derived the memory source of rear delivery system server.
In one embodiment, in data query step, the querying condition of user's input is the SQL statement containing No. ID, and system server inquires about the number of qualified data item in a database according in SQL statement No. ID, if do not meet the data item of querying condition, then output abnormality report.
In one embodiment, in TXT file generated step, system server performs successively:
File layout according to TXT file generates TXT file on the WEB application server of this locality;
Complete Path file being write TXT file is set;
File opening write stream in the internal memory of system server, according to Complete Path by described data item write TXT file;
Write rear closedown described file write stream, the memory source of delivery system server.
In one embodiment, data item is write in the process of TXT file, each data item in TXT file:
The form of output is set according to the data type of wherein each field;
Before character string numeral, increase "=" number and increase double quotation marks.
In one embodiment, the form arranging output according to the data type of wherein each field comprises:
Be numeric type for data type, the form of output is directly export;
Be date type for data type, the form of output is form fixed dates;
For other data types, the form of output is character string type.
In one embodiment, in output stream write step, system server turn-on data stream in internal memory, as output stream, writes gauge outfit in output stream, and reads the data item TXT file from the WEB application server cocycle of this locality and write output stream.
In one embodiment, in statistical conversion step, system server refreshes output stream derives all buffering fields in output stream with derived data item from output stream, has derived rear closedown output stream and derivation program with the memory source of delivery system server.
Big data quantity method for down loading on data platform of the present invention decreases the taking of internal memory for system server, data item in processing procedure to be landed to the WEB application server of this locality in the mode of data stream and preserve with the form of TXT file, from TXT file, read data item again after being disposed and be supplied to user, all the memory source of system server is discharged after every one-phase completes, effectively improve the internal memory service efficiency of system server, improve the stability of system cloud gray model.
Accompanying drawing explanation
Below in conjunction with accompanying drawing, specific description is in detail done further to the specific embodiment of the present invention.
Fig. 1 discloses the process flow diagram according to the big data quantity method for down loading on the data platform of one embodiment of the invention.
Fig. 2 discloses the process schematic according to the big data quantity method for down loading on a kind of data platform of one embodiment of the invention.
Embodiment
Shown in figure 1, present invention is disclosed the big data quantity method for down loading on a kind of data platform, for big data quantity is downloaded to local WEB application server by database, the method comprises following step:
102. data query steps, system server inquires about the number of qualified data item in a database according to the querying condition that user inputs.In one embodiment, in data query step 102, the querying condition of user's input is the SQL statement containing No. ID, system server inquires about the number of qualified data item in a database according in SQL statement No. ID, if do not meet the data item of querying condition, then output abnormality report.
104.TXT file calculation procedure, system server is by described qualified data item paging, and system server is according to the total number of qualified data item, and the TXT file number that often one page is corresponding determines the file layout of TXT file.
106.TXT file generated step, the file layout according to TXT file generates TXT file on the WEB application server of this locality, and system server, by described data item write TXT file, has write the memory source of rear delivery system server.In one embodiment, in TXT file generated step 106, system server performs following process successively:
A) on the WEB application server of this locality, TXT file is generated according to the file layout of TXT file.
B) Complete Path file being write TXT file is set.
C) file opening write stream in the internal memory of system server, writes data item in TXT file according to Complete Path.In one embodiment, data item is being write in the process of TXT file, each data item in TXT file: the form that first output is set according to the data type of wherein each field, then before character string numeral, increase "=" number and increase double quotation marks."=" number is increased and the object increasing double quotation marks solves not show 0 and the numeral above situation more than 11 science countings with the data item of 0 beginning before character string numeral.The form of output is set according to the data type of each field, for different data types, processes as follows:
Be numeric type for data type, the form of output is directly export;
Be date type for data type, the form of output is form fixed dates;
For other data types, the form of output is character string type.
D) rear close file write stream has been write, the memory source of delivery system server.
The example code of step 106 is as follows:
108. output stream write steps, form output stream in the internal memory of system server, and output stream is data stream, read the data item TXT file from the WEB application server cocycle of this locality and write output stream.In one embodiment, in output stream write step 108, system server turn-on data stream in internal memory, as output stream, writes gauge outfit in output stream, and reads the data item TXT file from the WEB application server cocycle of this locality and write output stream.
110. statistical conversion steps, derived data item from output stream, has derived the memory source of rear delivery system server.In one embodiment, in statistical conversion step, system server refreshes output stream derives all buffering fields in output stream with derived data item from output stream, has derived rear closedown output stream and derivation program with the memory source of delivery system server.In one embodiment, the data item derived from output stream is saved the form into csv file.
The code sample of the method is as follows:
Fig. 2 discloses the process schematic according to the big data quantity method for down loading on a kind of data platform of one embodiment of the invention.Disclose in the implementation of the method, at user side, data exchange process between system server and the WEB application server of this locality, mainly comprise:
User side sends inquiry request to system server.
The number of system server data query item in a database, if number is empty, then sends exception reporting.If number is not empty, then system server determines the file layout of TXT file according to the total number of data item and TXT file number corresponding to every one page.Local WEB application server sets up TXT file, and data item is written in TXT file in the mode of file write stream by system server, the memory source of delivery system server after complete.
After completing the write of whole TXT file, system server reads data item in the mode of output stream from TXT file, at derived data item from output stream to user side.In one embodiment, the file exporting to user side is csv file form.
Big data quantity method for down loading on data platform of the present invention decreases the taking of internal memory for system server, data item in processing procedure to be landed to the WEB application server of this locality in the mode of data stream and preserve with the form of TXT file, from TXT file, read data item again after being disposed and be supplied to user, all the memory source of system server is discharged after every one-phase completes, effectively improve the internal memory service efficiency of system server, improve the stability of system cloud gray model.
Claims (7)
1. the big data quantity method for down loading on data platform, for big data quantity is downloaded to local WEB application server by database, is characterized in that, comprising:
Data query step, system server inquires about the number of qualified data item in a database according to the querying condition that user inputs;
TXT file calculation procedure, system server is by described qualified data item paging, and system server is according to the total number of described qualified data item, and the TXT file number that often one page is corresponding determines the file layout of TXT file;
TXT file generated step, the file layout according to described TXT file generates TXT file on the WEB application server of this locality, and system server, by described data item write TXT file, has write the memory source of rear delivery system server;
Output stream write step, forms output stream in the internal memory of system server, and described output stream is data stream, reads the data item TXT file from the WEB application server cocycle of this locality and writes output stream;
Statistical conversion step, derived data item from described output stream, has derived the memory source of rear delivery system server.
2. the big data quantity method for down loading on data platform as claimed in claim 1, it is characterized in that, in described data query step, the querying condition of user's input is the SQL statement containing No. ID, system server inquires about the number of qualified data item in a database according in SQL statement No. ID, if do not meet the data item of querying condition, then output abnormality report.
3. the big data quantity method for down loading on data platform as claimed in claim 1, it is characterized in that, in described TXT file generated step, system server performs successively:
File layout according to described TXT file generates TXT file on the WEB application server of this locality;
Complete Path file being write TXT file is set;
File opening write stream in the internal memory of system server, according to described Complete Path by described data item write TXT file;
Write rear closedown described file write stream, the memory source of delivery system server.
4. the big data quantity method for down loading on data platform as claimed in claim 3, is characterized in that, data item write in the process of TXT file, each data item in TXT file:
The form of output is set according to the data type of wherein each field;
Before character string numeral, increase "=" number and increase double quotation marks.
5. the big data quantity method for down loading on data platform as claimed in claim 4, is characterized in that, the form arranging output according to the data type of wherein each field comprises:
Be numeric type for data type, the form of output is directly export;
Be date type for data type, the form of output is form fixed dates;
For other data types, the form of output is character string type.
6. the big data quantity method for down loading on data platform as claimed in claim 1, it is characterized in that, in described output stream write step, system server in internal memory turn-on data stream as output stream, in output stream, write gauge outfit, and read the data item TXT file from the WEB application server cocycle of this locality and write output stream.
7. the big data quantity method for down loading on data platform as claimed in claim 6, it is characterized in that, in described statistical conversion step, system server refreshes output stream derives all buffering fields in output stream with derived data item from output stream, has derived rear closedown output stream and derivation program with the memory source of delivery system server.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410060855.3A CN104866485A (en) | 2014-02-24 | 2014-02-24 | Method for downloading mass data on data platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410060855.3A CN104866485A (en) | 2014-02-24 | 2014-02-24 | Method for downloading mass data on data platform |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104866485A true CN104866485A (en) | 2015-08-26 |
Family
ID=53912324
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410060855.3A Pending CN104866485A (en) | 2014-02-24 | 2014-02-24 | Method for downloading mass data on data platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104866485A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107145551A (en) * | 2017-04-27 | 2017-09-08 | 努比亚技术有限公司 | A kind of server and data export method |
| CN110874345A (en) * | 2018-08-29 | 2020-03-10 | 阿里巴巴集团控股有限公司 | Data processing method, device and system in distributed storage system |
-
2014
- 2014-02-24 CN CN201410060855.3A patent/CN104866485A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107145551A (en) * | 2017-04-27 | 2017-09-08 | 努比亚技术有限公司 | A kind of server and data export method |
| CN107145551B (en) * | 2017-04-27 | 2021-01-01 | 努比亚技术有限公司 | Server and data exporting method |
| CN110874345A (en) * | 2018-08-29 | 2020-03-10 | 阿里巴巴集团控股有限公司 | Data processing method, device and system in distributed storage system |
| CN110874345B (en) * | 2018-08-29 | 2023-04-11 | 阿里巴巴集团控股有限公司 | Data processing method, device and system in distributed storage system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11347761B1 (en) | System and methods for distributed database query engines | |
| US11216302B2 (en) | Modifying task dependencies at worker nodes using precompiled libraries | |
| US9817877B2 (en) | Optimizing data processing using dynamic schemas | |
| US8738649B2 (en) | Distributed processing of streaming data records | |
| CN104123374A (en) | Method and device for aggregate query in distributed databases | |
| CN106951552A (en) | A kind of user behavior data processing method based on Hadoop | |
| CN103106249A (en) | Data parallel processing system based on Cassandra | |
| CN103034658A (en) | Method and system for querying data and exporting report | |
| CN104239572A (en) | System and method for achieving metadata analysis based on distributed cache | |
| US11061964B2 (en) | Techniques for processing relational data with a user-defined function (UDF) | |
| CN103761102B (en) | A kind of uniform data service platform and its implementation | |
| CN105183901A (en) | Method and device for reading database table through data query engine | |
| CN104317899A (en) | Big-data analyzing and processing system and access method | |
| CN104239470A (en) | Distributed environment-oriented space data compound processing system and method | |
| CN104866485A (en) | Method for downloading mass data on data platform | |
| US11620284B2 (en) | Backend data aggregation system and method | |
| CN104077355B (en) | A kind of method, apparatus and system of unstructured data storage and inquiry | |
| CN107896246A (en) | Message dissemination system based on XBRL | |
| JP2016071837A (en) | Data virtualization device and large scale data processing program | |
| CN109151074A (en) | Centralized buffer queue service architecture method and network structure based on Redis | |
| WO2022178931A1 (en) | Implementation method, apparatus and device for querying dynamic columns | |
| CN105243063A (en) | Information recommendation method and device | |
| CN114741414A (en) | Crowd circling method and device, electronic equipment and storage medium | |
| CN116662458A (en) | Real-time tag processing method and device, electronic equipment and storage medium | |
| CN107193874A (en) | A kind of data query method based on finger URL Yu Boolean query condition |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20150826 |
|
| WD01 | Invention patent application deemed withdrawn after publication |