CN111427970A - Data preprocessing and downloading method, system, computer equipment and readable storage medium - Google Patents
Data preprocessing and downloading method, system, computer equipment and readable storage medium Download PDFInfo
- Publication number
- CN111427970A CN111427970A CN202010200782.9A CN202010200782A CN111427970A CN 111427970 A CN111427970 A CN 111427970A CN 202010200782 A CN202010200782 A CN 202010200782A CN 111427970 A CN111427970 A CN 111427970A
- Authority
- CN
- China
- Prior art keywords
- database
- server
- data
- client
- downloading
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 40
- 238000007781 pre-processing Methods 0.000 title claims abstract description 24
- 238000003860 storage Methods 0.000 title claims description 7
- 230000006835 compression Effects 0.000 claims description 11
- 238000007906 compression Methods 0.000 claims description 11
- 238000004590 computer program Methods 0.000 claims description 9
- 230000006837 decompression Effects 0.000 claims description 3
- 230000008569 process Effects 0.000 abstract description 6
- 230000009286 beneficial effect Effects 0.000 description 3
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/219—Managing data history or versioning
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data preprocessing and downloading method, which relates to the technical field of internet and comprises the following steps: the server loads data required by each client and generates a corresponding sqlite database; compressing each generated sqlite database by the server; uploading the compressed database file to the CDN, and deleting the local sqlite database stored in the server; and the client requests and obtains the compressed database file, downloads and decompresses the compressed database file. The method provided by the invention reduces the consumption of the client terminal to download the network, reduces the downloading time, improves the user experience, and simultaneously relieves the pressure of the server to process data in real time.
Description
[ technical field ] A method for producing a semiconductor device
The invention relates to the technical field of internet, in particular to a data preprocessing and downloading method, a data preprocessing and downloading system, computer equipment and a readable storage medium.
[ background of the invention ]
In the prior art, in order to improve the use experience, a client includes an apple mobile phone/tablet, an android mobile phone/tablet, and a windows application, in some scenarios, for example, when network data does not need to be real-time, a request of a network is removed to accelerate the display of a page. The method generally needs to circularly request an interface in advance, performs format conversion on data, finally stores the data in a local database, and opens a page only by reading the content in the database after the data pulling interface is finished. However, the methods adopted in the prior art have the following drawbacks:
when the data volume is very large, the waiting time is very long, which not only wastes the network flow but also reduces the user experience; depending on the performance of the server, server performance may be degraded during peak traffic periods, at which time the pulled interface may be slower.
[ summary of the invention ]
In order to solve the foregoing problems, the present invention provides a data preprocessing and downloading method, which reduces consumption of a client downloading network, reduces downloading time, improves user experience, and relieves pressure of a server on processing data in real time.
In order to achieve the purpose, the invention adopts the following technical scheme:
a data preprocessing and downloading method comprises the following steps:
the server loads data required by each client and generates a corresponding database based on the data required by different clients;
the server respectively compresses the generated databases to form standby files corresponding to the databases;
uploading all the files to be used to the CDN;
the client requests an interface of the server when data is required so as to obtain a corresponding CDN download address of the file to be used;
and the client downloads the file to be used and decompresses the file to be used.
Optionally, after the server generates the database, the WA L mode of the database is opened for writing.
Optionally, the server follows the generated database to write the required data increment into the generated database, according to the data required by the client who has generated the database.
Optionally, after the server executes checkpoint on each database, 7z compression processing is performed on each generated database.
Optionally, after all the standby files are uploaded to the CDN, the database stored locally in the server is deleted.
Optionally, the standby file includes a version number of the corresponding database, the client requests the interface of the server with the version number of the corresponding database, and the server searches for the corresponding standby file according to the version number of the database and provides the CDN download link of the corresponding standby file to the client.
Optionally, the database is a sqlite database.
The method provided by the invention has the following beneficial effects:
1. saving the flow of the client: when the method provided by the invention is adopted to download data, the flow required by the user to use the App is obviously reduced compared with the flow required by the user to use the App when the data is downloaded by adopting the prior art;
2. pressure on the server is saved: by adopting the method provided by the invention, the data is prepared in advance and cached in the CDN, so that the data is hardly acquired through a server.
3. Reducing the damage rate of the database: in the prior art, when a client generates a database, the database damage rate is high due to power failure or interruption of an App process, and by adopting the method provided by the invention, the client downloads the sqlite database generated by the server, so the database damage rate is also improved.
In addition, the invention also provides a data preprocessing and downloading system, which comprises a server subsystem and a client subsystem,
the server subsystem includes:
the loading module is used for loading data required by each client;
the database generation module is used for generating corresponding databases based on data required by different clients;
the compression module is used for respectively compressing the generated databases to form standby files corresponding to the databases;
the uploading module is used for uploading all the standby files to the CDN;
an interface for receiving a request from a client
The client subsystem includes:
the request module is used for requesting an interface of the server when data is required so as to obtain a CDN download address of a corresponding standby file;
the downloading module is used for downloading the standby file;
and the decompression module is used for decompressing the file to be used.
Optionally, after the database is generated, the WA L mode of the database is opened for writing.
Optionally, for data required by the client that has generated the database, the database generation module follows the generated database and incrementally writes the required data into the generated database.
Optionally, after the compression module executes checkpoint on each database, 7z compression processing is performed on each generated database.
Optionally, the upload module deletes the local database of the server after uploading all the standby files to the CDN.
Optionally, the standby file includes a version number of the corresponding database, the request module attaches the version number of the corresponding database when requesting the interface of the server, and the server searches for the corresponding standby file according to the version number of the database and provides the CDN download link of the corresponding standby file to the client.
Optionally, the database generated by the database generation module is an sql ite database.
The beneficial effects of the data preprocessing and downloading system provided by the invention are similar to the beneficial effect reasoning process of the data preprocessing and downloading method, and are not repeated herein.
Meanwhile, the invention also provides computer equipment which comprises a memory and a processor, wherein the memory stores a computer program, and the processor realizes the method of any one of the above items when executing the computer program.
Meanwhile, the invention also provides a computer readable storage medium, on which a computer program is stored, which when executed by a processor implements the method of any one of the above.
These features and advantages of the present invention will be disclosed in more detail in the following detailed description and the accompanying drawings. The best mode or means of the present invention will be described in detail with reference to the accompanying drawings, but the present invention is not limited thereto. In addition, the features, elements and components appearing in each of the following and in the drawings are plural and different symbols or numerals are labeled for convenience of representation, but all represent components of the same or similar construction or function.
[ description of the drawings ]
The invention will be further described with reference to the accompanying drawings in which:
fig. 1 is a flowchart of a first embodiment of the present invention.
[ detailed description ] embodiments
The technical solutions of the embodiments of the present invention are explained and illustrated below with reference to the drawings of the embodiments of the present invention, but the following embodiments are only preferred embodiments of the present invention, and not all embodiments. Based on the embodiments in the implementation, other embodiments obtained by those skilled in the art without any creative effort belong to the protection scope of the present invention.
Reference in the specification to "one embodiment" or "an example" means that a particular feature, structure or characteristic described in connection with the embodiment itself may be included in at least one embodiment of the patent disclosure. The appearances of the phrase "in one embodiment" in various places in the specification are not necessarily all referring to the same embodiment.
The first embodiment is as follows:
as shown in fig. 1, this embodiment provides a data preprocessing and downloading method, where a sqlite database that a client needs to use is generated in advance in a server, some unnecessary data is deleted, and the sqlite database is compressed and then finally uploaded to a CDN, i.e., a Content Delivery Network Content distribution Network. And the client acquires the CDN address by accessing the server interface to perform downloading operation. And decompressing the downloaded data to obtain the locally required database. The method specifically comprises the following steps:
the server loads data required by each client, generates a corresponding sqlite database based on the data required by different clients, starts the WA L mode of the sqlite database, writes the data so as to enable the data to be written more quickly, and then processes a transaction when writing, namely, accesses and possibly updates one program execution unit of various data items in the database;
because WA L mechanism causes the sqlite database file to be very large, after the server executes checkpoint on each database, the server respectively performs 7z compression processing on each generated sqlite database to reduce the size of the file and form a standby file corresponding to the sqlite database;
uploading all standby files to the CDN, and deleting the sqlite database stored locally in the server. Because the data is prepared in advance and cached in the CDN, little need exists to go through the server to obtain the data;
the method comprises the steps that a client requests an interface of a server when data is required, a corresponding version number of a sqlite database is attached, the server searches a corresponding standby file according to the version number of the sqlite database, and a CDN download link of the corresponding standby file is provided for the client to obtain;
and the client downloads the file to be used and decompresses the file to be used. In the prior art, when a client generates a database, the database damage rate is high due to power failure or interruption of an App process, and by adopting the method provided by the embodiment, the client downloads the sql ite database generated by the server, so that the database damage rate is also improved.
Example two
The present embodiment provides a data preprocessing and downloading system for implementing the data preprocessing and downloading method of the first embodiment, which includes a server subsystem and a client subsystem,
the server subsystem includes:
the loading module is used for loading data required by each client;
the database generation module is used for generating a corresponding sqlite database based on data required by different clients, starting a WA L mode of the sqlite database and writing the sqlite database, and for the data required by the clients which have generated the database, the database generation module continues to use the generated database and writes required data increment into the generated database;
the compression module is used for performing 7z compression processing on each generated database after checkpoint is performed on each database to form a standby file corresponding to the database;
and the uploading module is used for uploading all the standby files to the CDN. After uploading is completed, deleting a database stored in the local server;
an interface to receive a request from a client;
the client subsystem includes:
the request module is used for requesting an interface of the server when data is required so as to obtain a CDN download address of a corresponding standby file, the standby file comprises a version number of a corresponding database, the request module is attached with the version number of the corresponding database when the interface of the server is requested, the server searches the corresponding standby file according to the version number of the database, and provides a CDN download link of the corresponding standby file for the client;
the downloading module is used for downloading the standby file;
and the decompression module is used for decompressing the file to be used.
EXAMPLE III
The present embodiments provide a computer apparatus comprising a memory and a processor, the memory storing a computer program that when executed implements the methods of any of the embodiments described above, wherein all or a portion of the processes implementing the methods of any of the embodiments described above are performed by the computer program instructing associated hardware, whereby the computer program may be stored on a non-volatile computer readable storage medium, which when executed implements the methods of any of the embodiments described above, wherein any reference to memory, storage, database or other medium used in the embodiments provided herein may comprise non-volatile and/or volatile memory, and wherein the non-volatile memory may comprise read-only memory (ROM), programmable ROM (prom), electrically programmable ROM (eprom), electrically erasable programmable ROM (eeprom), or flash memory.
While the present invention has been described with reference to the preferred embodiments, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. Any modification which does not depart from the functional and structural principles of the present invention is intended to be included within the scope of the claims.
Claims (16)
1. A data preprocessing and downloading method is characterized by comprising the following steps:
the server loads data required by each client and generates a corresponding database based on the data required by different clients;
the server respectively compresses the generated databases to form standby files corresponding to the databases;
uploading all the files to be used to the CDN;
the client requests an interface of the server when data is required so as to obtain a corresponding CDN download address of the file to be used;
and the client downloads the file to be used and decompresses the file to be used.
2. The method as claimed in claim 1, wherein the WA L mode of the database is opened for writing after the server generates the database.
3. The data preprocessing and downloading method of claim 1, wherein the client that has generated the database needs data, and the server follows the generated database to write the needed data increment into the generated database.
4. The data preprocessing and downloading method of claim 1, wherein the server performs 7z compression on each generated database after performing checkpoint on each database.
5. The data pre-processing and downloading method of claim 1 wherein the database locally stored on the server is deleted after all inactive files are uploaded to the CDN.
6. The data preprocessing and downloading method of claim 1, wherein the standby file includes a version number of a corresponding database, the client requests an interface of the server with the version number of the corresponding database, the server searches for the corresponding standby file according to the version number of the database, and provides the corresponding CDN download link of the standby file to the client.
7. The data pre-processing and downloading method of one of claims 1 to 6, wherein the database is a sqlite database.
8. A data preprocessing and downloading system is characterized in that the data preprocessing and downloading system comprises a server subsystem and a client subsystem,
the server subsystem includes:
the loading module is used for loading data required by each client;
the database generation module is used for generating corresponding databases based on data required by different clients;
the compression module is used for respectively compressing the generated databases to form standby files corresponding to the databases;
the uploading module is used for uploading all the standby files to the CDN;
an interface to receive a request from a client;
the client subsystem includes:
the request module is used for requesting an interface of the server when data is required so as to obtain a CDN download address of a corresponding standby file;
the downloading module is used for downloading the standby file;
and the decompression module is used for decompressing the file to be used.
9. The system of claim 8, wherein the database generation module is configured to initiate a WA L mode for writing data into the database after the database generation module generates the database.
10. The data preprocessing and downloading system of claim 8 wherein for data needed by a client that has generated the database, the database generation module follows the generated database and writes the needed data increments into the generated database.
11. The system of claim 8, wherein the compression module performs 7z compression on each generated database after performing checkpoint on each database.
12. The system of claim 8, wherein the upload module deletes the local database of the server after uploading all the files to be used to the CDN.
13. The data preprocessing and downloading system of claim 8 wherein the standby file includes a version number of the corresponding database, the request module attaches the version number of the corresponding database when requesting the interface of the server, the server finds the corresponding standby file according to the version number of the database, and provides the corresponding CDN download link of the standby file to the client.
14. The data preprocessing and downloading method as claimed in any one of claims 8 to 13, wherein the database generated by the database generation module is a sqlite database.
15. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the method of any one of claims 1 to 8 when executing the computer program.
16. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the method of any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010200782.9A CN111427970A (en) | 2020-03-20 | 2020-03-20 | Data preprocessing and downloading method, system, computer equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010200782.9A CN111427970A (en) | 2020-03-20 | 2020-03-20 | Data preprocessing and downloading method, system, computer equipment and readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111427970A true CN111427970A (en) | 2020-07-17 |
Family
ID=71548378
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010200782.9A Pending CN111427970A (en) | 2020-03-20 | 2020-03-20 | Data preprocessing and downloading method, system, computer equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111427970A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090276447A1 (en) * | 2008-05-05 | 2009-11-05 | Microsoft Corporation | Formats for database template files shared between client and server environments |
| CN103370704A (en) * | 2010-12-21 | 2013-10-23 | 赛贝斯股份有限公司 | Bulk initial download of mobile databases |
| US20150058403A1 (en) * | 2013-08-20 | 2015-02-26 | Alibaba Group Holding Limited | Method and system of dispatching requests in a content delivery network |
| CN108337327A (en) * | 2018-04-26 | 2018-07-27 | 拉扎斯网络科技（上海）有限公司 | A kind of resource acquiring method and proxy server |
| CN109982111A (en) * | 2017-12-28 | 2019-07-05 | 贵州白山云科技股份有限公司 | Content of text transmission optimization method and content of text transmission optimization method, apparatus based on Living Network system |
| CN110716825A (en) * | 2018-07-12 | 2020-01-21 | 阿里巴巴集团控股有限公司 | Data disaster tolerance processing method, device and system |
-
2020
- 2020-03-20 CN CN202010200782.9A patent/CN111427970A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090276447A1 (en) * | 2008-05-05 | 2009-11-05 | Microsoft Corporation | Formats for database template files shared between client and server environments |
| CN103370704A (en) * | 2010-12-21 | 2013-10-23 | 赛贝斯股份有限公司 | Bulk initial download of mobile databases |
| US20150058403A1 (en) * | 2013-08-20 | 2015-02-26 | Alibaba Group Holding Limited | Method and system of dispatching requests in a content delivery network |
| CN109982111A (en) * | 2017-12-28 | 2019-07-05 | 贵州白山云科技股份有限公司 | Content of text transmission optimization method and content of text transmission optimization method, apparatus based on Living Network system |
| CN108337327A (en) * | 2018-04-26 | 2018-07-27 | 拉扎斯网络科技（上海）有限公司 | A kind of resource acquiring method and proxy server |
| CN110716825A (en) * | 2018-07-12 | 2020-01-21 | 阿里巴巴集团控股有限公司 | Data disaster tolerance processing method, device and system |
Non-Patent Citations (1)
| Title |
|---|
| 朱尚明等: "基于CDN构建Web应用云计算平台", 《武汉大学学报(理学版)》 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US7484007B2 (en) | System and method for partial data compression and data transfer | |
| CN107832406B (en) | Method, device, equipment and storage medium for removing duplicate entries of mass log data | |
| CN102841901B (en) | A kind of method and apparatus of web displaying | |
| CN108334367B (en) | Firmware remote upgrading method for BMC chip | |
| CN108804768A (en) | One kind making the light-weighted method of BIM models | |
| CN104090862A (en) | Network disk multi-type file online preview method based on Linux+Ngnix+Mysql+PHP (LNMP) | |
| US10210164B2 (en) | Systems and methods for data archival | |
| US7257649B2 (en) | Method and system for transferring information during server synchronization with a computing device | |
| CN103902309A (en) | Method for remotely upgrading wireless module firmware | |
| CN112583889A (en) | Large file transmission method and device | |
| CN104092777A (en) | Method and system for synchronously storing layout setting of clients in server | |
| CN111857695A (en) | Software project processing method, device, equipment and storage medium | |
| CN111427970A (en) | Data preprocessing and downloading method, system, computer equipment and readable storage medium | |
| CN109062710B (en) | Remote procedure calling method, terminal equipment and network equipment | |
| WO2024125002A1 (en) | Firmware updating method, system and apparatus, electronic device, and storage medium | |
| CN106844600A (en) | A kind of website picture processing system and method | |
| CN113495783A (en) | Service data processing method and device | |
| CN117215804A (en) | Data processing method, device, electronic equipment and storage medium | |
| CN107908634B (en) | Cache control method of browser and mobile terminal | |
| CN110866201A (en) | Cache processing method and device, computer equipment and storage medium | |
| CN111443992A (en) | Docker mirror image difference derivation method, computer storage medium and electronic device | |
| CN106681761B (en) | Interactive upgrading method and system | |
| CN111970161B (en) | Internet of things application differential upgrading method based on address prediction | |
| CN107229655A (en) | The update method of page static file, apparatus and system | |
| CN112491953B (en) | Implementation method, system, equipment and medium supporting continuous transmission of cloud platform mirror image data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200717 |