CN110572422A - Data downloading method and device - Google Patents
Data downloading method and device Download PDFInfo
- Publication number
- CN110572422A CN110572422A CN201810573022.5A CN201810573022A CN110572422A CN 110572422 A CN110572422 A CN 110572422A CN 201810573022 A CN201810573022 A CN 201810573022A CN 110572422 A CN110572422 A CN 110572422A
- Authority
- CN
- China
- Prior art keywords
- file
- downloaded
- compression
- packaging
- compressed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/04—Protocols for data compression, e.g. ROHC
Abstract
The invention discloses a data downloading method and device, and relates to the technical field of computers. One embodiment of the method comprises: receiving a file packaging and downloading request sent by a terminal; compressing and packaging at least one file to be downloaded based on the request to obtain a compressed package; sending the compressed packet to the terminal; wherein the compression parameters include at least one of: and identifying the file to be downloaded and storing the compressed packet. According to the embodiment, at least one file to be downloaded can be compressed and packaged according to the file packaging and downloading request, the obtained compression package is sent to the local part of the user, the requirement of the user for packaging and downloading a plurality of files is met, the downloading time is shortened, the downloading speed of the file to be downloaded is improved, the downloading bandwidth is saved, and the resource consumption is reduced.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data downloading method and apparatus.
Background
Object stores, also called object-based stores, are methods used to describe the resolution and processing of discrete units, referred to as objects. Object storage is a data storage method without a hierarchy, and is generally used in the cloud. Unlike other data storage methods, object-based storage does not use a directory tree. Individual data units (objects) exist at the same level in the storage pool.
In the process of implementing the invention, the inventor finds that at least the following problems exist in the prior art:
Users always expect a convenient and fast information organization method when using object storage products, and therefore, common object storage products are managed in units of objects. However, when a user needs to download a plurality of objects, the objects can only be downloaded in sequence, and the requirement that the user packs and compresses a plurality of objects as one object to download cannot be met, which wastes time and broadband resources.
Disclosure of Invention
in view of this, embodiments of the present invention provide a data downloading method and apparatus, which can compress and package at least one file to be downloaded according to a file packaging and downloading request, and send the obtained compression package to a local user, so as to meet a requirement of the user on packaging and downloading a plurality of files, reduce downloading time, improve downloading speed of the file to be downloaded, and save downloading bandwidth.
To achieve the above object, according to an aspect of an embodiment of the present invention, there is provided a data downloading method including: receiving a file packaging and downloading request sent by a terminal; compressing and packaging at least one file to be downloaded based on the request to obtain a compressed package; sending the compressed packet to the terminal; wherein the file packaging download request comprises compression parameters, the compression parameters comprising at least one of: and identifying the file to be downloaded and storing the compressed packet.
Optionally, the compression parameters further include compression rules;
based on the request, the compressing and packaging at least one file to be downloaded comprises: determining at least one file to be downloaded according to the file identifier to be downloaded; and compressing and packaging at least one file to be downloaded according to the compression rule to obtain a compression package, and storing the compression package in the storage position of the compression package.
Optionally, compression-packaging at least one file to be downloaded to obtain a compression package comprises: when a plurality of files to be downloaded are available, compressing and packaging the plurality of files to be downloaded in parallel to obtain compressed files; storing the obtained compressed file in a buffer area; and uploading the compressed file fragments of the buffer area to the storage position of the compressed packet to obtain the compressed packet.
Optionally, the method further comprises: generating a packaging downloading task according to the request to form a task list; and distributing the plurality of tasks in the task list to a plurality of task processing nodes so as to execute the plurality of tasks in parallel.
Optionally, the at least one file to be downloaded is stored in an object storage manner.
To achieve the above object, according to another aspect of the embodiments of the present invention, there is provided a data downloading apparatus including: the receiving module is used for receiving a file packaging and downloading request sent by a terminal; the compression module is used for compressing and packaging at least one file to be downloaded based on the request to obtain a compression package; a sending module, configured to send the compressed packet to the terminal; wherein the file packaging download request comprises compression parameters, the compression parameters comprising at least one of: and identifying the file to be downloaded and storing the compressed packet.
Optionally, the compression parameters further include compression rules;
the compression module is further to: determining at least one file to be downloaded according to the file identifier to be downloaded; and compressing and packaging at least one file to be downloaded according to the compression rule to obtain a compression package, and storing the compression package in the storage position of the compression package.
Optionally, the compression module is further configured to: when a plurality of files to be downloaded are available, compressing and packaging the plurality of files to be downloaded in parallel to obtain compressed files; storing the obtained compressed file in a buffer area; and uploading the compressed file fragments of the buffer area to the storage position of the compressed packet to obtain the compressed packet.
Optionally, the apparatus further comprises a list generation module configured to: generating a packaging downloading task according to the request to form a task list; and distributing the plurality of tasks in the task list to a plurality of task processing nodes so as to execute the plurality of tasks in parallel.
Optionally, the at least one file to be downloaded is stored in an object storage manner.
To achieve the above object, according to still another aspect of embodiments of the present invention, there is provided an electronic apparatus including: one or more processors; a storage device, configured to store one or more programs, which when executed by the one or more processors, cause the one or more processors to implement the data downloading method according to the embodiment of the present invention.
To achieve the above object, according to still another aspect of the embodiments of the present invention, there is provided a computer-readable medium on which a computer program is stored, the program implementing the data downloading method of the embodiments of the present invention when executed by a processor.
one embodiment of the above invention has the following advantages or benefits: because the technical means of compressing and packaging at least one file to be downloaded according to the received file packaging and downloading request to obtain the compressed package and then sending the compressed package to the terminal is adopted, the technical problem that a plurality of objects cannot be used as one object for downloading in the prior art is solved, the downloading time is further reduced, the downloading speed of the file to be downloaded is improved, and the downloading bandwidth is saved.
further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.
drawings
the drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
fig. 1 is a schematic diagram of a main flow of a data downloading method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of the main modules of a data download device according to an embodiment of the present invention;
FIG. 3 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;
fig. 4 is a schematic block diagram of a computer system suitable for use in implementing a terminal device or server of an embodiment of the invention.
Detailed Description
exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Fig. 1 is a schematic diagram of a main flow of a data downloading method according to an embodiment of the present invention, as shown in fig. 1, the method includes:
step S101: receiving a file packaging and downloading request sent by a terminal; wherein the file packaging download request comprises compression parameters, the compression parameters comprising at least one of: the file to be downloaded is identified and the storage position of the compression packet is stored;
Step S102: compressing and packaging at least one file to be downloaded based on the request to obtain a compressed package;
Step S103: and sending the compressed packet to the terminal.
For step S101, the file identifier to be downloaded is used to distinguish the files to be downloaded, for example, the file identifier to be downloaded may be a URL (Uniform Resource Locator), and the user may specify a plurality of files to be downloaded by specifying a series of URLs. The identifier of the file to be downloaded may also be the name of the file to be downloaded. In other alternative embodiments, as shown in table 1 below, the compression parameters may further include a compression rule (rule), a user account number (principal), a storage location of the file to be downloaded, and a callback notification interface (notifyurl). The compression rules may include, among other things, the compression format, the compression mode (e.g., fastest, faster, standard, or stored), whether to delete source files, and whether to add user identity check information. The compression format refers to a compression method for performing compression on a file through a certain program, for example, tar. The callback notification interface is used to submit the processing result to the address specified by notifyurl after completion of the compression packing.
table 1:
| Parameter name | Meaning of parameters | description of the invention |
| rule | Compression rules | must fill in |
| principle | User account | must fill in |
| source | The file identifier to be downloaded represents a plurality of files needing to be packed and compressed | must fill in |
| destination | Compressed packet storage location | must fill in |
| type | Compressed formats, e.g. tar. gz, zip, tar, default to zip | non-necessary filling |
| notifyurl | after the packing compression is finished, the processing result is submitted to the address specified by notifyurl | Non-necessary filling |
After receiving a file packaging downloading request of a user, checking whether parameters in the request are legal, for example, checking whether a user account exists and whether the user account is overdue; whether the file to be downloaded exists and belongs to the user; whether the storage position of the compressed packet exists or not and has authority to operate or not; whether compression formats are supported. If the parameter combination rule in the file packaging downloading request can put the request into the task queue, an error is returned and illegal parameters are indicated if the parameter verification fails.
the file to be downloaded can be stored in an object storage mode and can be stored in a public cloud. The public cloud generally refers to a cloud which can be used and is provided by a third-party provider for a user, the public cloud can be generally used through a network, and the core attribute of the public cloud is shared with a resource service.
For step S102, compressing and packaging at least one file to be downloaded based on the request to obtain a compressed package may include:
Determining at least one file to be downloaded according to the file identifier to be downloaded;
and compressing and packaging at least one file to be downloaded according to the compression rule to obtain a compression package, and storing the compression package in the compression package storage position.
The file to be downloaded can be audio and video, picture, document or data such as filing file and log. The compression and packaging means that one or more files are compressed by a compression algorithm and pressed into a compressed file package, so that the space can be saved, and the transmission among networks is facilitated.
In an alternative embodiment, the process of compression-packaging at least one file to be downloaded to obtain a compression package comprises:
When a plurality of files to be downloaded are available, compressing and packaging the plurality of files to be downloaded in parallel to obtain compressed files;
storing the obtained compressed file in a buffer area;
And uploading the compressed file fragments of the buffer area to the storage position of the compressed packet to obtain the compressed packet.
specifically, the packaged downloading request is immediately processed after the file packaged downloading request is received, and the files to be downloaded are downloaded in parallel in a thread pool mode inside by utilizing the characteristic of high throughput of object storage. In order to improve the efficiency of task operation, the tasks can be packaged while downloading, a part of packaged files are put into a buffer area, and the compressed files in the buffer area are sent to the storage position of a compressed packet in real time in a fragmentation uploading mode, so that the compressed packet is obtained. The fragment uploading supports that a file is cut into a series of small data fragments with specific sizes, the small data fragments are uploaded to the server respectively, and after all the small data fragments are uploaded, the small data fragments are combined into a file at the server.
Optionally, md5(Message-Digest Algorithm 5, version 5 of the Message Digest Algorithm, to ensure complete and consistent information transfer) of the compressed package and md5 of the compressed package downloaded locally to the user may be calculated to verify the integrity of the compressed package downloaded by the user. When the md5 of the two is the same, it indicates that the compressed package downloaded by the user is complete, otherwise, it indicates that the compressed package downloaded by the user is incomplete.
more specifically, compressing and packaging at least one file to be downloaded based on the request includes: :
Generating a packaging downloading task according to the request to form a task list;
And distributing the plurality of tasks in the task list to a plurality of task processing nodes so as to execute the plurality of tasks in parallel.
In this embodiment, a distributed task scheduling framework may be adopted, and the nodes of the cluster share the large-batch packed download tasks, so as to improve the processing efficiency of the batch tasks. The purpose of the distributed task scheduling framework is to enable the packed download tasks to be distributed to processing programs of a plurality of hosts or different thread groups to be executed in parallel, all the packed download tasks can be rapidly processed without repetition and omission, the task non-repetition execution is realized, the tasks can be automatically retried when errors occur, and the high availability, the availability and the flexibility of the system are ensured.
As a specific example, a distributed task scheduling framework may be implemented by using a zookeeper (zookeeper is an open source file application program interface, and enables distributed processes of a large system to be synchronized with each other, so that all requesting clients can obtain consistent data), when a packed download task is started, the packed download task is registered by creating EPHEMERAL nodes (temporary nodes), a task serialization result is stored in PERSISTENT nodes (permanent nodes), and the packed download task is guaranteed to be processed only once by locking.
In this embodiment, after the package download task is completed, the processing result status is submitted to the website pointed to by notifyurl.
The data downloading method of the embodiment of the invention can compress and pack at least one file to be downloaded according to the file packing and downloading request, and download the obtained compressed packet, thereby reducing the downloading time, improving the downloading speed of the file to be downloaded and saving the downloading bandwidth.
Fig. 2 is a schematic diagram of main modules of a data downloading device 200 according to an embodiment of the present invention. As shown in fig. 2, the apparatus 200 includes:
A receiving module 201, configured to receive a file packaging and downloading request sent by a terminal;
a compressing module 202, configured to compress and package at least one file to be downloaded based on the request to obtain a compressed package;
A sending module 203, configured to send the compressed packet to the terminal;
wherein the file packaging download request comprises compression parameters, the compression parameters comprising at least one of: and identifying the file to be downloaded and storing the compressed packet.
Optionally, the compression parameters further include compression rules;
The compression module 202 is further configured to: determining at least one file to be downloaded according to the file identifier to be downloaded; and compressing and packaging at least one file to be downloaded according to the compression rule to obtain a compression package, and storing the compression package in the storage position of the compression package.
Optionally, the compression module 202 is further configured to: when a plurality of files to be downloaded are available, compressing and packaging the plurality of files to be downloaded in parallel to obtain compressed files; storing the obtained compressed file in a buffer area; and uploading the compressed file fragments of the buffer area to the storage position of the compressed packet to obtain the compressed packet.
Optionally, the apparatus 200 further includes a list generating module, configured to: generating a packaging downloading task according to the document request to form a task list; and distributing the plurality of tasks in the task list to a plurality of task processing nodes so as to execute the plurality of tasks in parallel.
optionally, the at least one file to be downloaded is stored in an object storage manner.
The data downloading device provided by the embodiment of the invention can compress and pack at least one file to be downloaded according to the file packing and downloading request, and send the obtained compression packet to the local user, so that the requirement of the user for packing and downloading a plurality of files is met, the downloading time is reduced, the downloading speed of the file to be downloaded is improved, the downloading bandwidth is saved, and the resource consumption is reduced.
the device can execute the method provided by the embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method. For technical details that are not described in detail in this embodiment, reference may be made to the method provided by the embodiment of the present invention.
Fig. 3 illustrates an exemplary system architecture 300 to which a data downloading method or a data downloading apparatus according to an embodiment of the present invention may be applied.
as shown in fig. 3, the system architecture 300 may include terminal devices 301, 302, 303, a network 304, and a server 305. The network 304 serves as a medium for providing communication links between the terminal devices 301, 302, 303 and the server 305. Network 304 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.
The user may use the terminal device 301, 302, 303 to interact with the server 305 via the network 304 to receive or send messages or the like. The terminal devices 301, 302, 303 may have various communication client applications installed thereon, such as shopping applications, web browser applications, search applications, instant messaging tools, mailbox clients, social platform software, and the like.
the terminal devices 301, 302, 303 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
the server 305 may be a server providing various services, such as a background management server providing support for shopping websites browsed by the user using the terminal devices 301, 302, 303. The background management server may analyze and perform other processing on the received data such as the product information query request, and feed back a processing result (e.g., target push information and product information) to the terminal device.
It should be noted that the data downloading method provided by the embodiment of the present invention is generally executed by the server 305, and accordingly, the data downloading device is generally disposed in the server 305.
It should be understood that the number of terminal devices, networks, and servers in fig. 3 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
referring now to FIG. 4, a block diagram of a computer system 400 suitable for use with a terminal device implementing an embodiment of the invention is shown. The terminal device shown in fig. 4 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 4, the computer system 400 includes a Central Processing Unit (CPU)401 that can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)402 or a program loaded from a storage section 408 into a Random Access Memory (RAM) 403. In the RAM 403, various programs and data necessary for the operation of the system 400 are also stored. The CPU 401, ROM 402, and RAM 403 are connected to each other via a bus 404. An input/output (I/O) interface 405 is also connected to bus 404.
The following components are connected to the I/O interface 405: an input section 406 including a keyboard, a mouse, and the like; an output section 407 including a display device such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 408 including a hard disk and the like; and a communication section 409 including a network interface card such as a LAN card, a modem, or the like. The communication section 409 performs communication processing via a network such as the internet. A driver 410 is also connected to the I/O interface 405 as needed. A removable medium 411 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 410 as necessary, so that a computer program read out therefrom is mounted into the storage section 408 as necessary.
in particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 409, and/or installed from the removable medium 411. The computer program performs the above-described functions defined in the system of the present invention when executed by a Central Processing Unit (CPU) 401.
It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
the modules described in the embodiments of the present invention may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor includes a sending module, an obtaining module, a determining module, and a first processing module. The names of these modules do not in some cases constitute a limitation on the unit itself, and for example, the sending module may also be described as a "module that sends a picture acquisition request to a connected server".
as another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise:
receiving a file packaging and downloading request sent by a terminal;
Compressing and packaging at least one file to be downloaded based on the request to obtain a compressed package;
Sending the compressed packet to the terminal;
wherein the file packaging download request comprises compression parameters, the compression parameters comprising at least one of: and identifying the file to be downloaded and storing the compressed packet.
According to the technical scheme of the embodiment of the invention, at least one file to be downloaded can be compressed and packaged according to the file packaging and downloading request, and the obtained compressed package is sent to the local user, so that the requirement of packaging and downloading a plurality of files by the user is met, the downloading time is reduced, the downloading speed of the file to be downloaded is improved, and the downloading bandwidth is saved.
The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (12)
1. A method for downloading data, comprising:
Receiving a file packaging and downloading request sent by a terminal;
compressing and packaging at least one file to be downloaded based on the request to obtain a compressed package;
Sending the compressed packet to the terminal;
Wherein the file packaging download request comprises compression parameters, the compression parameters comprising at least one of: and identifying the file to be downloaded and storing the compressed packet.
2. The method of claim 1, wherein the compression parameters further comprise compression rules;
Compressing and packaging at least one file to be downloaded based on the request to obtain a compressed package comprises:
Determining at least one file to be downloaded according to the file identifier to be downloaded;
and compressing and packaging at least one file to be downloaded according to the compression rule to obtain a compression package, and storing the compression package in the storage position of the compression package.
3. The method of claim 2, wherein compression packaging at least one file to be downloaded to obtain a compressed package comprises:
when a plurality of files to be downloaded are available, compressing and packaging the plurality of files to be downloaded in parallel to obtain compressed files;
Storing the obtained compressed file in a buffer area;
and uploading the compressed file fragments of the buffer area to the storage position of the compressed packet to obtain the compressed packet.
4. The method of claim 1, wherein compressing and packaging at least one file to be downloaded based on the request comprises:
generating a packaging downloading task according to the request to form a task list;
And distributing the plurality of tasks in the task list to a plurality of task processing nodes so as to execute the plurality of tasks in parallel.
5. the method according to any of claims 1-4, wherein the at least one file to be downloaded is stored as an object store.
6. a data download apparatus, comprising:
The receiving module is used for receiving a file packaging and downloading request sent by a terminal;
The compression module is used for compressing and packaging at least one file to be downloaded based on the request to obtain a compression package;
a sending module, configured to send the compressed packet to the terminal;
Wherein the file packaging download request comprises compression parameters, the compression parameters comprising at least one of: and identifying the file to be downloaded and storing the compressed packet.
7. the apparatus of claim 6, wherein the compression parameters further comprise compression rules;
The compression module is further to:
determining at least one file to be downloaded according to the file identifier to be downloaded;
And compressing and packaging at least one file to be downloaded according to the compression rule to obtain a compression package, and storing the compression package in the storage position of the compression package.
8. The apparatus of claim 7, wherein the compression module is further configured to:
When a plurality of files to be downloaded are available, compressing and packaging the plurality of files to be downloaded in parallel to obtain compressed files;
Storing the obtained compressed file in a buffer area;
And uploading the compressed file fragments of the buffer area to the storage position of the compressed packet to obtain the compressed packet.
9. The apparatus of claim 6, further comprising a list generation module configured to:
Generating a packaging downloading task according to the document request to form a task list;
and distributing the plurality of tasks in the task list to a plurality of task processing nodes so as to execute the plurality of tasks in parallel.
10. the apparatus according to any one of claims 6-9, wherein the at least one file to be downloaded is stored as an object store.
11. An electronic device, comprising:
One or more processors;
A storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-5.
12. a computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810573022.5A CN110572422A (en) | 2018-06-06 | 2018-06-06 | Data downloading method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810573022.5A CN110572422A (en) | 2018-06-06 | 2018-06-06 | Data downloading method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110572422A true CN110572422A (en) | 2019-12-13 |
Family
ID=68772366
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810573022.5A Pending CN110572422A (en) | 2018-06-06 | 2018-06-06 | Data downloading method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110572422A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112269672A (en) * | 2020-10-27 | 2021-01-26 | 上海淇馥信息技术有限公司 | File download exception handling method and device and electronic equipment |
| CN112929444A (en) * | 2021-02-19 | 2021-06-08 | 广东云智安信科技有限公司 | Data security protection method, system, device and medium based on compressed file |
| CN114221950A (en) * | 2020-09-03 | 2022-03-22 | 华为技术有限公司 | Method for uploading configuration file, method and device for downloading configuration file |
| CN114422499A (en) * | 2021-12-27 | 2022-04-29 | 北京奇艺世纪科技有限公司 | File downloading method, system and device |
| CN114697316A (en) * | 2022-04-24 | 2022-07-01 | 平安科技（深圳）有限公司 | Batch downloading method, device and equipment of data and computer readable medium |
| CN115348254A (en) * | 2022-07-21 | 2022-11-15 | 昆仑数智科技有限责任公司 | File packaging and downloading method and device, electronic equipment and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020049853A1 (en) * | 2000-08-16 | 2002-04-25 | Tan-Na Chu | End-to-end secure file transfer method and system |
| CN103020205A (en) * | 2012-12-05 | 2013-04-03 | 北京普泽天玑数据技术有限公司 | Compression and decompression method based on hardware accelerator card on distributive-type file system |
| CN103559020A (en) * | 2013-11-07 | 2014-02-05 | 中国科学院软件研究所 | Method for realizing parallel compression and parallel decompression on FASTQ file containing DNA (deoxyribonucleic acid) sequence read data |
| CN103841135A (en) * | 2012-11-22 | 2014-06-04 | 腾讯科技（深圳）有限公司 | File accelerative download method and apparatus |
| CN106126756A (en) * | 2016-08-25 | 2016-11-16 | 乐视控股（北京）有限公司 | Webpage subframe resource downloading processing method, terminal and Resource Server |
| CN106559427A (en) * | 2016-11-22 | 2017-04-05 | 台州库得软件有限公司 | A kind of data storage and data transmission system |
| CN106993054A (en) * | 2017-05-05 | 2017-07-28 | 腾讯科技（深圳）有限公司 | Document distribution method, node and system |
-
2018
- 2018-06-06 CN CN201810573022.5A patent/CN110572422A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020049853A1 (en) * | 2000-08-16 | 2002-04-25 | Tan-Na Chu | End-to-end secure file transfer method and system |
| CN103841135A (en) * | 2012-11-22 | 2014-06-04 | 腾讯科技（深圳）有限公司 | File accelerative download method and apparatus |
| CN103020205A (en) * | 2012-12-05 | 2013-04-03 | 北京普泽天玑数据技术有限公司 | Compression and decompression method based on hardware accelerator card on distributive-type file system |
| CN103559020A (en) * | 2013-11-07 | 2014-02-05 | 中国科学院软件研究所 | Method for realizing parallel compression and parallel decompression on FASTQ file containing DNA (deoxyribonucleic acid) sequence read data |
| CN106126756A (en) * | 2016-08-25 | 2016-11-16 | 乐视控股（北京）有限公司 | Webpage subframe resource downloading processing method, terminal and Resource Server |
| CN106559427A (en) * | 2016-11-22 | 2017-04-05 | 台州库得软件有限公司 | A kind of data storage and data transmission system |
| CN106993054A (en) * | 2017-05-05 | 2017-07-28 | 腾讯科技（深圳）有限公司 | Document distribution method, node and system |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114221950A (en) * | 2020-09-03 | 2022-03-22 | 华为技术有限公司 | Method for uploading configuration file, method and device for downloading configuration file |
| CN112269672A (en) * | 2020-10-27 | 2021-01-26 | 上海淇馥信息技术有限公司 | File download exception handling method and device and electronic equipment |
| CN112269672B (en) * | 2020-10-27 | 2023-05-02 | 上海淇馥信息技术有限公司 | File downloading exception handling method and device and electronic equipment |
| CN112929444A (en) * | 2021-02-19 | 2021-06-08 | 广东云智安信科技有限公司 | Data security protection method, system, device and medium based on compressed file |
| CN114422499A (en) * | 2021-12-27 | 2022-04-29 | 北京奇艺世纪科技有限公司 | File downloading method, system and device |
| CN114422499B (en) * | 2021-12-27 | 2023-12-05 | 北京奇艺世纪科技有限公司 | File downloading method, system and device |
| CN114697316A (en) * | 2022-04-24 | 2022-07-01 | 平安科技（深圳）有限公司 | Batch downloading method, device and equipment of data and computer readable medium |
| CN115348254A (en) * | 2022-07-21 | 2022-11-15 | 昆仑数智科技有限责任公司 | File packaging and downloading method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110572422A (en) | Data downloading method and device | |
| WO2019201039A1 (en) | Method and system for updating application and application server | |
| CN106161633B (en) | Transmission method and system for packed files based on cloud computing environment | |
| CN107766509B (en) | Method and device for static backup of webpage | |
| CN111478781B (en) | Message broadcasting method and device | |
| CN110928905B (en) | Data processing method and device | |
| CN111460129A (en) | Method and device for generating identification, electronic equipment and storage medium | |
| CN110798495B (en) | Method and server for end-to-end message push in cluster architecture mode | |
| CN113312355A (en) | Data management method and device | |
| CN111241189A (en) | Method and device for synchronizing data | |
| CN113282589A (en) | Data acquisition method and device | |
| CN113452733A (en) | File downloading method and device | |
| CN111984615A (en) | Method, device and system for sharing files | |
| CN111800223B (en) | Method, device and system for generating sending message and processing receiving message | |
| CN111343220B (en) | Repeater, distributed file transmission method, distributed file transmission system, medium and electronic equipment | |
| CN109144991B (en) | Method and device for dynamic sub-metering, electronic equipment and computer-storable medium | |
| CN116069725A (en) | File migration method, device, apparatus, medium and program product | |
| CN112688982B (en) | User request processing method and device | |
| CN113760861A (en) | Data migration method and device | |
| CN110705935B (en) | Logistics document processing method and device | |
| CN111984616A (en) | Method, device and system for updating shared file | |
| CN113765983A (en) | Site service deployment method and device | |
| CN111984613A (en) | Method, device and system for sharing files | |
| CN111984614A (en) | Method, device and system for sharing files | |
| CN113556370A (en) | Service calling method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |