CN110572422B - Data downloading method, device, equipment and medium - Google Patents
Data downloading method, device, equipment and medium Download PDFInfo
- Publication number
- CN110572422B CN110572422B CN201810573022.5A CN201810573022A CN110572422B CN 110572422 B CN110572422 B CN 110572422B CN 201810573022 A CN201810573022 A CN 201810573022A CN 110572422 B CN110572422 B CN 110572422B
- Authority
- CN
- China
- Prior art keywords
- file
- downloaded
- compressed
- downloading
- compression
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 38
- 230000006835 compression Effects 0.000 claims abstract description 54
- 238000007906 compression Methods 0.000 claims abstract description 54
- 238000004806 packaging method and process Methods 0.000 claims abstract description 35
- 238000012545 processing Methods 0.000 claims description 18
- 238000012856 packing Methods 0.000 claims description 11
- 238000004590 computer program Methods 0.000 claims description 9
- 239000012634 fragment Substances 0.000 claims description 6
- 238000012795 verification Methods 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 11
- 238000004891 communication Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/04—Protocols for data compression, e.g. ROHC
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Security & Cryptography (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention discloses a data downloading method and device, and relates to the technical field of computers. One embodiment of the method comprises the following steps: receiving a file packaging downloading request sent by a terminal; based on the request, compressing and packaging at least one file to be downloaded to obtain a compressed package; transmitting the compressed packet to the terminal; wherein the compression parameters include at least one of: the file to be downloaded identifies and compresses the package storage location. According to the embodiment, at least one file to be downloaded can be compressed and packaged according to the file packaging and downloading request, and the obtained compressed package is sent to the local of the user, so that the requirement of the user for packaging and downloading a plurality of files is met, the downloading time is shortened, the downloading speed of the file to be downloaded is improved, the downloading bandwidth is saved, and the resource consumption is reduced.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a method, an apparatus, a device, and a medium for downloading data.
Background
Object storage, also known as object-based storage, is a method used to describe solving and processing discrete units, which are referred to as objects. Object storage is a data storage method without hierarchy structure, and is commonly used in the cloud. Unlike other data storage methods, object-based storage does not use directory trees. Individual data units (objects) exist at the same level in the storage pool.
In the process of implementing the present invention, the inventor finds that at least the following problems exist in the prior art:
A user always desires a convenient and quick information organization manner when using an object storage product, and thus, a common object storage product is managed in units of objects. However, when the user needs to download a plurality of objects, the objects can only be downloaded in sequence, and the requirement that the user packs and compresses the objects as one object for downloading cannot be met, so that time and broadband resources are wasted.
Disclosure of Invention
In view of this, embodiments of the present invention provide a data downloading method, apparatus, device, and medium, which can compress and package at least one file to be downloaded according to a file package downloading request, and send the obtained compressed package to a user's local, so as to realize the requirement of the user for packaging and downloading a plurality of files, reduce the downloading time, improve the downloading speed of the file to be downloaded, and save the downloading bandwidth.
To achieve the above object, according to an aspect of an embodiment of the present invention, there is provided a data downloading method, including: receiving a file packaging downloading request sent by a terminal; based on the request, compressing and packaging at least one file to be downloaded to obtain a compressed package; transmitting the compressed packet to the terminal; wherein the file package download request includes compression parameters including at least one of: the file to be downloaded identifies and compresses the package storage location.
Optionally, the compression parameter further includes a compression rule;
based on the request, compressing and packaging at least one file to be downloaded includes: determining at least one file to be downloaded according to the file identification to be downloaded; and compressing and packaging at least one file to be downloaded according to the compression rule to obtain a compressed package, and storing the compressed package in the compressed package storage position.
Optionally, compressing the at least one file to be downloaded to obtain the compressed package includes: when a plurality of files to be downloaded are provided, compressing and packaging the plurality of files to be downloaded in parallel to obtain compressed files; storing the obtained compressed file in a buffer; and uploading the compressed file fragments of the buffer area to the compressed packet storage position to obtain a compressed packet.
Optionally, the method further comprises: generating a packaged downloading task according to the request to form a task list; and distributing the plurality of tasks in the task list to a plurality of task processing nodes so as to execute the plurality of tasks in parallel.
Optionally, the at least one file to be downloaded is stored in an object storage manner.
To achieve the above object, according to another aspect of the embodiments of the present invention, there is provided a data downloading apparatus, including: the receiving module is used for receiving a file packaging downloading request sent by the terminal; the compression module is used for compressing and packaging at least one file to be downloaded based on the request to obtain a compressed package; a transmitting module, configured to transmit the compressed packet to the terminal; wherein the file package download request includes compression parameters including at least one of: the file to be downloaded identifies and compresses the package storage location.
Optionally, the compression parameter further includes a compression rule;
The compression module is further configured to: determining at least one file to be downloaded according to the file identification to be downloaded; and compressing and packaging at least one file to be downloaded according to the compression rule to obtain a compressed package, and storing the compressed package in the compressed package storage position.
Optionally, the compression module is further configured to: when a plurality of files to be downloaded are provided, compressing and packaging the plurality of files to be downloaded in parallel to obtain compressed files; storing the obtained compressed file in a buffer; and uploading the compressed file fragments of the buffer area to the compressed packet storage position to obtain a compressed packet.
Optionally, the apparatus further includes a list generation module configured to: generating a packaged downloading task according to the request to form a task list; and distributing the plurality of tasks in the task list to a plurality of task processing nodes so as to execute the plurality of tasks in parallel.
Optionally, the at least one file to be downloaded is stored in an object storage manner.
To achieve the above object, according to still another aspect of the embodiments of the present invention, there is provided an electronic device including: one or more processors; and the storage device is used for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors are enabled to realize the data downloading method of the embodiment of the invention.
To achieve the above object, according to still another aspect of the embodiments of the present invention, there is provided a computer-readable medium having stored thereon a computer program which, when executed by a processor, implements a data downloading method of the embodiments of the present invention.
One embodiment of the above invention has the following advantages or benefits: because the technical means of compressing and packing at least one file to be downloaded according to the received file packing and downloading request to obtain a compressed package and then sending the compressed package to the terminal is adopted, the technical problem that a plurality of objects cannot be used as one object to be downloaded in the prior art is solved, the downloading time is further shortened, the downloading speed of the file to be downloaded is improved, and the downloading bandwidth is saved.
Further effects of the above-described non-conventional alternatives are described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
FIG. 1 is a schematic diagram of the main flow of a data download method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of the main modules of a data downloading apparatus according to an embodiment of the present invention;
FIG. 3 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;
Fig. 4 is a schematic diagram of a computer system suitable for use in implementing an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention will now be described with reference to the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding, and are to be considered merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Fig. 1 is a schematic diagram of main flow of a data downloading method according to an embodiment of the present invention, as shown in fig. 1, the method includes:
Step S101: receiving a file packaging downloading request sent by a terminal; wherein the file package download request includes compression parameters including at least one of: a file identifier to be downloaded and a compressed package storage position;
step S102: based on the request, compressing and packaging at least one file to be downloaded to obtain a compressed package;
step S103: and sending the compressed packet to the terminal.
For step S101, where the file identifier to be downloaded is used to distinguish the files to be downloaded, for example, the file identifier to be downloaded may be a URL (Uniform Resource Locator uniform resource locator), and the user may specify a plurality of files to be downloaded by specifying a series of URLs. The file identification to be downloaded may also be the name of the file to be downloaded. In other alternative embodiments, as shown in table 1 below, the compression parameters may also include compression rules (rule), user account numbers (principle), storage locations of files to be downloaded, and callback notification interfaces (notifyurl). Compression rules may include, among other things, compression format, compression mode (e.g., fastest, faster, standard or stored), whether to delete a source file, and whether to add user identity verification information. The compression format refers to a compression method that a file is compressed by a certain program, for example, tar.gz, zip, tar, and the default compression format may be zip. The callback notification interface is used to submit the processing results to the address specified by notifyurl after the compression packaging is completed.
Table 1:
| Parameter name | Meaning of parameters | Description of the invention |
| rule | Compression rules | Must be filled with |
| principle | User account | Must be filled with |
| source | File identification to be downloaded, representing a plurality of files to be packed and compressed | Must be filled with |
| destination | Compressed package storage location | Must be filled with |
| type | Compression formats, e.g. tar. Gz, zip, tar, default to zip | Not necessary to be filled with |
| notifyurl | After packing compression is completed, the processing result is submitted to the address designated by notifyurl | Not necessary to be filled with |
After receiving the file package downloading request of the user, the parameters in the request can be checked to be legal, for example, whether the user account exists and whether the arrears are caused; whether the file to be downloaded exists or not and whether the file belongs to the user or not; whether the storage position of the compressed packet exists or not and the operation is authorized; whether the compression format is supported. If the parameter rule in the file package download request can place the request in the task queue, the parameter check fails and an error is returned and an illegal parameter is indicated.
The files to be downloaded can be stored in an object storage mode and can be stored in public cloud. The public cloud generally refers to a cloud which can be used and is provided by a third party provider for a user, and the public cloud can be generally used through a network and shares resource services when the public cloud has core properties.
For step S102, based on the request, the process of compressing and packaging at least one file to be downloaded to obtain a compressed package may include:
determining at least one file to be downloaded according to the file identification to be downloaded;
And compressing and packing at least one file to be downloaded according to the compression rule to obtain a compressed package, and storing the compressed package in the compressed package storage position.
The files to be downloaded can be audio and video, pictures, documents or archive files, logs and other data. The compression packaging refers to compressing one or more files by a compression algorithm to form a compressed file packet, which can save space and facilitate transmission between networks.
In an alternative embodiment, the process of compression packaging at least one file to be downloaded to obtain a compressed package includes:
When a plurality of files to be downloaded are provided, compressing and packaging the plurality of files to be downloaded in parallel to obtain compressed files;
Storing the obtained compressed file in a buffer;
and uploading the compressed file fragments of the buffer area to the compressed packet storage position to obtain a compressed packet.
Specifically, the package downloading request is processed immediately after the package downloading request is received, and the file to be downloaded is downloaded in parallel in a thread pool mode by utilizing the characteristic of high throughput of object storage. In order to improve the task operation efficiency, the method can be used for downloading and packing at the same time, and the partially packed files are placed in the buffer area, and the compressed files in the buffer area are sent to the compressed package storage position in real time in a fragmented uploading mode, so that the compressed package is obtained. The method comprises the steps of dividing a file into a series of small data pieces with specific sizes, uploading the small data pieces to a server, and merging the small data pieces into a file at the server after all uploading is completed.
Alternatively, md5 of the compressed package (Message-Digest Algorithm 5, 5 th edition of the Message Digest Algorithm for ensuring that the transmission of the information is completely consistent) may be calculated, as well as md5 of the compressed package downloaded locally to the user, to verify the integrity of the compressed package downloaded by the user. When md5 of the two are the same, the compressed package downloaded by the user is indicated to be complete, otherwise, the compressed package downloaded by the user is indicated to be incomplete.
More specifically, based on the request, compressing and packaging at least one file to be downloaded includes: :
Generating a packaged downloading task according to the request to form a task list;
And distributing the plurality of tasks in the task list to a plurality of task processing nodes so as to execute the plurality of tasks in parallel.
In this embodiment, a distributed task scheduling framework may be adopted, and a large amount of packaging and downloading tasks are shared by the nodes of the cluster, so as to improve the processing efficiency of the batch tasks. The distributed task scheduling framework aims to enable the packaged downloading tasks to be distributed to processing programs of a plurality of hosts or different thread groups for parallel execution, all the packaged downloading tasks can be processed quickly without repetition and omission, the task is not repeatedly executed, and the task can be automatically retried when errors occur, so that the high availability, usability and scalability of the system are ensured.
As a specific example, a zookeeper (zookeeper is an open source file application program interface, which can synchronize distributed processes of a large-scale system with each other, so that all clients making requests can obtain consistent data) can realize a distributed task scheduling framework, when a package downloading task is started, the package downloading task is registered by creating a EPHEMERAL node (temporary node), a task serialization result is stored in a PERSISTENT node (permanent node), and the package downloading task is ensured to be processed only once in a locking mode.
In this embodiment, after the package download task is completed, the processing result status is submitted to the website to which notifyurl points.
The data downloading method provided by the embodiment of the invention can compress and pack at least one file to be downloaded according to the file packing and downloading request, and download the obtained compressed package, thereby reducing the downloading time, improving the downloading speed of the file to be downloaded and saving the downloading bandwidth.
Fig. 2 is a schematic diagram of main modules of a data downloading apparatus 200 according to an embodiment of the present invention. As shown in fig. 2, the apparatus 200 includes:
a receiving module 201, configured to receive a file package downloading request sent by a terminal;
A compression module 202, configured to compress and package at least one file to be downloaded based on the request, so as to obtain a compressed package;
a transmitting module 203, configured to transmit the compressed packet to the terminal;
Wherein the file package download request includes compression parameters including at least one of: the file to be downloaded identifies and compresses the package storage location.
Optionally, the compression parameter further includes a compression rule;
The compression module 202 is further configured to: determining at least one file to be downloaded according to the file identification to be downloaded; and compressing and packaging at least one file to be downloaded according to the compression rule to obtain a compressed package, and storing the compressed package in the compressed package storage position.
Optionally, the compression module 202 is further configured to: when a plurality of files to be downloaded are provided, compressing and packaging the plurality of files to be downloaded in parallel to obtain compressed files; storing the obtained compressed file in a buffer; and uploading the compressed file fragments of the buffer area to the compressed packet storage position to obtain a compressed packet.
Optionally, the apparatus 200 further includes a list generating module configured to: generating a packing download task according to the text request to form a task list; and distributing the plurality of tasks in the task list to a plurality of task processing nodes so as to execute the plurality of tasks in parallel.
Optionally, the at least one file to be downloaded is stored in an object storage manner.
The data downloading device of the embodiment of the invention can compress and pack at least one file to be downloaded according to the file packing and downloading request, and send the obtained compressed package to the user local so as to realize the requirement of the user for packing and downloading a plurality of files, reduce the downloading time, improve the downloading speed of the file to be downloaded, save the downloading bandwidth and reduce the resource consumption.
The device can execute the method provided by the embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method. Technical details not described in detail in this embodiment may be found in the methods provided in the embodiments of the present invention.
Fig. 3 illustrates an exemplary system architecture 300 to which the data download method or data download apparatus of embodiments of the present invention may be applied.
As shown in fig. 3, the system architecture 300 may include terminal devices 301, 302, 303, a network 304, and a server 305. The network 304 is used as a medium to provide communication links between the terminal devices 301, 302, 303 and the server 305. The network 304 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
A user may interact with the server 305 via the network 304 using the terminal devices 301, 302, 303 to receive or send messages or the like. Various communication client applications, such as shopping class applications, web browser applications, search class applications, instant messaging tools, mailbox clients, social platform software, etc., may be installed on the terminal devices 301, 302, 303.
The terminal devices 301, 302, 303 may be a variety of electronic devices having a display screen and supporting web browsing, including but not limited to smartphones, tablets, laptop and desktop computers, and the like.
The server 305 may be a server providing various services, such as a background management server providing support for shopping-type websites browsed by the user using the terminal devices 301, 302, 303. The background management server can analyze and other processing on the received data such as the product information inquiry request and the like, and feed back processing results (such as target push information and product information) to the terminal equipment.
It should be noted that, the data downloading method provided in the embodiment of the present invention is generally executed by the server 305, and accordingly, the data downloading device is generally disposed in the server 305.
It should be understood that the number of terminal devices, networks and servers in fig. 3 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 4, there is illustrated a schematic diagram of a computer system 400 suitable for use in implementing an embodiment of the present invention. The terminal device shown in fig. 4 is only an example, and should not impose any limitation on the functions and the scope of use of the embodiment of the present invention.
As shown in fig. 4, the computer system 400 includes a Central Processing Unit (CPU) 401, which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 402 or a program loaded from a storage section 408 into a Random Access Memory (RAM) 403. In RAM 403, various programs and data required for the operation of system 400 are also stored. The CPU 401, ROM 402, and RAM 403 are connected to each other by a bus 404. An input/output (I/O) interface 405 is also connected to bus 404.
The following components are connected to the I/O interface 405: an input section 406 including a keyboard, a mouse, and the like; an output portion 407 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker, and the like; a storage section 408 including a hard disk or the like; and a communication section 409 including a network interface card such as a LAN card, a modem, or the like. The communication section 409 performs communication processing via a network such as the internet. The drive 410 is also connected to the I/O interface 405 as needed. A removable medium 411 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is installed on the drive 410 as needed, so that a computer program read therefrom is installed into the storage section 408 as needed.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication portion 409 and/or installed from the removable medium 411. The above-described functions defined in the system of the present invention are performed when the computer program is executed by a Central Processing Unit (CPU) 401.
The computer readable medium shown in the present invention may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules involved in the embodiments of the present invention may be implemented in software or in hardware. The described modules may also be provided in a processor, for example, as: a processor includes a sending module, an obtaining module, a determining module, and a first processing module. The names of these modules do not constitute a limitation on the unit itself in some cases, and for example, the transmitting module may also be described as "a module that transmits a picture acquisition request to a connected server".
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be present alone without being fitted into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to include:
Receiving a file packaging downloading request sent by a terminal;
based on the request, compressing and packaging at least one file to be downloaded to obtain a compressed package;
transmitting the compressed packet to the terminal;
Wherein the file package download request includes compression parameters including at least one of: the file to be downloaded identifies and compresses the package storage location.
According to the technical scheme, at least one file to be downloaded can be compressed and packaged according to the file packaging and downloading request, and the obtained compressed package is sent to the user local, so that the requirement of the user for packaging and downloading a plurality of files is met, the downloading time is shortened, the downloading speed of the file to be downloaded is improved, and the downloading bandwidth is saved.
The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives can occur depending upon design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.
Claims (8)
1. A method of data download, comprising:
receiving a file packaging and downloading request sent by a terminal, identifying compression parameters in the request, and calling a corresponding verification mode to determine that the compression parameters are legal;
Based on the request, determining at least one file to be downloaded according to the file identification to be downloaded, compressing and packing the at least one file to be downloaded according to a compression rule, wherein the obtained partially packed file is placed in a buffer area and uploaded to a compressed package storage position in real time so as to obtain a compressed package; the at least one file to be downloaded is stored in an object storage mode;
transmitting the compressed packet to the terminal;
wherein the file package download request includes compression parameters including: file identification to be downloaded, compressed package storage location, user account number and compression rule.
2. The method of claim 1, wherein compressing and packaging at least one file to be downloaded according to a compression rule, wherein placing the obtained partially packaged file in a buffer and uploading in real time to a compressed package storage location to obtain a compressed package comprises:
When a plurality of files to be downloaded are provided, compressing and packaging the plurality of files to be downloaded in parallel to obtain compressed files;
Storing the obtained compressed file in a buffer;
and uploading the compressed file fragments of the buffer area to the compressed packet storage position to obtain a compressed packet.
3. The method of claim 1, wherein determining at least one file to be downloaded based on the request according to the file identification to be downloaded, compressing and packaging the at least one file to be downloaded according to a compression rule comprises:
Generating a packaged downloading task according to the request to form a task list;
And distributing the plurality of tasks in the task list to a plurality of task processing nodes so as to execute the plurality of tasks in parallel.
4. A data downloading apparatus, comprising:
The receiving module is used for receiving a file packaging downloading request sent by the terminal, identifying compression parameters in the request, and calling a corresponding verification mode to determine that the compression parameters are legal;
The compression module is used for determining at least one file to be downloaded according to the file identification to be downloaded based on the request, compressing and packing the at least one file to be downloaded according to a compression rule, wherein the obtained partially packed file is placed in a buffer area and uploaded to a compression packet storage position in real time so as to obtain a compression packet; the at least one file to be downloaded is stored in an object storage mode;
a transmitting module, configured to transmit the compressed packet to the terminal;
wherein the file package download request includes compression parameters including: file identification to be downloaded, compressed package storage location, user account number and compression rule.
5. The apparatus of claim 4, wherein the compression module is further to:
When a plurality of files to be downloaded are provided, compressing and packaging the plurality of files to be downloaded in parallel to obtain compressed files;
Storing the obtained compressed file in a buffer;
and uploading the compressed file fragments of the buffer area to the compressed packet storage position to obtain a compressed packet.
6. The apparatus of claim 4, further comprising a list generation module configured to:
Generating a packaged downloading task according to the request to form a task list;
And distributing the plurality of tasks in the task list to a plurality of task processing nodes so as to execute the plurality of tasks in parallel.
7. An electronic device, comprising:
One or more processors;
storage means for storing one or more programs,
When executed by the one or more processors, causes the one or more processors to implement the method of any of claims 1-3.
8. A computer readable medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method according to any of claims 1-3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810573022.5A CN110572422B (en) | 2018-06-06 | 2018-06-06 | Data downloading method, device, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810573022.5A CN110572422B (en) | 2018-06-06 | 2018-06-06 | Data downloading method, device, equipment and medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110572422A CN110572422A (en) | 2019-12-13 |
| CN110572422B true CN110572422B (en) | 2024-07-19 |
Family
ID=68772366
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810573022.5A Active CN110572422B (en) | 2018-06-06 | 2018-06-06 | Data downloading method, device, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110572422B (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114221950A (en) * | 2020-09-03 | 2022-03-22 | 华为技术有限公司 | Method for uploading configuration file, method and device for downloading configuration file |
| CN112269672B (en) * | 2020-10-27 | 2023-05-02 | 上海淇馥信息技术有限公司 | File downloading exception handling method and device and electronic equipment |
| CN112929444B (en) * | 2021-02-19 | 2022-09-23 | 广东云智安信科技有限公司 | Data security protection method, system, device and medium based on compressed file |
| CN114253932A (en) * | 2021-12-22 | 2022-03-29 | 徐工汉云技术股份有限公司 | Batch file access method and system based on Internet of things terminal |
| CN114422499B (en) * | 2021-12-27 | 2023-12-05 | 北京奇艺世纪科技有限公司 | File downloading method, system and device |
| CN114697316B (en) * | 2022-04-24 | 2024-03-12 | 平安科技（深圳）有限公司 | Batch downloading method, device and equipment of data and computer readable medium |
| CN115269510A (en) * | 2022-06-30 | 2022-11-01 | 启明信息技术股份有限公司 | Python-based file batch processing method |
| CN115348254B (en) * | 2022-07-21 | 2023-05-30 | 昆仑数智科技有限责任公司 | File packing and downloading method and device, electronic equipment and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103020205A (en) * | 2012-12-05 | 2013-04-03 | 北京普泽天玑数据技术有限公司 | Compression and decompression method based on hardware accelerator card on distributive-type file system |
| CN103841135A (en) * | 2012-11-22 | 2014-06-04 | 腾讯科技（深圳）有限公司 | File accelerative download method and apparatus |
| CN106559427A (en) * | 2016-11-22 | 2017-04-05 | 台州库得软件有限公司 | A kind of data storage and data transmission system |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1312193A2 (en) * | 2000-08-16 | 2003-05-21 | Filestream, Inc. | End-to-end secure file transfer method and system |
| CN103559020B (en) * | 2013-11-07 | 2016-07-06 | 中国科学院软件研究所 | A kind of DNA reads ordinal number according to the compression of FASTQ file in parallel and decompression method |
| CN106126756A (en) * | 2016-08-25 | 2016-11-16 | 乐视控股（北京）有限公司 | Webpage subframe resource downloading processing method, terminal and Resource Server |
| CN106993054B (en) * | 2017-05-05 | 2022-06-03 | 腾讯科技（深圳）有限公司 | File distribution method, node and system |
-
2018
- 2018-06-06 CN CN201810573022.5A patent/CN110572422B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103841135A (en) * | 2012-11-22 | 2014-06-04 | 腾讯科技（深圳）有限公司 | File accelerative download method and apparatus |
| CN103020205A (en) * | 2012-12-05 | 2013-04-03 | 北京普泽天玑数据技术有限公司 | Compression and decompression method based on hardware accelerator card on distributive-type file system |
| CN106559427A (en) * | 2016-11-22 | 2017-04-05 | 台州库得软件有限公司 | A kind of data storage and data transmission system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110572422A (en) | 2019-12-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110572422B (en) | Data downloading method, device, equipment and medium | |
| CN106161633B (en) | Transmission method and system for packed files based on cloud computing environment | |
| CN111045833B (en) | Interface calling method and device | |
| CN103401934A (en) | Method and system for acquiring log data | |
| CN110928905B (en) | Data processing method and device | |
| CN113517985B (en) | File data processing method and device, electronic equipment and computer readable medium | |
| CN113452733A (en) | File downloading method and device | |
| CN110162410B (en) | A message processing method and device | |
| CN111478781B (en) | Message broadcasting method and device | |
| CN107809470B (en) | Method and device for automatically deploying application program | |
| CN111800223A (en) | Method, device and system for generating sending message and processing receiving message | |
| CN113282589A (en) | Data acquisition method and device | |
| CN112416396A (en) | Application program updating method and system | |
| CN110798495B (en) | Method and server for end-to-end message push in cluster architecture mode | |
| CN113312355A (en) | Data management method and device | |
| CN109144991B (en) | Method and device for dynamic sub-metering, electronic equipment and computer-storable medium | |
| CN111984616B (en) | A method, device and system for updating shared files | |
| CN111984615B (en) | File sharing method, device and system | |
| CN110705935A (en) | Logistics document processing method and device | |
| CN112688982B (en) | User request processing method and device | |
| CN111984613B (en) | Method, device and system for sharing files | |
| CN111984614B (en) | Method, device and system for sharing files | |
| CN111984612A (en) | A method, device and system for sharing files | |
| CN110597802B (en) | Message processing method and device | |
| CN110019671B (en) | Method and system for processing real-time message |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |