CN112087512A - Method for automatically expanding material downloading task - Google Patents
Method for automatically expanding material downloading task Download PDFInfo
- Publication number
- CN112087512A CN112087512A CN202010935899.1A CN202010935899A CN112087512A CN 112087512 A CN112087512 A CN 112087512A CN 202010935899 A CN202010935899 A CN 202010935899A CN 112087512 A CN112087512 A CN 112087512A
- Authority
- CN
- China
- Prior art keywords
- downloading
- disk
- task
- downloaded
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention discloses a method for automatically expanding a material downloading task, which relates to the technical field of material sorting, and aims at solving the problems of low downloading speed, large occupied file storage and low preview of the existing material, the following scheme is proposed, and the method comprises the steps of improving the downloading speed, automatically expanding a white list and automatically expanding a disk: the steps for increasing the download speed are as follows: s1, the client sends the material address to be uploaded to the special staging system, the special staging system saves the material address to be downloaded to the material task list, and the marking states are not downloaded, the creation time, the update time and the failure times are 0. The invention can adjust the downloading speed and improve the stability of the whole system, thereby improving the safety and reliability, improving the approval efficiency of bank personnel, improving the material incoming efficiency of customers, reducing the waiting time of the customers, reducing the time cost of the customers and lightening the burden of banks.
Description
Technical Field
The invention relates to the technical field of material sorting, in particular to a method for automatically expanding a material downloading task.
Background
With the continuous improvement of banking information, internet finance is applied more and more in banking. In internet finance, a banking staff can obtain a large amount of important information related to a client in real time by applying an advanced informatization technology to financial business, so that the operation is convenient, and the efficiency is improved. Internet financial technology has originated in recent years and has been developed based on the business needs of banking personnel. The establishment of internet finance does not require banking staff to be operated by a mobile phone to improve the safety and efficiency of business. Although the technology content behind the device is high, the device is very simple and humanized for users, and the difficulty of business personnel is not increased.
Because the bank special financial system is an independent local area network system, a large amount of personal information needs to be uploaded to the special financial system when a customer conducts business from the internet environment, and the special financial system has complex business and large customer volume, so that a plurality of defects exist. For example, current material download systems suffer from the following drawbacks: (1) the diversified configuration of the inline white list access address of the client material address is complicated; (2) after the client is big, the downloading rhythm cannot be controlled by the client material to meet the business requirement; (3) the material file format and the file of the client do not accord with the account display problem; (4) the problems of slow downloading, large storage occupation, slow preview and the like caused by overlarge material files of a client from an interconnection area; (5) disk storage space cannot meet the requirement of material growth too fast, and therefore a method for automatically expanding the material downloading task is provided.
Disclosure of Invention
The method for automatically expanding the material downloading task solves the problems of low downloading speed of the existing material, large occupied storage of large files and low preview speed.
In order to achieve the purpose, the invention adopts the following technical scheme:
a method for automatically expanding a material downloading task comprises the steps of improving the downloading speed, automatically expanding a white list and automatically expanding a disk:
the steps for increasing the download speed are as follows:
s1, the client sends the material address to be sent to the special staging system, the special staging system saves the material address to be downloaded to the material task list, and the marking state is not downloaded, the creating time, the updating time and the failure frequency is 0;
s2, the material downloading system adopts a production consumption model, the producer takes the data from the material task table and marks the materials in the material task table as read, then the taken data is stored in a message blocking queue, the producer takes the data according to the rule that the data is not downloaded or fails to be downloaded and the failure times are less than 5, the priority time is short, if the time is the same, the priority downloading failure times are less, and if the data is not downloaded and fails to be downloaded, the priority downloading is performed;
s3, the frequency of fetching data by producer is automatically adjusted, if the download speed is fast, the frequency of fetching data by producer will also be higher, the corresponding relation is if the number of message blocking queue tasks is less than the number of threads, the message blocking queue outflow speed and the average time of fetching data by producer for a single time
S4, the consumer takes out the task of the message blocking queue to download, if the download fails, the download state of the material task list is marked as failure and the failure times are increased by one, if the download succeeds, the download state of the material task list is marked as success;
s4, the failed task will follow the data fetching rule of the generator and return to the message blocking queue again, if the consumer finds that the material is downloaded before or not downloaded, the consumer directly skips the downloaded part which is directly downloaded but not downloaded, which is the parameter setting at the http request head, which is the breakpoint resume mechanism;
s5, the number of consumers determines the downloading speed to a certain extent, so the control of the number of consumer threads is also critical, and the size of the number of consumer threads is inversely proportional to the current residual bandwidth size;
s6, the http request header parameter value of the breakpoint continuous transmission mechanism is a result obtained through an algorithm;
the automatic expansion steps of the white list are as follows:
s7, the material downloading system needs to download the material files, the material files are on the Internet, the material downloading system is in an in-line local area network, the material downloading system needs to go through a proxy service to an out-of-line material downloading transfer service, the in-line white list only needs to be configured with an out-of-line material downloading transfer service ip, the transfer service can be communicated with the Internet, and therefore the material downloading system directly goes through the material downloading transfer service to obtain the material files which the task wants to download, and the problem of automatic expansion of the white list can be perfectly solved;
the automatic expansion of the disk comprises the following steps:
s8, customer feed material;
s9, writing a material downloading task table by the special staging system;
s10, the material downloading system reads the material downloading task to download;
s11, the material downloading system reads the disk identifier of the stored disk and generates a file address;
s12, the material downloading system keeps the downloading file to the disk address generated in the last step;
s13, judging whether the file downloading by the material downloading system is successful, if not, setting the material downloading table to change the material task to a failure state and entering a material downloading task waiting for the next reading failure, if so, saving the address of the previous step to the material table and carrying out the next step;
s14, the single material download is finished;
s15, the bank manager previews the material file on the web page;
s16, the web browser requests the special staging system according to the address of the material file;
s17, the special staging system reads the corresponding disk material file according to the drive symbol information contained in the material address and returns the corresponding disk material file to the web browser;
s18, the bank manager successfully previews the material file.
Preferably, the algorithm for reading the stored drive letter in S11 is specifically:
I. storing the drive letter keywords, the drive letter path and the drive letter state into a database;
II. Reading the drive letter and the drive letter path in the in-use state;
III, returning the drive letter path to a material downloading task;
the I records stored in the database can be multiple, namely configuration information of dynamic expansion of the disk, when the last disk is full and a disk is newly added, a record can be newly added to the disk configuration table, the drive letter path and the drive letter key words of the record are information of the newly added disk, the drive letter state is in use, and other drive letters are marked as used.
Preferably, the algorithm for reading the corresponding disk material file in S17 is specifically:
a. acquiring a material address, wherein the address rule is as follows: d { N }/xxx/yy.zz, wherein D is a fixed value, N is a variable representing that the current block of disk is the second block of disk, x is a file path, y is a file name, and z is a file suffix;
b. reading disc character information, e.g., D1;
c. reading a corresponding disk path according to the drive letter information;
d. splicing disk paths and file paths, such as/data/images/D1/xxx/yy.zz;
and c, reading the disk label information of the disk label key word corresponding to the disk configuration table.
Compared with the prior art, the invention has the beneficial effects that:
1. the invention not only comprises the steps of automatically adjusting the number of tasks to be downloaded by the production consumption model and automatically adjusting the downloading speed, but also comprises the steps of improving the downloading speed to the maximum extent by breakpoint transmission, and improving the stability of the whole system by a failure retry mechanism, thereby improving the safety and reliability and improving the approval efficiency of bank personnel.
2. The invention not only comprises the automatic expansion of the white list, but also comprises the automatic expansion of the disk for storing the file, under the condition that the business volume is continuously increased and the cooperation mechanisms are continuously increased, the bank operation and maintenance personnel do not need to separately maintain the material downloading system, the client material entering efficiency is improved, the waiting time of the client is reduced, the time cost of the client is reduced, and the burden of the bank is lightened.
Drawings
FIG. 1 is a material download flow diagram of a method for automatically expanding a material download task according to the present invention;
fig. 2 is a file reading flow chart of a method for automatically expanding a material downloading task according to the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments.
Referring to fig. 1-2, a method for automatically expanding a material download task includes the steps of increasing a download speed, automatically expanding a white list, and automatically expanding a disk:
the steps for increasing the download speed are as follows:
s1, the client sends the material address to be sent to the special staging system, the special staging system saves the material address to be downloaded to the material task list, and the marking state is not downloaded, the creating time, the updating time and the failure frequency is 0;
s2, the material downloading system adopts a production consumption model, the producer takes the data from the material task table and marks the materials in the material task table as read, then the taken data is stored in a message blocking queue, the producer takes the data according to the rule that the data is not downloaded or fails to be downloaded and the failure times are less than 5, the priority time is short, if the time is the same, the priority downloading failure times are less, and if the data is not downloaded and fails to be downloaded, the priority downloading is performed;
s3, the frequency of fetching data by producer is automatically adjusted, if the download speed is fast, the frequency of fetching data by producer will also be higher, the corresponding relation is if the number of message blocking queue tasks is less than the number of threads, the message blocking queue outflow speed and the average time of fetching data by producer for a single time
S4, the consumer takes out the task of the message blocking queue to download, if the download fails, the download state of the material task list is marked as failure and the failure times are increased by one, if the download succeeds, the download state of the material task list is marked as success;
s4, the failed task will follow the data fetching rule of the generator and return to the message blocking queue again, if the consumer finds that the material is downloaded before or not downloaded, the consumer directly skips the downloaded part which is directly downloaded but not downloaded, which is the parameter setting at the http request head, which is the breakpoint resume mechanism;
s5, the number of consumers determines the downloading speed to a certain extent, so the control of the number of consumer threads is also critical, and the size of the number of consumer threads is inversely proportional to the current residual bandwidth size;
s6, the http request header parameter value of the breakpoint continuous transmission mechanism is a result obtained through an algorithm;
the automatic expansion steps of the white list are as follows:
s7, the material downloading system needs to download the material files, the material files are on the Internet, the material downloading system is in an in-line local area network, the material downloading system needs to go through a proxy service to an out-of-line material downloading transfer service, the in-line white list only needs to be configured with an out-of-line material downloading transfer service ip, the transfer service can be communicated with the Internet, and therefore the material downloading system directly goes through the material downloading transfer service to obtain the material files which the task wants to download, and the problem of automatic expansion of the white list can be perfectly solved;
the automatic expansion of the disk comprises the following steps:
s8, customer feed material;
s9, writing a material downloading task table by the special staging system;
s10, the material downloading system reads the material downloading task to download;
s11, the material downloading system reads the disk identifier of the stored disk and generates a file address;
s12, the material downloading system keeps the downloading file to the disk address generated in the last step;
s13, judging whether the file downloading by the material downloading system is successful, if not, setting the material downloading table to change the material task to a failure state and entering a material downloading task waiting for the next reading failure, if so, saving the address of the previous step to the material table and carrying out the next step;
s14, the single material download is finished;
s15, the bank manager previews the material file on the web page;
s16, the web browser requests the special staging system according to the address of the material file;
s17, the special staging system reads the corresponding disk material file according to the drive symbol information contained in the material address and returns the corresponding disk material file to the web browser;
s18, the bank manager successfully previews the material file.
In this embodiment, the algorithm for reading and storing the disk identifier in S11 specifically includes:
I. storing the drive letter keywords, the drive letter path and the drive letter state into a database;
II. Reading the drive letter and the drive letter path in the in-use state;
III, returning the drive letter path to a material downloading task;
when the last disk is full and a new disk is added, a record is added to the disk configuration table, the drive letter path and the drive letter key word of the record are the information of the new disk, the drive letter state is in use, and other drive letters are marked as used.
In this embodiment, the algorithm for reading the corresponding disk material file in S17 specifically includes:
a. acquiring a material address, wherein the address rule is as follows: d { N }/xxx/yy.zz, wherein D is a fixed value, N is a variable representing that the current block of disk is the second block of disk, x is a file path, y is a file name, and z is a file suffix;
b. reading disc character information, e.g., D1;
c. reading a corresponding disk path according to the drive letter information;
d. splicing disk paths and file paths, such as/data/images/D1/xxx/yy.zz;
and c, reading the disk label information of the disk label key word corresponding to the disk configuration table.
The working principle is as follows:
the client opens the app.
Collecting the authentication material.
And thirdly, submitting the authentication material of the last step to a staging system.
Fourthly, the staging system saves the address of the authentication material to be downloaded to the material task table.
Fifthly, reading the un-downloaded or failed task of the material task table by the material download producer.
Storing the read material task into a message queue.
Material download consumers take message queue material tasks out.
And generating the disk address of the download file according to the rule by the material download consumer.
Ninthly, the material downloading consumer downloads the materials according to the address information contained in the extracted material task
And the downloading consumer of the R material judges whether the downloading is successful, if so, the next step is carried out, and if not, the material changing state is set to be a failure state.
Saving and ninthly downloading the file according to the generated disk address
The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art should be considered to be within the technical scope of the present invention, and the technical solutions and the inventive concepts thereof according to the present invention should be equivalent or changed within the scope of the present invention.
Claims (3)
1. A method for automatically expanding a material downloading task comprises the steps of improving the downloading speed, automatically expanding a white list and automatically expanding a disk:
the steps for increasing the download speed are as follows:
s1, the client sends the material address to be sent to the special staging system, the special staging system saves the material address to be downloaded to the material task list, and the marking state is not downloaded, the creating time, the updating time and the failure frequency is 0;
s2, the material downloading system adopts a production consumption model, the producer takes the data from the material task table and marks the materials in the material task table as read, then the taken data is stored in a message blocking queue, the producer takes the data according to the rule that the data is not downloaded or fails to be downloaded and the failure times are less than 5, the priority time is short, if the time is the same, the priority downloading failure times are less, and if the data is not downloaded and fails to be downloaded, the priority downloading is performed;
s3, the frequency of fetching data by producer is automatically adjusted, if the download speed is fast, the frequency of fetching data by producer will also be higher, the corresponding relation is if the number of message blocking queue tasks is less than the number of threads, the message blocking queue outflow speed and the average time of fetching data by producer for a single time
S4, the consumer takes out the task of the message blocking queue to download, if the download fails, the download state of the material task list is marked as failure and the failure times are increased by one, if the download succeeds, the download state of the material task list is marked as success;
s4, the failed task will follow the data fetching rule of the generator and return to the message blocking queue again, if the consumer finds that the material is downloaded before or not downloaded, the consumer directly skips the downloaded part which is directly downloaded but not downloaded, which is the parameter setting at the http request head, which is the breakpoint resume mechanism;
s5, the number of consumers determines the downloading speed to a certain extent, so the control of the number of consumer threads is also critical, and the size of the number of consumer threads is inversely proportional to the current residual bandwidth size;
s6, the http request header parameter value of the breakpoint continuous transmission mechanism is a result obtained through an algorithm;
the automatic expansion steps of the white list are as follows:
s7, the material downloading system needs to download the material files, the material files are on the Internet, the material downloading system is in an in-line local area network, the material downloading system needs to go through a proxy service to an out-of-line material downloading transfer service, the in-line white list only needs to be configured with an out-of-line material downloading transfer service ip, the transfer service can be communicated with the Internet, and therefore the material downloading system directly goes through the material downloading transfer service to obtain the material files which the task wants to download, and the problem of automatic expansion of the white list can be perfectly solved;
the automatic expansion of the disk comprises the following steps:
s8, customer feed material;
s9, writing a material downloading task table by the special staging system;
s10, the material downloading system reads the material downloading task to download;
s11, the material downloading system reads the disk identifier of the stored disk and generates a file address;
s12, the material downloading system keeps the downloading file to the disk address generated in the last step;
s13, judging whether the file downloading by the material downloading system is successful, if not, setting the material downloading table to change the material task to a failure state and entering a material downloading task waiting for the next reading failure, if so, saving the address of the previous step to the material table and carrying out the next step;
s14, the single material download is finished;
s15, the bank manager previews the material file on the web page;
s16, the web browser requests the special staging system according to the address of the material file;
s17, the special staging system reads the corresponding disk material file according to the drive symbol information contained in the material address and returns the corresponding disk material file to the web browser;
s18, the bank manager successfully previews the material file.
2. The method according to claim 1, wherein the algorithm for reading the stored drive letter in S11 is specifically:
I. storing the drive letter keywords, the drive letter path and the drive letter state into a database;
II. Reading the drive letter and the drive letter path in the in-use state;
III, returning the drive letter path to a material downloading task;
the I records stored in the database can be multiple, namely configuration information of dynamic expansion of the disk, when the last disk is full and a disk is newly added, a record can be newly added to the disk configuration table, the drive letter path and the drive letter key words of the record are information of the newly added disk, the drive letter state is in use, and other drive letters are marked as used.
3. The method according to claim 1, wherein the algorithm for reading the corresponding disk material file in S17 specifically includes:
a. acquiring a material address, wherein the address rule is as follows: d { N }/xxx/yy.zz, wherein D is a fixed value, N is a variable representing that the current block of disk is the second block of disk, x is a file path, y is a file name, and z is a file suffix;
b. reading disc character information, e.g., D1;
c. reading a corresponding disk path according to the drive letter information;
d. splicing disk paths and file paths, such as/data/images/D1/xxx/yy.zz;
and c, reading the disk label information of the disk label key word corresponding to the disk configuration table.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010935899.1A CN112087512A (en) | 2020-09-08 | 2020-09-08 | Method for automatically expanding material downloading task |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010935899.1A CN112087512A (en) | 2020-09-08 | 2020-09-08 | Method for automatically expanding material downloading task |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112087512A true CN112087512A (en) | 2020-12-15 |
Family
ID=73732152
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010935899.1A Pending CN112087512A (en) | 2020-09-08 | 2020-09-08 | Method for automatically expanding material downloading task |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112087512A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150074168A1 (en) * | 2013-06-19 | 2015-03-12 | Hitachi Data Systems Engineering UK Limited | Decentralized distributed computing system |
| CN107515795A (en) * | 2017-09-08 | 2017-12-26 | 北京京东尚科信息技术有限公司 | Multi-task parallel data processing method, device, medium and equipment based on queue |
| CN110737534A (en) * | 2019-10-29 | 2020-01-31 | 京东数字科技控股有限公司 | Task processing method and device and server |
| CN113553346A (en) * | 2021-07-22 | 2021-10-26 | 中国电子科技集团公司第十五研究所 | Large-scale real-time data stream integrated processing, forwarding and storing method and system |
-
2020
- 2020-09-08 CN CN202010935899.1A patent/CN112087512A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150074168A1 (en) * | 2013-06-19 | 2015-03-12 | Hitachi Data Systems Engineering UK Limited | Decentralized distributed computing system |
| CN107515795A (en) * | 2017-09-08 | 2017-12-26 | 北京京东尚科信息技术有限公司 | Multi-task parallel data processing method, device, medium and equipment based on queue |
| CN110737534A (en) * | 2019-10-29 | 2020-01-31 | 京东数字科技控股有限公司 | Task processing method and device and server |
| CN113553346A (en) * | 2021-07-22 | 2021-10-26 | 中国电子科技集团公司第十五研究所 | Large-scale real-time data stream integrated processing, forwarding and storing method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8131276B2 (en) | Method for extracting content, content extraction server based on RSS and apparatus for managing the same and system for providing standby screen of mobile communication terminal using the same | |
| US6606525B1 (en) | System and method of merging static data in web pages | |
| US7492365B2 (en) | Method and system for font building | |
| US6961737B2 (en) | Serving signals | |
| CN100447787C (en) | Methods, systems, and computer program products for client side prefetching and caching of portlets | |
| US9462038B2 (en) | Methods for seamlessly providing content to a client system and devices thereof | |
| US8321875B2 (en) | Selectively updating web pages on a mobile client | |
| US20040201618A1 (en) | Streaming of real-time data to a browser | |
| CN104714965B (en) | Static resource De-weight method, static resource management method and device | |
| US20190205056A1 (en) | Transparent data movement between a private cloud and storage ecosystem and another storage system | |
| GB2371896A (en) | Web Server | |
| JP2009199606A (en) | System and method for stateful web-based computing | |
| CN105787091B (en) | Support the content delivering system and method for multiple terminals | |
| CN110784498B (en) | Personalized data disaster tolerance method and device | |
| CN112131504B (en) | Webpage editing and displaying method, device, equipment and storage medium | |
| US10733068B1 (en) | Method and system for implementing a digital application architecture with distinct processing lanes | |
| WO2010144351A1 (en) | Electronic paper display whitespace utilization | |
| KR20070028014A (en) | Method and system for individuality service on webpage | |
| CN108763500A (en) | Voice-based Web browser method, device, equipment and storage medium | |
| CN110503545A (en) | Loan is independently into part method, terminal device, storage medium and device | |
| CN112087512A (en) | Method for automatically expanding material downloading task | |
| JP6924544B2 (en) | Cartoon data display system, method and program | |
| CN115495513A (en) | Event standardization method and device | |
| US20210092199A1 (en) | Cachability of single page applications | |
| CN111045614B (en) | Method for printing running water report, method for generating overprint file, client and server |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |