CN112269672A - File download exception handling method and device and electronic equipment - Google Patents
File download exception handling method and device and electronic equipment Download PDFInfo
- Publication number
- CN112269672A CN112269672A CN202011166383.1A CN202011166383A CN112269672A CN 112269672 A CN112269672 A CN 112269672A CN 202011166383 A CN202011166383 A CN 202011166383A CN 112269672 A CN112269672 A CN 112269672A
- Authority
- CN
- China
- Prior art keywords
- downloading
- file
- download
- abnormal
- tasks
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 38
- 230000002159 abnormal effect Effects 0.000 claims abstract description 75
- 230000005856 abnormality Effects 0.000 abstract description 4
- 238000010586 diagram Methods 0.000 description 8
- 230000000694 effects Effects 0.000 description 4
- 238000004590 computer program Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0706—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment
- G06F11/0715—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment in a system implementing multitasking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0751—Error or fault detection not based on redundancy
- G06F11/0754—Error or fault detection not based on redundancy by exceeding limits
- G06F11/0757—Error or fault detection not based on redundancy by exceeding limits by exceeding a time limit, i.e. time-out, e.g. watchdogs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0766—Error or fault reporting or storing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0766—Error or fault reporting or storing
- G06F11/0787—Storage of error reports, e.g. persistent data storage, storage using memory protection
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a file downloading exception handling method, a device and electronic equipment, wherein the method comprises the following steps: storing a downloading task corresponding to the ID of the file to be downloaded into a downloading queue; calling a distributed task scheduling system to read a plurality of downloading tasks from the downloading queue and distributing the downloading tasks to a plurality of machines to be executed simultaneously; processing abnormal information of file downloading and reading in parallel by adopting a plurality of threads, and storing the abnormal information of the downloading tasks executed by the machines into a database; and acquiring the heartbeat of the thread with abnormal execution according to the abnormal information, and sending a download abnormal report. According to the invention, a plurality of download tasks are distributed to a plurality of machines to be executed simultaneously through the distributed task scheduling system, a plurality of threads are adopted to capture various abnormal information of file reading and downloading in real time, and the abnormal information of file downloading and reading is processed in parallel, so that the abnormality in file downloading or reading is compensated in time, and the data loss is reduced.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a file downloading exception handling method and device, electronic equipment and a computer readable medium.
Background
In the running process of a system (such as an operating system, a software system, etc.), the running environment of the system and the system itself may affect the running of the system, and for example, hardware resources, hardware working conditions, network communication, system errors, system running defects, environmental temperature, etc. may cause system running abnormalities, thereby causing a downtime situation.
For the list pushed from the external system (such as some apps), during the process of reading or downloading the file, if the server is suddenly down, the client may cause data loss, thereby causing an abnormal data downloading situation.
In the prior art, data downloading abnormal information needs to be acquired from a database at regular time, and the reason of the downloading abnormal information cannot be timely and accurately positioned due to the time delay of the abnormal information, so that lost data cannot be compensated, and the integrity of the downloaded data is influenced.
Disclosure of Invention
The invention aims to solve the technical problem that abnormal downloading data is lost and is difficult to compensate due to the fact that data downloading abnormality cannot be located in time in the file downloading process.
In order to solve the above technical problem, a first aspect of the present invention provides a method for processing file download exception, where the method includes:
receiving an ID of a file to be downloaded sent by an external system;
storing the downloading task corresponding to the ID of the file to be downloaded into a downloading queue;
calling a distributed task scheduling system to read a plurality of downloading tasks from the downloading queue and distributing the downloading tasks to a plurality of machines to be executed simultaneously;
processing abnormal information of file downloading and reading in parallel by adopting a plurality of threads, and storing the abnormal information of the downloading tasks executed by the machines into a database;
and acquiring the heartbeat of the thread with abnormal execution according to the abnormal information, and sending a download abnormal report.
According to a preferred embodiment of the present invention, the allocating the plurality of download tasks to a plurality of machines to be executed simultaneously comprises:
evenly distributing the plurality of download tasks to the plurality of machines;
distributing the residual number of downloading tasks to the specified machine;
and controlling the plurality of machines to simultaneously execute the downloading task.
According to a preferred embodiment of the present invention, the processing, by using multiple threads, of the exception information of file downloading and reading in parallel includes:
if the file downloading of a certain machine fails, controlling the machine to re-download the file which fails in downloading;
if the file downloading of a certain machine is successful but the file reading is failed, controlling the machine to read the downloaded file again from the next line of the current reading line number.
According to a preferred embodiment of the present invention, the storing the exception information of the plurality of machines performing the download task into the database includes:
if the file downloading of a certain machine fails, storing the ID of the machine, the ID of the downloaded file and the downloading time into a database;
if the file downloading of a certain machine is successful but the file reading is failed, storing the ID of the machine, the ID of the downloaded file, the current reading line number and the downloading time into a database;
if the file downloading of a certain machine is successful and the file reading is successful, the ID of the downloaded file in the ID of the machine in the database is cleared.
According to a preferred embodiment of the present invention, the obtaining a heartbeat of a thread with an execution exception according to the exception information and sending a download exception report includes:
acquiring the downloading time in the current abnormal information from the database;
judging whether the heartbeat of the thread executing the corresponding downloading task is abnormal or not according to the current time and the downloading time;
and if the thread heartbeat is abnormal, sending an abnormal report.
According to a preferred embodiment of the invention, the distributed task scheduling system is any one of xxl-joba, cronsun, Elastic-joba, saturn and TBSchedule.
According to a preferred embodiment of the present invention, the database is any one of Redis, Oracle, Sybase, and Informix.
In order to solve the above technical problem, a second aspect of the present invention provides a file download exception handling apparatus, including:
the receiving module is used for receiving the ID of the file to be downloaded sent by the external system;
the first storage module is used for storing the downloading task corresponding to the ID of the file to be downloaded into a downloading queue;
the distribution module is used for calling the distributed task scheduling system to read a plurality of downloading tasks from the downloading queue and distributing the plurality of downloading tasks to a plurality of machines to be executed simultaneously;
the second storage module is used for processing the abnormal information of file downloading and reading in parallel by adopting a plurality of threads and storing the abnormal information of the downloading tasks executed by the machines into a database;
and the sending module is used for acquiring the thread heartbeat with abnormal execution according to the abnormal information and sending a download abnormal report.
According to a preferred embodiment of the invention, the distribution module comprises:
a first distribution module for distributing the plurality of download tasks to the plurality of machines on average;
the second distribution module is used for distributing the residual number of the downloading tasks to the specified machine;
and the control module is used for controlling the machines to simultaneously execute the downloading tasks.
According to a preferred embodiment of the present invention, the second storage module includes: the exception handling module is used for controlling a certain machine to re-download the file which fails to download if the file of the machine fails to download; if the file downloading of a certain machine is successful but the file reading is failed, controlling the machine to read the downloaded file again from the next line of the current reading line number.
According to a preferred embodiment of the present invention, the second storage module further comprises: the sub-storage module is used for storing the ID of a certain machine, the ID of a downloaded file and the downloading time into a database if the file downloading of the machine fails; if the file downloading of a certain machine is successful but the file reading is failed, storing the ID of the machine, the ID of the downloaded file, the current reading line number and the downloading time into a database; if the file downloading of a certain machine is successful and the file reading is successful, the ID of the downloaded file in the ID of the machine in the database is cleared.
According to a preferred embodiment of the present invention, the sending module includes:
the acquisition module is used for acquiring the downloading time in the current abnormal information from the database;
the judging module is used for judging whether the heartbeat of the thread executing the corresponding downloading task is abnormal or not according to the current time and the downloading time;
and the sub-sending module is used for sending an exception report if the heartbeat of the thread is abnormal.
According to a preferred embodiment of the invention, the distributed task scheduling system is any one of xxl-joba, cronsun, Elastic-joba, saturn and TBSchedule.
According to a preferred embodiment of the present invention, the database is any one of Redis, Oracle, Sybase, and Informix.
To solve the above technical problem, a third aspect of the present invention provides an electronic device, comprising:
a processor; and
a memory storing computer executable instructions that, when executed, cause the processor to perform the method described above.
To solve the above technical problems, a fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs which, when executed by a processor, implement the above method.
The method combines a distributed task scheduling system with multi-thread downloading, reads a plurality of downloading tasks from a downloading queue through the distributed task scheduling system, distributes the plurality of downloading tasks to a plurality of machines to be executed simultaneously, adopts a plurality of threads to capture various abnormal information of file reading and downloading in real time, and processes the abnormal information of file downloading and reading in parallel, thereby realizing timely compensation of abnormality in file downloading or reading and effectively reducing data loss caused by abnormal data downloading. Simultaneously storing the abnormal information of the plurality of thread downloading tasks to a database; and acquiring the heartbeat of the thread with abnormal execution according to the abnormal information, sending a download abnormal report, recording the information of the abnormal thread, and facilitating subsequent repair.
Drawings
In order to make the technical problems solved by the present invention, the technical means adopted and the technical effects obtained more clear, the following will describe in detail the embodiments of the present invention with reference to the accompanying drawings. It should be noted, however, that the drawings described below are only illustrations of exemplary embodiments of the invention, from which other embodiments can be derived by those skilled in the art without inventive step.
FIG. 1 is a flowchart illustrating a file download exception handling method according to the present invention;
FIG. 2 is a schematic diagram of file download exception handling according to the present invention;
FIG. 3 is a schematic diagram of a structural framework of a file download exception handling apparatus according to the present invention;
FIG. 4 is a block diagram of an exemplary embodiment of an electronic device in accordance with the present invention;
FIG. 5 is a schematic diagram of one embodiment of a computer-readable medium of the present invention.
Detailed Description
Exemplary embodiments of the present invention will now be described more fully hereinafter with reference to the accompanying drawings, in which exemplary embodiments of the invention may be embodied in many specific forms, and should not be construed as limited to the embodiments set forth herein. Rather, these exemplary embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the invention to those skilled in the art.
The structures, properties, effects or other characteristics described in a certain embodiment may be combined in any suitable manner in one or more other embodiments, while still complying with the technical idea of the invention.
In describing particular embodiments, specific details of structures, properties, effects, or other features are set forth in order to provide a thorough understanding of the embodiments by one skilled in the art. However, it is not excluded that a person skilled in the art may implement the invention in a specific case without the above-described structures, performances, effects or other features.
The flow chart in the drawings is only an exemplary flow demonstration, and does not represent that all the contents, operations and steps in the flow chart are necessarily included in the scheme of the invention, nor does it represent that the execution is necessarily performed in the order shown in the drawings. For example, some operations/steps in the flowcharts may be divided, some operations/steps may be combined or partially combined, and the like, and the execution order shown in the flowcharts may be changed according to actual situations without departing from the gist of the present invention.
The block diagrams in the figures generally represent functional entities and do not necessarily correspond to physically separate entities. I.e. these functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor means and/or microcontroller means.
The same reference numerals denote the same or similar elements, components, or parts throughout the drawings, and thus, a repetitive description thereof may be omitted hereinafter. It will be further understood that, although the terms first, second, third, etc. may be used herein to describe various elements, components, or sections, these elements, components, or sections should not be limited by these terms. That is, these phrases are used only to distinguish one from another. For example, a first device may also be referred to as a second device without departing from the spirit of the present invention. Furthermore, the term "and/or", "and/or" is intended to include all combinations of any one or more of the listed items.
Referring to fig. 1, fig. 1 is a flowchart illustrating a file download exception handling method according to the present invention. As shown in fig. 1, the method includes:
s1, receiving the ID of the file to be downloaded sent by the external system;
wherein the external system may be a business system, such as: a financial loan service system, an insurance service system, an acquisition system, a logistics service system and the like; or some APP application. In this embodiment, the external system may call the list push service of the list system through the http interface, and the external system may provide a plurality of IDs of files to be downloaded.
S2, storing the downloading task corresponding to the ID of the file to be downloaded into a downloading queue;
wherein, one file ID to be downloaded corresponds to the unique downloading task. The downloading task comprises the following steps: the ID of the file to be downloaded, the downloading mode, the request serial number, the number of the ID of the file to be downloaded and the like.
In order to prevent repeated submission of the task to be downloaded, the downloading task is subjected to idempotent processing according to the request serial number. And storing the downloading tasks processed by the idempotent process into a downloading queue, wherein the downloading queue can be a redis queue.
S3, calling a distributed task scheduling system to read a plurality of downloading tasks from the downloading queue, and distributing the downloading tasks to a plurality of machines to be executed simultaneously;
the distributed task scheduling system can be any one of xxl-joba, cronsun, Elastic-joba, saturn and TBschedule. The invention preferably selects xxl-joba, and then the step acquires at least one download task needing downloading from the redis queue through the segment broadcasting strategy of xxljoba.
As shown in fig. 2, for a plurality of download tasks, xxljoba distributes the plurality of download tasks to a plurality of machines on average; distributing the residual number of downloading tasks to the specified machine; and controlling the plurality of machines to simultaneously execute the downloading task. The number of the equally distributed download tasks is the download task summary/machine number. For example, if 56 download tasks are allocated to 5 machines, 11 download tasks are equally allocated to the 5 machines, and the remaining 1 download task is allocated to a designated machine, such as the first machine. After the downloading task is distributed, controlling the 5 machines to simultaneously execute the downloading task.
S4, processing the abnormal information of file downloading and reading in parallel by adopting a plurality of threads, and storing the abnormal information of the downloading tasks executed by the machines into a database;
in this step, each thread downloads and reads the file from the fileSource server according to the file ID to be downloaded corresponding to the download task, and processes the abnormal information of file downloading and reading in parallel.
Wherein the anomaly information includes: file download fails, file download succeeds but file read fails. If the file downloading of a certain machine fails, controlling the machine to re-download the file which fails in downloading; if the file downloading of a certain machine is successful but the file reading is failed, controlling the machine to read the downloaded file again from the next line of the current reading line number. By the method, the reason for the file downloading abnormity can be found in time, and corresponding measures can be taken in time to make up, so that data loss caused by data downloading abnormity is reduced.
Further, if the file downloading of a certain machine fails, the ID of the machine, the ID of the downloaded file and the downloading time are stored in a database; if the file downloading of a certain machine is successful but the file reading is failed, the ID of the machine, the ID of the downloaded file, the current reading line number and the downloading time are stored in the database. In the invention, ten thousand behaviors are used as a batch, and when reading fails, the current reading line number is recorded and stored. If the file downloading of a certain machine is successful and the file reading is successful, the ID of the downloaded file in the ID of the machine in the database is cleared. As shown in fig. 2, the downloading task is distributed to the machine 1, the machine 2 and the machine 3, if the machine 1 fails to download the file, the machine 1 is controlled to download the file again, and the ID of the machine 1, the ID of the downloaded file and the downloading time are stored in the database; if the file downloading of the machine 2 is successful and the file reading is successful, the ID of the downloaded file in the ID of the machine 2 in the database is cleared. If the file downloading of the machine 3 is successful but the file reading is failed, controlling the machine 3 to read the downloaded file from the next line of the current reading line number again, and simultaneously storing the ID of the machine 3, the ID of the downloaded file, the current reading line number and the downloading time in the database.
Wherein, the database can be any one of Redis, Oracle, Sybase and Informmix. The database in the present invention is preferably a Redis database. According to the invention, the Redis database stores and downloads the abnormal information, and whether the thread heartbeat processing is abnormal can be judged through the abnormal information.
And S5, acquiring the thread heartbeat with abnormal execution according to the abnormal information, and sending a download abnormal report.
Specifically, the method comprises the following steps:
s501, acquiring the downloading time in the current abnormal information from the database;
in this step, the download time in the current abnormal information is obtained from the database in real time. The downloading time refers to a time point when the downloading task is abnormal.
S502, judging whether the heartbeat of the thread executing the corresponding downloading task is abnormal or not according to the current time and the downloading time;
specifically, if the time interval between the current time and the download time is greater than the preset time interval, it is determined that the thread that executes the download task corresponding to the download time is abnormal in heartbeat. Wherein the preset time interval is preferably 30 minutes.
And S503, if the heartbeat of the thread is abnormal, sending an abnormal report.
Specifically, a report of thread heartbeat exception may be sent to the download manager; the report of thread heartbeat exception can also be sent in a log downloading mode.
Further, the method may further include:
and pushing the downloaded and read file and the list information to a downstream system.
Fig. 3 is a schematic structural diagram of a file download exception handling apparatus according to the present invention, as shown in fig. 3, the apparatus includes:
a receiving module 31, configured to receive an ID of a file to be downloaded sent by an external system;
the first storage module 32 is configured to store the download task corresponding to the to-be-downloaded file ID into a download queue;
the distribution module 33 is configured to invoke the distributed task scheduling system to read a plurality of download tasks from the download queue, and distribute the plurality of download tasks to a plurality of machines to be executed simultaneously;
the second storage module 34 is configured to use multiple threads to process file downloading and read exception information in parallel, and store the exception information of the multiple machines executing downloading tasks in the database;
and the sending module 35 is configured to obtain a thread heartbeat with an execution exception according to the exception information, and send a download exception report.
In one embodiment, the allocation module 33 includes:
a first distribution module 331 configured to distribute the plurality of download tasks to the plurality of machines on average;
a second assignment module 332 for assigning the remaining number of download tasks to the specified machine;
a control module 333, configured to control the multiple machines to execute the download task at the same time.
The second storage module 34 includes:
an exception handling module 341, configured to, if a file of a certain machine fails to be downloaded, control the machine to re-download the file that fails to be downloaded; if the file downloading of a certain machine is successful but the file reading is failed, controlling the machine to read the downloaded file again from the next line of the current reading line number.
The sub-storage module 342 is configured to, if a file download of a certain machine fails, store the ID of the certain machine, the ID of the downloaded file, and the download time in the database; if the file downloading of a certain machine is successful but the file reading is failed, storing the ID of the machine, the ID of the downloaded file, the current reading line number and the downloading time into a database; if the file downloading of a certain machine is successful and the file reading is successful, the ID of the downloaded file in the ID of the machine in the database is cleared.
The sending module 35 includes:
an obtaining module 351, configured to obtain, from the database, the download time in the current abnormal information;
a judging module 352, configured to judge whether a heartbeat of a thread executing a corresponding download task is abnormal according to the current time and the download time;
and the sub-sending module 353 is configured to send an exception report if the thread heartbeat is abnormal.
In the invention, the distributed task scheduling system is any one of xxl-joba, cronsun, Elastic-joba, saturn and TBschedule. The database is any one of Redis, Oracle, Sybase and Informmix.
Those skilled in the art will appreciate that the modules in the above-described embodiments of the apparatus may be distributed as described in the apparatus, and may be correspondingly modified and distributed in one or more apparatuses other than the above-described embodiments. The modules of the above embodiments may be combined into one module, or further split into multiple sub-modules.
In the following, embodiments of the electronic device of the present invention are described, which may be regarded as an implementation in physical form for the above-described embodiments of the method and apparatus of the present invention. Details described in the embodiments of the electronic device of the invention should be considered supplementary to the embodiments of the method or apparatus described above; for details which are not disclosed in embodiments of the electronic device of the invention, reference may be made to the above-described embodiments of the method or the apparatus.
Fig. 4 is a block diagram of an exemplary embodiment of an electronic device according to the present invention. The electronic device shown in fig. 4 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 4, the electronic device 400 of the exemplary embodiment is represented in the form of a general-purpose data processing device. The components of electronic device 400 may include, but are not limited to: at least one processing unit 410, at least one memory unit 420, a bus 430 connecting different electronic device components (including the memory unit 420 and the processing unit 410), a display unit 440, and the like.
The storage unit 420 stores a computer-readable program, which may be a code of a source program or a read-only program. The program may be executed by the processing unit 410 such that the processing unit 410 performs the steps of various embodiments of the present invention. For example, the processing unit 410 may perform the steps as shown in fig. 1.
The storage unit 420 may include readable media in the form of volatile storage units, such as a random access memory unit (RAM)4201 and/or a cache memory unit 4202, and may further include a read only memory unit (ROM) 4203. The storage unit 420 may also include a program/utility 4204 having a set (at least one) of program modules 4205, such program modules 4205 including, but not limited to: operating the electronic device, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.
The electronic device 400 may also communicate with one or more external devices 300 (e.g., keyboard, display, network device, bluetooth device, etc.), enable a user to interact with the electronic device 400 via the external devices 300, and/or enable the electronic device 400 to communicate with one or more other data processing devices (e.g., router, modem, etc.). Such communication may occur via input/output (I/O) interfaces 450, and may also occur via a network adapter 460 with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN) and/or a public network such as the Internet). The network adapter 460 may communicate with other modules of the electronic device 400 via the bus 430. It should be appreciated that although not shown in FIG. 4, other hardware and/or software modules may be used in the electronic device 400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID electronics, tape drives, and data backup storage electronics, among others.
FIG. 5 is a schematic diagram of one computer-readable medium embodiment of the present invention. As shown in fig. 5, the computer program may be stored on one or more computer readable media. The computer readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electronic device, apparatus, or device that is electronic, magnetic, optical, electromagnetic, infrared, or semiconductor, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. The computer program, when executed by one or more data processing devices, enables the computer-readable medium to implement the above-described method of the invention, namely: receiving an ID of a file to be downloaded sent by an external system; storing the downloading task corresponding to the ID of the file to be downloaded into a downloading queue; calling a distributed task scheduling system to read a plurality of downloading tasks from the downloading queue and distributing the downloading tasks to a plurality of machines to be executed simultaneously; processing abnormal information of file downloading and reading in parallel by adopting a plurality of threads, and storing the abnormal information of the downloading tasks executed by the machines into a database; and acquiring the heartbeat of the thread with abnormal execution according to the abnormal information, and sending a download abnormal report.
Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments of the present invention described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a computer-readable storage medium (which can be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to make a data processing device (which can be a personal computer, a server, or a network device, etc.) execute the above-mentioned method according to the present invention.
The computer readable storage medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable storage medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution electronic device, apparatus, or device. Program code embodied on a readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations of the present invention may be written in any combination of one or more programming languages, including object oriented programming languages such as Java, C + + or the like and conventional procedural programming languages, such as "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).
In summary, the present invention can be implemented as a method, an apparatus, an electronic device, or a computer-readable medium executing a computer program. Some or all of the functions of the present invention may be implemented in practice using a general purpose data processing device such as a microprocessor or a Digital Signal Processor (DSP).
While the foregoing embodiments have described the objects, aspects and advantages of the present invention in further detail, it should be understood that the present invention is not inherently related to any particular computer, virtual machine or electronic device, and various general-purpose machines may be used to implement the present invention. The invention is not to be considered as limited to the specific embodiments thereof, but is to be understood as being modified in all respects, all changes and equivalents that come within the spirit and scope of the invention.
Claims (10)
1. A file download exception handling method is characterized by comprising the following steps:
receiving an ID of a file to be downloaded sent by an external system;
storing the downloading task corresponding to the ID of the file to be downloaded into a downloading queue;
calling a distributed task scheduling system to read a plurality of downloading tasks from the downloading queue and distributing the downloading tasks to a plurality of machines to be executed simultaneously;
processing abnormal information of file downloading and reading in parallel by adopting a plurality of threads, and storing the abnormal information of the downloading tasks executed by the machines into a database;
and acquiring the heartbeat of the thread with abnormal execution according to the abnormal information, and sending a download abnormal report.
2. The method of claim 1, wherein the distributing the plurality of download tasks to a plurality of machines for simultaneous execution comprises:
evenly distributing the plurality of download tasks to the plurality of machines;
distributing the residual number of downloading tasks to the specified machine;
and controlling the plurality of machines to simultaneously execute the downloading task.
3. The method according to any one of claims 1-2, wherein the processing exception information for file downloading and reading in parallel by using multiple threads comprises:
if the file downloading of a certain machine fails, controlling the machine to re-download the file which fails in downloading;
if the file downloading of a certain machine is successful but the file reading is failed, controlling the machine to read the downloaded file again from the next line of the current reading line number.
4. The method of any of claims 1-3, wherein storing exception information for the plurality of machines performing the download task to a database comprises:
if the file downloading of a certain machine fails, storing the ID of the machine, the ID of the downloaded file and the downloading time into a database;
if the file downloading of a certain machine is successful but the file reading is failed, storing the ID of the machine, the ID of the downloaded file, the current reading line number and the downloading time into a database;
if the file downloading of a certain machine is successful and the file reading is successful, the ID of the downloaded file in the ID of the machine in the database is cleared.
5. The method according to any one of claims 1-4, wherein the obtaining a thread heartbeat with an execution exception according to the exception information and sending a download exception report comprises:
acquiring the downloading time in the current abnormal information from the database;
judging whether the heartbeat of the thread executing the corresponding downloading task is abnormal or not according to the current time and the downloading time;
and if the thread heartbeat is abnormal, sending an abnormal report.
6. The method according to any one of claims 1 to 5, wherein the distributed task scheduling system is any one of xxl-joba, cronsun, Elastic-joba, saturn and TBschedule.
7. The method according to any of claims 1-6, wherein the database is any of Redis, Oracle, Sybase, Informix.
8. An apparatus for handling file download exceptions, the apparatus comprising:
the receiving module is used for receiving the ID of the file to be downloaded sent by the external system;
the first storage module is used for storing the downloading task corresponding to the ID of the file to be downloaded into a downloading queue;
the distribution module is used for calling the distributed task scheduling system to read a plurality of downloading tasks from the downloading queue and distributing the plurality of downloading tasks to a plurality of machines to be executed simultaneously;
the second storage module is used for processing the abnormal information of file downloading and reading in parallel by adopting a plurality of threads and storing the abnormal information of the downloading tasks executed by the machines into a database;
and the sending module is used for acquiring the thread heartbeat with abnormal execution according to the abnormal information and sending a download abnormal report.
9. An electronic device, comprising:
a processor; and
a memory storing computer-executable instructions that, when executed, cause the processor to perform the method of any of claims 1-7.
10. A computer readable storage medium, wherein the computer readable storage medium stores one or more programs which, when executed by a processor, implement the method of any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011166383.1A CN112269672B (en) | 2020-10-27 | 2020-10-27 | File downloading exception handling method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011166383.1A CN112269672B (en) | 2020-10-27 | 2020-10-27 | File downloading exception handling method and device and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112269672A true CN112269672A (en) | 2021-01-26 |
| CN112269672B CN112269672B (en) | 2023-05-02 |
Family
ID=74342395
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011166383.1A Active CN112269672B (en) | 2020-10-27 | 2020-10-27 | File downloading exception handling method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112269672B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020178232A1 (en) * | 1997-12-10 | 2002-11-28 | Xavier Ferguson | Method of background downloading of information from a computer network |
| CN106302709A (en) * | 2016-08-12 | 2017-01-04 | 杭州东信北邮信息技术有限公司 | A kind of method and system of network file management |
| CN107347092A (en) * | 2017-06-30 | 2017-11-14 | 环球智达科技（北京）有限公司 | The method downloaded for multithreading |
| CN110572422A (en) * | 2018-06-06 | 2019-12-13 | 北京京东尚科信息技术有限公司 | Data downloading method and device |
| CN111741112A (en) * | 2020-06-22 | 2020-10-02 | 中国平安财产保险股份有限公司 | File downloading method, device, equipment and storage medium based on artificial intelligence |
-
2020
- 2020-10-27 CN CN202011166383.1A patent/CN112269672B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020178232A1 (en) * | 1997-12-10 | 2002-11-28 | Xavier Ferguson | Method of background downloading of information from a computer network |
| CN106302709A (en) * | 2016-08-12 | 2017-01-04 | 杭州东信北邮信息技术有限公司 | A kind of method and system of network file management |
| CN107347092A (en) * | 2017-06-30 | 2017-11-14 | 环球智达科技（北京）有限公司 | The method downloaded for multithreading |
| CN110572422A (en) * | 2018-06-06 | 2019-12-13 | 北京京东尚科信息技术有限公司 | Data downloading method and device |
| CN111741112A (en) * | 2020-06-22 | 2020-10-02 | 中国平安财产保险股份有限公司 | File downloading method, device, equipment and storage medium based on artificial intelligence |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112269672B (en) | 2023-05-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11106459B2 (en) | Distributed parallel build system | |
| US8301935B2 (en) | Distributed batch runner | |
| CN109614227B (en) | Task resource allocation method and device, electronic equipment and computer readable medium | |
| US20090013321A1 (en) | Managing virtual computers | |
| CN104731706A (en) | Method and device for test management using distributed computing | |
| CN114610499A (en) | Task scheduling method and device, computer readable storage medium and electronic equipment | |
| CN111026439A (en) | Application program compatibility method, device, equipment and computer storage medium | |
| US20220326927A1 (en) | Abort installation of firmware bundles | |
| CN113805925A (en) | Online upgrading method, device, equipment and medium for distributed cluster management software | |
| CN111295648A (en) | Job management in a data processing system | |
| CN112269672A (en) | File download exception handling method and device and electronic equipment | |
| CN114696888B (en) | Port task processing method, device, equipment and medium of satellite measurement, operation and control system | |
| CN115437766A (en) | Task processing method and device | |
| CN115291928A (en) | Task automatic integration method and device of multiple technology stacks and electronic equipment | |
| US10169076B2 (en) | Distributed batch job promotion within enterprise computing environments | |
| US9298449B2 (en) | Composite program history | |
| CN113312900A (en) | Data verification method and device | |
| CN111666156B (en) | Container arranging method, device, equipment and storage medium | |
| US20230401087A1 (en) | Method and system for automated migration of high performance computing application to serverless platform | |
| CN112925721A (en) | Distributed system testing method and device | |
| CN115827313A (en) | Data recovery method and device, electronic equipment and storage medium | |
| CN113641422A (en) | File generation method and device, electronic equipment and computer readable medium | |
| CN113656208A (en) | Data processing method, device, equipment and storage medium for distributed storage system | |
| CN116643946A (en) | Method, device, equipment and medium for collecting performance data | |
| CN113095919A (en) | Storage method, device and equipment of commodity file and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP01 | Change in the name or title of a patent holder | ||
| CP01 | Change in the name or title of a patent holder |
Address after: 201500 room a3-5588, 58 Fumin Branch Road, Hengsha Township, Chongming District, Shanghai (Shanghai Hengtai Economic Development Zone) Patentee after: Qifu Shuke (Shanghai) Technology Co.,Ltd. Address before: 201500 room a3-5588, 58 Fumin Branch Road, Hengsha Township, Chongming District, Shanghai (Shanghai Hengtai Economic Development Zone) Patentee before: Shanghai Qifu Information Technology Co.,Ltd. |