CN112131188A - Batch file distribution processing method and device - Google Patents
Batch file distribution processing method and device Download PDFInfo
- Publication number
- CN112131188A CN112131188A CN202011051946.2A CN202011051946A CN112131188A CN 112131188 A CN112131188 A CN 112131188A CN 202011051946 A CN202011051946 A CN 202011051946A CN 112131188 A CN112131188 A CN 112131188A
- Authority
- CN
- China
- Prior art keywords
- file
- processing
- processed
- state
- execution
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000009826 distribution Methods 0.000 title claims abstract description 35
- 238000003672 processing method Methods 0.000 title claims abstract description 18
- 238000012545 processing Methods 0.000 claims abstract description 369
- 238000000034 method Methods 0.000 claims abstract description 56
- 230000008569 process Effects 0.000 claims abstract description 35
- 238000012544 monitoring process Methods 0.000 claims abstract description 13
- 238000004590 computer program Methods 0.000 claims description 10
- VYPSYNLAJGMNEJ-UHFFFAOYSA-N silicon dioxide Inorganic materials O=[Si]=O VYPSYNLAJGMNEJ-UHFFFAOYSA-N 0.000 claims description 9
- 239000010453 quartz Substances 0.000 claims description 8
- 238000003860 storage Methods 0.000 claims description 7
- 238000010586 diagram Methods 0.000 description 15
- 230000007246 mechanism Effects 0.000 description 10
- 230000006870 function Effects 0.000 description 6
- 230000002159 abnormal effect Effects 0.000 description 2
- 238000012790 confirmation Methods 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Stored Programmes (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a batch file distribution processing method and a device, wherein the method comprises the following steps: acquiring an execution sequence and an execution state of file processing steps corresponding to each file to be processed; reading a file processing instruction for processing each file to be processed according to the execution sequence of the file processing steps corresponding to each file to be processed; distributing the read file processing instruction for processing each file to be processed to a plurality of batch processing servers, and calling corresponding file processing service programs by the plurality of batch processing servers according to the received file processing instruction to process each file to be processed; and updating the execution sequence and the execution state of the file processing step corresponding to each file to be processed according to the processing result. The invention disassembles a plurality of file processing steps in the file processing process of the file to be processed, and the file processing steps are processed by different batch processing servers in a concurrent manner, thereby greatly improving the file processing efficiency and facilitating the monitoring of the file processing state of each file.
Description
Technical Field
The invention relates to the field of software systems, in particular to a batch file distribution processing method and device.
Background
This section is intended to provide a background or context to the embodiments of the invention that are recited in the claims. The description herein is not admitted to be prior art by inclusion in this section.
A commercial bank, which is an open fund brokering system, needs to transmit client information, transaction information, and the like of an investor to an open fund registered post-consumer (TA) system in a file format and receive a confirmation return file of the TA system. The TA system is an important component of fund transactions for registering fund shares with investors. One commercial bank usually interfaces hundreds of TA systems, so that the number of files interacted by the affiliate system in batches every day is large, and the existing affiliate systems mostly adopt a serial file processing mechanism to process the files, so that the file processing efficiency is low.
In view of the above problems, no effective solution has been proposed.
Disclosure of Invention
The embodiment of the invention provides a batch file distribution processing method, which is used for solving the technical problem of low file processing efficiency caused by the fact that a TA file is processed by a serial file processing mechanism in the existing fund affiliation system, and comprises the following steps: acquiring file processing state information of a plurality of files to be processed, wherein the file processing state information comprises: each file to be processed corresponds to the execution sequence and the execution state of the file processing step; reading a file processing instruction for processing each file to be processed according to an execution sequence of the file processing step corresponding to each file to be processed, wherein the file processing steps with different execution sequences correspond to different file processing instructions; distributing the read file processing instruction for processing each file to be processed to a plurality of batch processing servers, and calling corresponding file processing service programs by the plurality of batch processing servers according to the received file processing instruction to process each file to be processed; and updating the execution sequence and the execution state of the file processing step corresponding to each file to be processed according to the processing result.
The embodiment of the invention also provides a batch file distribution processing device, which is used for solving the technical problem of low file processing efficiency caused by the adoption of a serial file processing mechanism to process the TA file in the existing fund affiliation system, and comprises the following components: the file state acquisition module is used for acquiring file processing state information of a plurality of files to be processed, wherein the file processing state information comprises: each file to be processed corresponds to the execution sequence and the execution state of the file processing step; the instruction reading module is used for reading a file processing instruction for processing each file to be processed according to the execution sequence of the file processing step corresponding to each file to be processed, wherein the file processing steps with different execution sequences correspond to different file processing instructions; the distribution processing module is used for distributing the read file processing instruction for processing each file to be processed to the batch processing servers, and the batch processing servers call the corresponding file processing service program according to the received file processing instruction to process each file to be processed; and the file state updating module is used for updating the execution sequence and the execution state of the file processing step corresponding to each file to be processed according to the processing result.
The embodiment of the invention also provides computer equipment for solving the technical problem of low file processing efficiency caused by the fact that the TA file is processed by adopting a serial file processing mechanism in the existing fund affiliation system.
The embodiment of the invention also provides a computer readable storage medium, which is used for solving the technical problem of low file processing efficiency caused by the fact that the TA file is processed by adopting a serial file processing mechanism in the existing fund affiliation system.
In the embodiment of the invention, the file processing process of each file to be processed is divided into a plurality of file processing steps in advance, different file processing instructions are configured for different file processing steps to call different file processing programs to process each file processing step, after the execution sequence and the execution state of the file processing steps corresponding to each file to be processed are obtained, the file processing instructions for processing each file to be processed are read according to the execution sequence of the file processing steps corresponding to each file to be processed, the read file processing instructions for processing each file to be processed are distributed to a plurality of batch processing servers, the plurality of batch processing servers call corresponding file processing service programs according to the received file processing instructions to process each file to be processed, and finally according to the processing results, compared with the technical scheme that a serial file processing mechanism is adopted to process a large number of files in the prior art, the file processing method and the file processing system have the advantages that the file processing process of the files to be processed is divided into a plurality of file processing steps, and the files are processed by different batch processing servers in a concurrent mode, so that the file processing efficiency can be greatly improved, and the file processing state of each file can be conveniently monitored.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts. In the drawings:
fig. 1 is a flowchart of a batch file distribution processing method provided in an embodiment of the present invention;
FIG. 2 is a flowchart of an alternative batch file distribution processing method provided in an embodiment of the present invention;
FIG. 3 is a schematic diagram illustrating an implementation of a document processing step according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of an instruction dispatch process provided in an embodiment of the present invention;
fig. 5 is a schematic diagram of a batch file distribution processing apparatus provided in an embodiment of the present invention;
fig. 6 is a schematic diagram of an alternative batch file distribution processing apparatus provided in the embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings. The exemplary embodiments and descriptions of the present invention are provided to explain the present invention, but not to limit the present invention.
The embodiment of the present invention provides a method for distributing and processing a batch of files, and fig. 1 is a flowchart of a method for distributing and processing a batch of files provided in the embodiment of the present invention, as shown in fig. 1, the method includes the following steps:
s101, acquiring file processing state information of a plurality of files to be processed, wherein the file processing state information comprises: each file to be processed corresponds to the execution sequence and the execution state of the file processing steps.
It should be noted that, in the embodiment of the present invention, each to-be-processed file may be, but is not limited to, a file interacted between the open fund brokering system and the TA system. In the embodiment of the invention, the file processing process of each file to be processed is divided into a plurality of file processing steps in advance, and different file processing instructions are configured aiming at different file processing steps so as to call different file processing programs to process each file processing step. For example, the functions of service center service registration, service application, service call and the like of the TA file are realized through a plurality of batch processing servers, the batch processing function is efficiently completed, the customer experience is improved, and the processing efficiency is ensured.
It should be noted that, in the embodiment of the present invention, the document processing process of the document to be processed is disassembled, so that each document processing step of the document to be processed can be concurrently processed by different batch processing servers, and the document processing efficiency can be greatly improved. By recording the execution state of each file processing step, it is possible to automatically switch to the next step of the current file processing step in the case where the current file processing step is successfully processed.
In one embodiment, the above S101 may be implemented by the following steps: acquiring a file list to be processed, wherein the file list to be processed comprises: file information of a plurality of files to be processed; and acquiring the file processing state information of each file to be processed in the file list to be processed by adopting a timing polling mode based on the timing task of the Quartz cluster. The embodiment of the invention realizes the scheduling of the batch text processing tasks based on the Quartz framework, and has high availability and flexibility.
S102, reading a file processing instruction for processing each file to be processed according to the execution sequence of the file processing step corresponding to each file to be processed, wherein the file processing steps with different execution sequences correspond to different file processing instructions.
In specific implementation, after determining each file processing step corresponding to the file to be processed, different file processing instructions may be configured for each file processing step, so that the corresponding file processing service program is called by reading the corresponding file processing instruction to process the corresponding file processing step.
S103, distributing the read file processing instructions for processing the files to be processed to a plurality of batch processing servers, and calling corresponding file processing service programs by the plurality of batch processing servers according to the received file processing instructions to process the files to be processed.
In an embodiment, the S103 may distribute the read file processing instruction for processing each file to be processed to a plurality of batch processing servers based on the Dubbo and Zookeeper distributed service framework. In the embodiment of the invention, a Dubbo + Zookeeper micro-service framework is adopted to distribute the file processing instructions of each file processing step to a plurality of batch processing servers in a balanced manner, and the batch processing servers perform parallel processing to complete complex file logic processing, so that the file processing efficiency can be greatly improved, and the customer experience is improved.
And S104, updating the execution sequence and the execution state of the file processing step corresponding to each file to be processed according to the processing result.
In specific implementation, the execution state of the file processing step corresponding to each file to be processed may include: an initial state, a process success state, and a process failure state, as shown in fig. 2, the above S104 may be implemented by the following steps:
s1041, if the file processing step corresponding to the file to be processed is successfully processed, updating the execution state of the file processing step corresponding to the file to be processed from the initial state to a processing success state;
s1042, if the file processing step corresponding to the file to be processed is not successfully processed, updating the execution state of the file processing step corresponding to the file to be processed from the initial state to a processing failure state.
Further, as shown in fig. 2, when the execution state of the current file processing step of the file to be processed is the processing success state, the batch file distribution processing method provided in the embodiment of the present invention may further include the following steps:
s1051, the current file processing step of the file to be processed is switched to the next step of the current file processing step, and the execution state of the file processing step is updated to be the initial state.
Further, as shown in fig. 2, when the execution state of the current file processing step of the file to be processed is a processing failure state, the batch file distribution processing method provided in the embodiment of the present invention may further include the following steps:
s1052, ending the processing of the current file processing step of the file to be processed.
In an embodiment, the batch file distribution processing method provided in the embodiment of the present invention may further include the following steps: monitoring the execution state and instruction calling condition of each file processing step corresponding to each file to be processed; and outputting early warning information according to the monitoring result, wherein the early warning information is used for prompting the file to be processed which fails to be executed or fails to be called by the instruction.
As can be seen from the above, the method for distributing and processing batch files provided in the embodiments of the present invention divides the file processing process of each to-be-processed file into a plurality of file processing steps in advance, configures different file processing instructions for different file processing steps, calls different file processing programs to process each file processing step, reads the file processing instruction for processing each to-be-processed file according to the execution sequence and execution state of the file processing step corresponding to each to-be-processed file after obtaining the execution sequence and execution state of the file processing step corresponding to each to-be-processed file, further distributes the read file processing instruction for processing each to-be-processed file to a plurality of batch processing servers, calls corresponding file processing service programs according to the received file processing instruction by the plurality of batch processing servers, and processing each file to be processed, and finally updating the execution sequence and the execution state of the file processing step corresponding to each file to be processed according to the processing result.
According to the batch file distribution processing method provided by the embodiment of the invention, a plurality of file processing steps are disassembled in the file processing process of the file to be processed, and different batch processing servers concurrently process the file, so that the file processing efficiency can be greatly improved, and the file processing state of each file can be conveniently monitored.
The following describes in detail a batch file distribution processing method according to an embodiment of the present invention, taking a TA file as an example. The method specifically comprises the following steps:
creating a file list which is sent or received as needed, wherein each file is configured with different processing steps and states corresponding to the processing steps.
② register on Quartz cluster to send or receive Job task regularly, polling every two minutes. The Quartz clustering can utilize resources to the maximum extent, and avoids the problems of resource waste and resource contention.
And thirdly, the Job task judges whether the steps are switched or not by reading the steps and the states of the current file, or writes an instruction, and the instruction completes the update of the file state.
Fig. 3 is a schematic diagram illustrating the execution of a file processing step according to an embodiment of the present invention, as shown in fig. 3, when the Job task reads that the current file is in the step 1 — initial state, an instruction with name a (i.e., an a instruction) in the instruction table is written, and assuming that the a instruction is responsible for acquiring an FTP remote file, a service for acquiring the remote file is called, and a call result is updated successfully or unsuccessfully. And (4) the Job task reads the processing success state currently in the step 1, and the Job task rewrites the initial state of the step into the step 2. Other steps are as described above, thus completing the traversal of all steps and states.
And fourthly, the instruction registers service on the Dubbo, the long hanging mechanism reads the instruction and distributes the instruction to different batch processing servers through the Zookeeper server, and the batch processing servers consume the instruction to complete corresponding logic processing. Fig. 4 is a schematic diagram of instruction dispatch processing according to an embodiment of the present invention.
And monitoring the failure condition of the batch processing server based on a double monitoring mechanism called by the file state and the instruction, capturing the abnormal condition by the instruction, accurately positioning to a code level error, monitoring whether the file state fails or not at regular time, and sending a short message to alarm to a maintainer, thereby quickly finding and positioning the abnormal condition.
As can be seen from the above, in the embodiment of the present invention, the processing procedure of the TA file is summarized and arranged into the specific steps and states, the switching of each step is realized based on the timed task of the Quartz cluster, the instruction is distributed to the plurality of batch processing servers in a balanced manner by using the Dubbo + Zookeeper micro service framework, and the batch processing servers complete the complex file logic processing, so that the efficient parallel is realized, the processing rate is greatly improved, and the instruction process is easy to monitor.
Based on the same inventive concept, the embodiment of the present invention further provides a batch file distribution processing apparatus, as described in the following embodiments. Because the principle of solving the problems of the device is similar to the batch file distribution processing method, the implementation of the device can refer to the implementation of the batch file distribution processing method, and repeated parts are not described again.
Fig. 5 is a schematic diagram of a batch file distribution processing apparatus provided in an embodiment of the present invention, and as shown in fig. 5, the apparatus includes: a file status acquisition module 51, an instruction reading module 52, a distribution processing module 53, and a file status update module 54.
The file state obtaining module 51 is configured to obtain file processing state information of multiple files to be processed, where the file processing state information includes: each file to be processed corresponds to the execution sequence and the execution state of the file processing step; the instruction reading module 52 is configured to read a file processing instruction for processing each to-be-processed file according to an execution sequence of the file processing step corresponding to each to-be-processed file, where file processing steps in different execution sequences correspond to different file processing instructions; the distribution processing module 53 is configured to distribute the read file processing instruction for processing each file to be processed to the multiple batch processing servers, and the multiple batch processing servers call corresponding file processing service programs according to the received file processing instruction to process each file to be processed; and the file state updating module 54 is configured to update the execution sequence and the execution state of the file processing step corresponding to each file to be processed according to the processing result.
Optionally, the distribution processing module 53 in the embodiment of the present invention may distribute the read file processing instruction for processing each file to be processed to a plurality of batch processing servers based on a Dubbo and Zookeeper distributed service framework.
As can be seen from the above, the batch file distribution processing apparatus provided in the embodiment of the present invention divides the file processing process of each to-be-processed file into a plurality of file processing steps in advance, configures different file processing instructions for different file processing steps, so as to call different file processing programs to process each file processing step, and after the execution sequence and the execution state of the file processing step corresponding to each to-be-processed file are obtained by the file state obtaining module 51; reading a file processing instruction for processing each file to be processed by the instruction reading module 52 according to the execution sequence of the file processing steps corresponding to each file to be processed; the read file processing instructions for processing the files to be processed are distributed to a plurality of batch processing servers through the distribution processing module 53, and the plurality of batch processing servers call corresponding file processing service programs according to the received file processing instructions to process the files to be processed; finally, the file state updating module 54 updates the execution sequence and the execution state of the file processing step corresponding to each file to be processed according to the processing result.
According to the batch file distribution processing device provided by the embodiment of the invention, a plurality of file processing steps are disassembled in the file processing process of the files to be processed, and the files are concurrently processed by different batch processing servers, so that the file processing efficiency can be greatly improved, and the file processing state of each file can be conveniently monitored.
In an embodiment, in the batch file distribution processing apparatus provided in the embodiment of the present invention, the execution state of the file processing step corresponding to each to-be-processed file may include: as shown in fig. 6, the file status updating module 54 may specifically include: a first state updating module 541, configured to update an execution state of a file processing step currently corresponding to a to-be-processed file from an initial state to a processing success state if the file processing step currently corresponding to the to-be-processed file is successfully processed; the second state updating module 542 is configured to, if the file processing step corresponding to the to-be-processed file is not successfully processed, update the execution state of the file processing step corresponding to the to-be-processed file from the initial state to a processing failure state.
In an embodiment, the batch file distribution processing apparatus provided in the embodiment of the present invention may further include: a step switching module 55, configured to switch the current file processing step of the file to be processed to a next step of the current file processing step and update the execution state of the file processing step to be an initial state when the execution state of the current file processing step of the file to be processed is a processing success state; a step ending processing module 56, configured to end the processing of the current file processing step of the file to be processed when the execution state of the current file processing step of the file to be processed is a processing failure state.
In an embodiment, the file status obtaining module 51 may specifically include: the file list obtaining module 511 is configured to obtain a to-be-processed file list, where the to-be-processed file list includes: file information of a plurality of files to be processed; and the timing polling module 512 is configured to obtain file processing state information of each to-be-processed file in the to-be-processed file list in a timing polling manner based on a timing task of the Quartz cluster.
In an embodiment, the batch file distribution processing apparatus provided in the embodiment of the present invention may further include: the monitoring module 57 is configured to monitor an execution state and an instruction calling condition of each file processing step corresponding to each file to be processed; and the early warning module 58 is configured to output early warning information according to the monitoring result, where the early warning information is used to prompt that the execution fails or an instruction calls a failed file to be processed.
Based on the same inventive concept, the embodiment of the invention also provides a computer device, which is used for solving the technical problem that the existing fund affiliation system adopts a serial file processing mechanism to process the TA file, so that the file processing efficiency is very low.
Based on the same inventive concept, the embodiment of the invention also provides a computer readable storage medium, which is used for solving the technical problem that the existing fund affiliation system adopts a serial file processing mechanism to process the TA file, so that the file processing efficiency is very low.
In summary, embodiments of the present invention provide a method, an apparatus, a computer device, and a computer-readable storage medium for distributing and processing a batch of files, in which a file processing procedure of each to-be-processed file is divided into a plurality of file processing steps in advance, and different file processing instructions are configured for different file processing steps to call different file processing programs to process each file processing step.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above-mentioned embodiments are intended to illustrate the objects, technical solutions and advantages of the present invention in further detail, and it should be understood that the above-mentioned embodiments are only exemplary embodiments of the present invention, and are not intended to limit the scope of the present invention, and any modifications, equivalent substitutions, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (14)
1. A batch file distribution processing method is characterized by comprising the following steps:
acquiring file processing state information of a plurality of files to be processed, wherein the file processing state information comprises: each file to be processed corresponds to the execution sequence and the execution state of the file processing step;
reading a file processing instruction for processing each file to be processed according to an execution sequence of the file processing step corresponding to each file to be processed, wherein the file processing steps with different execution sequences correspond to different file processing instructions;
distributing the read file processing instruction for processing each file to be processed to a plurality of batch processing servers, and calling corresponding file processing service programs by the plurality of batch processing servers according to the received file processing instruction to process each file to be processed;
and updating the execution sequence and the execution state of the file processing step corresponding to each file to be processed according to the processing result.
2. The method of claim 1, wherein the execution state of the file processing step corresponding to each file to be processed comprises: the method comprises an initial state, a processing success state and a processing failure state, wherein the execution sequence and the execution state of the file processing step corresponding to each file to be processed are updated according to a processing result, and the method comprises the following steps:
if the file processing step corresponding to the current file to be processed is successfully processed, updating the execution state of the file processing step corresponding to the current file to be processed from the initial state to a processing success state;
and if the file processing step corresponding to the current file to be processed is not successfully processed, updating the execution state of the file processing step corresponding to the current file to be processed from the initial state to a processing failure state.
3. The method of claim 2, wherein the method further comprises:
when the execution state of the current file processing step of the file to be processed is the processing success state, switching the current file processing step of the file to be processed to be the next step of the current file processing step, and updating the execution state of the file processing step to be the initial state;
and when the execution state of the current file processing step of the file to be processed is the processing failure state, ending the processing of the current file processing step of the file to be processed.
4. The method of claim 1, wherein the read file processing instructions for processing each file to be processed are distributed to a plurality of batch processing servers based on a Dubbo and Zookeeper distributed service framework.
5. The method of claim 1, wherein obtaining file processing state information for a plurality of files to be processed comprises:
acquiring a file list to be processed, wherein the file list to be processed comprises: file information of a plurality of files to be processed;
and acquiring the file processing state information of each file to be processed in the file list to be processed by adopting a timing polling mode based on a timing task of the Quartz cluster.
6. The method of claim 1, wherein the method further comprises:
monitoring the execution state and instruction calling condition of each file processing step corresponding to each file to be processed;
and outputting early warning information according to the monitoring result, wherein the early warning information is used for prompting the file to be processed which fails to be executed or fails to be called by the instruction.
7. A batch file distribution processing apparatus, characterized by comprising:
the file state acquiring module is used for acquiring file processing state information of a plurality of files to be processed, wherein the file processing state information comprises: each file to be processed corresponds to the execution sequence and the execution state of the file processing step;
the instruction reading module is used for reading a file processing instruction for processing each file to be processed according to the execution sequence of the file processing step corresponding to each file to be processed, wherein the file processing steps with different execution sequences correspond to different file processing instructions;
the distribution processing module is used for distributing the read file processing instruction for processing each file to be processed to the batch processing servers, and the batch processing servers call the corresponding file processing service program according to the received file processing instruction to process each file to be processed;
and the file state updating module is used for updating the execution sequence and the execution state of the file processing step corresponding to each file to be processed according to the processing result.
8. The apparatus of claim 7, wherein the execution status of the file processing step corresponding to each file to be processed comprises: the file updating method comprises an initial state, a processing success state and a processing failure state, wherein the file state updating module comprises:
the first state updating module is used for updating the execution state of the file processing step currently corresponding to the file to be processed from the initial state to a processing success state if the file processing step currently corresponding to the file to be processed is successfully processed;
and the second state updating module is used for updating the execution state of the current file processing step corresponding to the file to be processed from the initial state to a processing failure state if the current file processing step corresponding to the file to be processed is not successfully processed.
9. The apparatus of claim 8, wherein the apparatus further comprises:
the step switching module is used for switching the current file processing step of the file to be processed to the next step of the current file processing step under the condition that the execution state of the current file processing step of the file to be processed is a processing success state, and updating the execution state of the file processing step to be an initial state;
and the step ending processing module is used for ending the processing of the current file processing step of the file to be processed under the condition that the execution state of the current file processing step of the file to be processed is a processing failure state.
10. The apparatus of claim 7, wherein the distribution processing module is configured to distribute the read file processing instructions for processing each file to be processed to a plurality of batch processing servers based on a Dubbo and Zookeeper distributed service framework.
11. The apparatus of claim 7, wherein the file status acquisition module comprises:
the file list acquisition module is used for acquiring a file list to be processed, wherein the file list to be processed comprises: file information of a plurality of files to be processed;
and the timing polling module is used for acquiring the file processing state information of each file to be processed in the file list to be processed by adopting a timing polling mode based on a timing task of the Quartz cluster.
12. The apparatus of claim 7, wherein the apparatus further comprises:
the monitoring module is used for monitoring the execution state and the instruction calling condition of each file processing step corresponding to each file to be processed;
and the early warning module is used for outputting early warning information according to the monitoring result, wherein the early warning information is used for prompting the execution failure or instruction calling failure of the file to be processed.
13. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the batch file distribution processing method of any one of claims 1 to 6 when executing the computer program.
14. A computer-readable storage medium characterized by storing a computer program for executing the batch file distribution processing method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011051946.2A CN112131188B (en) | 2020-09-29 | 2020-09-29 | Batch file distribution processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011051946.2A CN112131188B (en) | 2020-09-29 | 2020-09-29 | Batch file distribution processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112131188A true CN112131188A (en) | 2020-12-25 |
| CN112131188B CN112131188B (en) | 2023-11-21 |
Family
ID=73844886
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011051946.2A Active CN112131188B (en) | 2020-09-29 | 2020-09-29 | Batch file distribution processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112131188B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113709208A (en) * | 2021-07-22 | 2021-11-26 | 青岛奥利普自动化控制系统有限公司 | Method for uploading large file to FTP through NET program |
| CN115080509A (en) * | 2022-06-10 | 2022-09-20 | 北京达佳互联信息技术有限公司 | Data processing method and device, electronic equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH07168745A (en) * | 1993-12-16 | 1995-07-04 | Nec Corp | File parallel processor |
| CN101145150A (en) * | 2006-09-15 | 2008-03-19 | 中国银联股份有限公司 | Batch file processing method and system |
| US20100042536A1 (en) * | 2008-08-15 | 2010-02-18 | Tim Thorson | System and method of transferring funds |
| JP2011070318A (en) * | 2009-09-24 | 2011-04-07 | Hitachi Information Systems Ltd | Method of performing batch processing, and system for performing batch processing |
| CN110955469A (en) * | 2019-11-25 | 2020-04-03 | 中国银行股份有限公司 | Method and device for online transaction by X86 platform distributed batch call |
-
2020
- 2020-09-29 CN CN202011051946.2A patent/CN112131188B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH07168745A (en) * | 1993-12-16 | 1995-07-04 | Nec Corp | File parallel processor |
| CN101145150A (en) * | 2006-09-15 | 2008-03-19 | 中国银联股份有限公司 | Batch file processing method and system |
| US20100042536A1 (en) * | 2008-08-15 | 2010-02-18 | Tim Thorson | System and method of transferring funds |
| JP2011070318A (en) * | 2009-09-24 | 2011-04-07 | Hitachi Information Systems Ltd | Method of performing batch processing, and system for performing batch processing |
| CN110955469A (en) * | 2019-11-25 | 2020-04-03 | 中国银行股份有限公司 | Method and device for online transaction by X86 platform distributed batch call |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113709208A (en) * | 2021-07-22 | 2021-11-26 | 青岛奥利普自动化控制系统有限公司 | Method for uploading large file to FTP through NET program |
| CN115080509A (en) * | 2022-06-10 | 2022-09-20 | 北京达佳互联信息技术有限公司 | Data processing method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112131188B (en) | 2023-11-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107783975B (en) | Method and device for synchronous processing of distributed databases | |
| CN109634728B (en) | Job scheduling method and device, terminal equipment and readable storage medium | |
| CN108566290B (en) | Service configuration management method, system, storage medium and server | |
| CN110795503A (en) | Multi-cluster data synchronization method and related device of distributed storage system | |
| CN110750592B (en) | Data synchronization method, device and terminal equipment | |
| CN109298924B (en) | Timing task management method, computer readable storage medium and terminal device | |
| CN112398673A (en) | Upgrading method, communication equipment and computer readable storage medium | |
| CN111125604B (en) | Page management method and device, terminal equipment and storage medium | |
| CN111371898B (en) | Message monitoring method, device, equipment and storage medium | |
| CN112131188A (en) | Batch file distribution processing method and device | |
| US11223522B1 (en) | Context-based intelligent re-initiation of microservices | |
| CN110611707A (en) | Task scheduling method and device | |
| US20200209817A1 (en) | Flexible assembly system for variable reconfiguration, and method of reconfiguring production line | |
| CN111090532B (en) | Application service calling method, device, electronic equipment and computer storage medium | |
| CN111147605A (en) | Service registration method, device and equipment | |
| CN112596871A (en) | Service processing method and device | |
| CN114090198A (en) | Distributed task scheduling method and device, electronic equipment and storage medium | |
| CN111338834B (en) | Data storage method and device | |
| CN107463390B (en) | Software upgrading method and upgrading server | |
| CN113485814A (en) | Batch task scheduling method and device | |
| CN112600842A (en) | Cluster shell method and device, electronic equipment and computer readable storage medium | |
| US11861214B2 (en) | Memory device forensics and preparation | |
| CN112825525A (en) | Method and apparatus for processing transactions | |
| CN110633914A (en) | Batch job scheduling processing method and device | |
| US11537455B2 (en) | Schema management using an event stream |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |