CN110365775B

CN110365775B - Service data uploading method and device and storage medium

Info

Publication number: CN110365775B
Application number: CN201910645888.7A
Authority: CN
Inventors: 王红波; 李志明; 王宝义; 宋金玲; 朱建波; 丁俊辉
Original assignee: Bank of China Ltd
Current assignee: Bank of China Ltd
Priority date: 2019-07-17
Filing date: 2019-07-17
Publication date: 2022-05-20
Anticipated expiration: 2039-07-17
Also published as: CN110365775A

Abstract

An embodiment of the present specification provides a method, an apparatus, and a storage medium for uploading service data, where the method includes: calling an FTP daemon to periodically scan a preset FTP running time table so as to judge whether the current time is the FTP running time; the FTP runtime is a subset of idle periods of a target online transaction system; when the current time is the FTP running time, starting a task confirmation process of a target node to check whether a task to be uploaded exists in a preset FTP task table; and when the FTP task list has the task to be uploaded, starting an FTP file uploading process of the target node to execute the uploading operation of the task to be uploaded. The embodiment of the specification can reduce or avoid resource conflict between the online transaction system and the batch FTP transaction system, and improve the uploading efficiency of the service data.

Description

Service data uploading method and device and storage medium

Technical Field

The present disclosure relates to the field of file transfer control technologies, and in particular, to a method and an apparatus for uploading service data, and a storage medium.

Background

With the continuous large scale of enterprises and the rapid development of internet technology, various IT business systems of enterprises are more and more, and the mutual influence is more and more complex, so that the situation that an online transaction system and a batch FTP (file transfer Protocol) transaction system compete for network bandwidth mutually often occurs, and then the interruption of the online transaction system or the batch transaction system is probably caused, and the normal operation of the enterprise business system is easily influenced.

Therefore, how to make the batch FTP trading system operate more intelligently without changing the existing network configuration and ensuring the normal operation of the online trading system is a technical problem to be solved at present.

Disclosure of Invention

An object of the embodiments of the present disclosure is to provide a method, an apparatus, and a storage medium for uploading service data, so as to reduce or avoid resource conflicts between an online transaction system and a batch FTP transaction system.

In order to achieve the above object, in one aspect, an embodiment of the present specification provides a service data uploading method, including:

calling an FTP daemon to periodically scan a preset FTP running time table so as to judge whether the current time is the FTP running time; the FTP running time is a subset of idle periods of a target online transaction system;

when the current time is the FTP running time, starting a task confirmation process of a target node to check whether a task to be uploaded exists in a preset FTP task table;

and when the FTP task list has the task to be uploaded, starting an FTP file uploading process of the target node to execute the uploading operation of the task to be uploaded.

In another aspect, an embodiment of the present specification provides a service data uploading apparatus, including:

the running time detection module is used for calling an FTP daemon process to periodically scan a preset FTP running time table so as to judge whether the current time is the FTP running time; the FTP running time is a subset of idle periods of a target online transaction system;

the uploading task confirmation module is used for starting a task confirmation process of the target node when the current time is the FTP running time so as to check whether a task to be uploaded exists in a preset FTP task table or not;

and the service file uploading module is used for starting an FTP file uploading process of the target node when the FTP task list has the task to be uploaded so as to execute the uploading operation of the task to be uploaded.

In another aspect, an embodiment of the present specification further provides another service data uploading apparatus, including a memory, a processor, and a computer program stored on the memory, where the computer program, when executed by the processor, performs the following steps:

calling an FTP daemon to periodically scan a preset FTP running time table so as to judge whether the current time is the FTP running time; the FTP runtime is a subset of idle periods of a target online transaction system;

when the current time is the FTP running time, starting a task confirmation process of the target node to check whether a task to be uploaded exists in a preset FTP task table or not;

In another aspect, an embodiment of the present specification provides a computer storage medium having a computer program stored thereon, where the computer program is executed by a processor to implement the following steps:

As can be seen from the technical solutions provided in the embodiments of the present specification, the FTP runtime specified in the FTP runtime table is a subset of the idle period of the online transaction system. Therefore, by the configuration of the FTP running time table, the timed starting and stopping of uploading tasks are realized, and the running time of the batch FTP transaction system is staggered with the service peak period of the online transaction system, so that resource occupation conflict caused by the simultaneous running of the batch FTP transaction system and the online transaction system is effectively reduced or avoided, namely occupation of too many FTP processes on system resources and occupation of too many FTP processes on network bandwidth are avoided, the availability and reliability of the online transaction system are enhanced, and the normal running of the batch FTP transaction system is actually facilitated.

Drawings

In order to more clearly illustrate the embodiments of the present specification or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only some embodiments described in the present specification, and for those skilled in the art, other drawings can be obtained according to the drawings without any creative effort. In the drawings:

fig. 1 is a flowchart of a method of a service data uploading method according to some embodiments of the present disclosure;

fig. 2 is a block diagram illustrating a structure of a service data uploading apparatus according to some embodiments of the present disclosure;

fig. 3 is a block diagram illustrating a structure of a service data uploading apparatus according to another embodiment of the present disclosure.

Detailed Description

In order to make those skilled in the art better understand the technical solutions in the present specification, the technical solutions in the embodiments of the present specification will be clearly and completely described below with reference to the drawings in the embodiments of the present specification, and it is obvious that the described embodiments are only a part of the embodiments of the present specification, and not all of the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present specification without making any creative effort shall fall within the protection scope of the present specification.

In the enterprise business system, the LINUX or UNIX operating system is mostly used for the online transaction system and the batch transaction system, and therefore, the following embodiments of the present specification mainly solve the problem of how to reduce or avoid resource conflicts between the online transaction system and the batch FTP transaction system in the LINUX or UNIX environment.

Referring to fig. 1, a service data uploading method according to some embodiments of the present specification may include the following steps:

s101, calling an FTP daemon to periodically scan a preset FTP running time table so as to judge whether the current time is the FTP running time; the FTP runtime is a subset of an idle period of the target online transaction system.

In some embodiments of the present description, the FTP runtime, as specified in the FTP runtime table, is a subset of the idle periods of the online transaction system. Therefore, the configuration of the FTP running time table not only realizes the timed starting and stopping of uploading tasks, but also staggers the running time of the batch FTP trading system and the service peak period of the online trading system, thereby effectively reducing or avoiding resource occupation conflict caused by the simultaneous running of the batch FTP trading system and the online trading system, namely avoiding the occupation of too many FTP processes on system resources and the occupation of too many FTP processes on network bandwidth, enhancing the availability and reliability of the online trading system, and being beneficial to the normal running of the batch FTP trading system in practice.

In some embodiments of the present description, the FTP runtime table may include FTP job date, FTP job start time, and FTP job end time. The FTP working date may be defined by week or specific date (for example, 12 months and 1 day in 2018). Before setting the FTP work starting time and the FTP work ending time, the idle time of the online transaction system needs to be confirmed. The idle period of the online transaction system refers to a non-service peak period of the online transaction system (e.g., a period from 10 pm to 6 am), and after the idle period of the online transaction system is confirmed, the FTP runtime can be configured according to the FTP runtime table to be configured.

In some embodiments of the present description, the FTP runtime in the FTP runtime table may be inserted through a dynamically configured first SQL script, so that immediate validation and automatic execution of a designated FTP runtime may be implemented, and a system does not need to be restarted, thereby improving efficiency of service data uploading processing.

And S102, when the current time is the FTP running time, starting a task confirmation process of the target node to check whether a task to be uploaded exists in a preset FTP task table.

In some embodiments of the present specification, the target node refers to a node to which traffic data is to be uploaded. For example, in a business system of a financial institution, a batch FTP transaction system may include a head office node and a provincial branch node, wherein the provincial branch node needs to upload business data to be uploaded to the head office node in a batch.

In some embodiments of the present specification, what service data the target node needs to upload may be configured by the FTP task table. In an exemplary embodiment, the FTP task table may include a target node identifier, a file name to be uploaded, a file upload status, and the like. Since the FTP operation time is specified in the FTP operation time table, there may be an exception that some or some files are aborted due to the arrival of the FTP operation end time during the uploading process, and at this time, the file uploading state needs to be recorded so as to continue uploading (i.e. performing breakpoint resuming) when the next FTP operation time arrives. Thus, the tasks to be uploaded in the FTP runtime table may include not only the uploading tasks that are not executed, but also the uploading tasks that are aborted. In addition, the file uploading state can be recorded, so that the progress (or uploading result) of uploading can be conveniently inquired.

In some embodiments of the present specification, the tasks to be uploaded in the FTP task table may be inserted in batch through a second SQL script configured dynamically; therefore, the assigned task to be uploaded can be immediately effective and automatically executed, the system does not need to be restarted, and the efficiency of uploading and processing the service data is improved.

S103, when the FTP task table has the task to be uploaded, starting an FTP file uploading process of the target node to execute the uploading operation of the task to be uploaded.

In some embodiments of the present specification, the FTP file uploading process is an FTP process, and the to-be-uploaded task (i.e., the to-be-uploaded service data) can be uploaded to a specified node by executing the FTP process. After the uploading operation of the task to be uploaded is successfully executed, the uploading state of the task to be uploaded in the FTP task table can be updated, so that uploading can be continued (namely breakpoint resuming) when the next FTP running time comes, and meanwhile, the filename of the uploaded file can be modified to be the real filename.

In some embodiments of the present specification, after step S101, the method may further include: when the current time is the FTP non-running time, an FTP stopping process can be started to stop all processes (such as a task confirmation process, an FTP file uploading process and the like) started by the FTP daemon process in the target process list. Namely, the running FTP process can be closed by calling the FTP stop process, and the running FTP process sleeps for a period of time (the sleeping time is less than the time interval of the periodic scanning); after the sleep is finished, the FTP daemon process can be called to continue polling. In an exemplary embodiment, all processes initiated by the FTP daemon can be found from the system process list by process number.

In some embodiments of this specification, the service data uploading method may further include:

when there are multiple started task confirmation processes, the FTP daemon may be called to periodically scan a preset node-level parallelism configuration table to determine whether the number of currently started task confirmation processes exceeds a task confirmation process parallelism upper limit specified in the node-level parallelism configuration table (i.e., how many target nodes are allowed to upload tasks at the same time). When the number of the currently started task confirmation processes exceeds the task confirmation process parallel upper limit, a specified first number of task confirmation processes can be randomly killed by utilizing a KILL command and the like, so that the problem that the system cannot normally run or even crashes due to overhigh parallelism is avoided.

when a plurality of currently started FTP file uploading processes exist in a single target node, the FTP daemon process can be called to periodically scan a preset file-level parallelism configuration table so as to judge whether the number of the currently started FTP file uploading processes of the target node exceeds the parallel upper limit of the FTP file uploading processes specified in the file-level parallelism configuration table. And when the number of the FTP file uploading processes started currently by the target node exceeds the parallel upper limit of the FTP file uploading processes, randomly killing the FTP file uploading processes with the specified second number so as to avoid that the target node and the system cannot normally operate or even crash due to overhigh parallelism. The file-level parallelism configuration table may include a target node identifier and parallelism corresponding to the target node.

In some embodiments of the present description, the node-level parallelism configuration table and the file-level parallelism configuration table may also be dynamically configured through a script, so that immediate validation and automatic execution are realized without restarting the system, and further, the efficiency of uploading the service data is further improved.

In some embodiments of this specification, since some processes are aborted due to the arrival of the FTP operation end time, which results in the abnormal end of the file being uploaded, when the uploading is started again, the size of the uploaded file needs to be queried, and the uploading is continued from the file size (i.e., breakpoint resuming); and when the process is abnormally terminated, the running position of the program is unknown, so that the return code cannot be defined, therefore, the return code when the process is abnormally terminated can be obtained through a large number of experiments, and meanwhile, because the return codes of different FTP servers have differences, the return codes of various FTP servers can also be processed through a large number of experiments.

While the process flows described above include operations that occur in a particular order, it should be appreciated that the processes may include more or less operations that are performed sequentially or in parallel (e.g., using parallel processors or a multi-threaded environment).

Referring to fig. 2, corresponding to the above-mentioned service data uploading, a service data uploading apparatus according to some embodiments of the present specification may include:

the running time detection module 21 may be configured to call an FTP daemon to periodically scan a preset FTP running time table, so as to determine whether the current time is an FTP running time; the FTP running time is a subset of idle periods of a target online transaction system;

the upload task confirmation module 22 may be configured to start a task confirmation process of the target node when the current time is the FTP operation time, so as to check whether a to-be-uploaded task exists in a preset FTP task table;

the service file uploading module 23 may be configured to start an FTP file uploading process of the target node when the to-be-uploaded task exists in the FTP task table, so as to execute an uploading operation of the to-be-uploaded task.

In some embodiments of the present specification, the FTP runtime in the FTP runtime table is inserted via a dynamically configured first SQL script; and the tasks to be uploaded in the FTP task table are inserted in batches through a second SQL script which is dynamically configured.

In some embodiments of the present specification, the service data uploading apparatus may further include a task table updating module. The task table updating module may be configured to update the uploading state of the to-be-uploaded task in the FTP task table after the uploading operation of the to-be-uploaded task is successfully performed, so as to continue uploading when the next FTP running time comes.

In some embodiments of the present specification, the service data uploading apparatus may further include a process first control module. The process first control module may be configured to start an FTP stop process when the current time is the FTP non-runtime, so as to stop all processes started by the FTP daemon process in the target process list.

In some embodiments of the present specification, the service data uploading apparatus may further include a process second control module. The second process control module may be configured to, when there are multiple started task confirmation processes, invoke the FTP daemon to periodically scan a preset node-level parallelism configuration table, so as to determine whether the number of currently started task confirmation processes exceeds a task confirmation process parallelism upper limit specified in the node-level parallelism configuration table; randomly killing a specified first number of task confirmation processes when the number of currently started task confirmation processes exceeds the task confirmation process parallelism upper limit.

In some embodiments of the present specification, the service data uploading apparatus may further include a process third control module. The process third control module may be configured to, when there are multiple currently started FTP file uploading processes of a single target node, call the FTP daemon to periodically scan a preset file level parallelism configuration table to determine whether the number of the currently started FTP file uploading processes of the target node exceeds an FTP file uploading process parallel upper limit specified in the file level parallelism configuration table; and randomly killing FTP file uploading processes of a specified second quantity when the quantity of the FTP file uploading processes started currently by the target node exceeds the parallel upper limit of the FTP file uploading processes.

Referring to fig. 3, corresponding to the above-mentioned service data uploading, a service data uploading apparatus according to another embodiment of the present specification may include a memory, a processor, and a computer program stored in the memory, where the computer program is executed by the processor to perform the following steps:

and when the FTP task table has the task to be uploaded, starting an FTP file uploading process of the target node to execute the uploading operation of the task to be uploaded.

For convenience of description, the above devices are described as being divided into various units by function, and are described separately. Of course, the functions of the various elements may be implemented in the same one or more software and/or hardware implementations of the present description.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.

Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method or apparatus that comprises the element.

As will be appreciated by one skilled in the art, embodiments of the present description may be provided as a method, system, or computer program product. Accordingly, the description may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the description may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

This description may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

All the embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.

The above description is only an example of the present disclosure, and is not intended to limit the present disclosure. Various modifications and alterations to this description will become apparent to those skilled in the art. Any modification, equivalent replacement, improvement or the like made within the spirit and principle of the present specification should be included in the scope of the claims of the present specification.

Claims

1. A service data uploading method is characterized by comprising the following steps:

calling an FTP daemon to periodically scan a preset FTP operation time table so as to judge whether the current time is the FTP operation time; the FTP running time is a subset of idle periods of a target online transaction system;

when a plurality of started task confirmation processes exist, calling the FTP daemon process to periodically scan a preset node level parallelism configuration table so as to judge whether the number of the currently started task confirmation processes exceeds a task confirmation process parallel upper limit specified in the node level parallelism configuration table; randomly killing a specified first number of task confirmation processes when the number of the currently started task confirmation processes exceeds the parallel upper limit of the task confirmation processes;

when a plurality of FTP file uploading processes started currently by a single target node exist, calling the FTP daemon process to periodically scan a preset file-level parallelism configuration table so as to judge whether the number of the FTP file uploading processes started currently by the target node exceeds the parallel upper limit of the FTP file uploading processes specified in the file-level parallelism configuration table or not; when the number of the FTP file uploading processes started currently by the target node exceeds the parallel upper limit of the FTP file uploading processes, randomly killing a specified second number of the FTP file uploading processes;

2. The service data uploading method according to claim 1, further comprising:

and after the uploading operation of the task to be uploaded is successfully executed, updating the uploading state of the task to be uploaded in the FTP task table so as to continue uploading when the next FTP running time comes.

3. The service data uploading method according to claim 1, further comprising:

and when the current time is the FTP non-running time, starting the FTP stopping process to stop all processes started by the FTP daemon process in the target process list.

4. The business data uploading method according to claim 1, wherein the FTP runtime in the FTP runtime table is inserted through a first SQL script that is dynamically configured; and the tasks to be uploaded in the FTP task table are inserted in batches through a second SQL script which is dynamically configured.

5. A service data uploading apparatus, comprising:

the FTP daemon is used for periodically scanning a preset FTP operation time table to judge whether the current time is the FTP operation time; the FTP running time is a subset of idle periods of a target online transaction system;

the second process control module is used for calling the FTP daemon process to periodically scan a preset node-level parallelism configuration table when a plurality of started task confirmation processes exist so as to judge whether the number of the currently started task confirmation processes exceeds a task confirmation process parallel upper limit specified in the node-level parallelism configuration table or not; randomly killing a specified first number of task confirmation processes when the number of the currently started task confirmation processes exceeds the parallel upper limit of the task confirmation processes;

the process third control module is used for calling the FTP daemon process to periodically scan a preset file-level parallelism configuration table when a plurality of currently started FTP file uploading processes exist on a single target node so as to judge whether the number of the currently started FTP file uploading processes of the target node exceeds an FTP file uploading process parallel upper limit specified in the file-level parallelism configuration table or not; when the number of the FTP file uploading processes started currently by the target node exceeds the parallel upper limit of the FTP file uploading processes, randomly killing a specified second number of the FTP file uploading processes;

6. The traffic data uploading apparatus according to claim 5, further comprising:

and the task table updating module is used for updating the uploading state of the task to be uploaded in the FTP task table after the uploading operation of the task to be uploaded is successfully executed so as to continue uploading when the next FTP running time comes.

7. The traffic data uploading apparatus according to claim 5, further comprising:

and the first process control module is used for starting the FTP stopping process when the current time is the FTP non-running time so as to stop all processes started by the FTP daemon process in the target process list.

8. The business data uploading apparatus according to claim 5, wherein the FTP runtime in the FTP runtime table is inserted through a first SQL script that is dynamically configured; and the tasks to be uploaded in the FTP task table are inserted in batches through a second SQL script which is dynamically configured.

9. A business data uploading apparatus comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program when executed by the processor performs the steps of:

when a plurality of currently started FTP file uploading processes exist in a single target node, calling the FTP daemon process to periodically scan a preset file-level parallelism configuration table so as to judge whether the number of the currently started FTP file uploading processes of the target node exceeds a specified FTP file uploading process parallel upper limit in the file-level parallelism configuration table or not; when the number of the currently started FTP file uploading processes of the target node exceeds the parallel upper limit of the FTP file uploading processes, randomly killing a specified second number of FTP file uploading processes;

10. A computer storage medium on which a computer program is stored, the computer program, when executed by a processor, performing the steps of:

when the FTP task table has tasks to be uploaded and a plurality of currently started FTP file uploading processes of a single target node exist, calling the FTP daemon process to periodically scan a preset file level parallelism configuration table so as to judge whether the number of the currently started FTP file uploading processes of the target node exceeds an FTP file uploading process parallel upper limit specified in the file level parallelism configuration table or not; when the number of the FTP file uploading processes started currently by the target node exceeds the parallel upper limit of the FTP file uploading processes, randomly killing a specified second number of the FTP file uploading processes; and starting an FTP file uploading process of the target node to execute the uploading operation of the task to be uploaded.