CN110191182B - Distributed file batch processing method, device, equipment and readable storage medium - Google Patents
Distributed file batch processing method, device, equipment and readable storage medium Download PDFInfo
- Publication number
- CN110191182B CN110191182B CN201910465966.5A CN201910465966A CN110191182B CN 110191182 B CN110191182 B CN 110191182B CN 201910465966 A CN201910465966 A CN 201910465966A CN 110191182 B CN110191182 B CN 110191182B
- Authority
- CN
- China
- Prior art keywords
- file
- downloading
- exported
- files
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/178—Techniques for file synchronisation in file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/2866—Architectures; Arrangements
- H04L67/30—Profiles
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a distributed file batch processing method, which comprises the following steps: when a file downloading notification is detected, determining a downloading end corresponding to the file downloading notification, downloading a file corresponding to the file downloading notification from a corresponding cloud storage to a local disk corresponding to the downloading end through the downloading end; reading the file content of the file in the local disk through the downloading end, and importing the file into a database corresponding to the downloading end through the downloading end based on the file content; processing the files in the database through the processing end corresponding to the database to obtain files to be exported; exporting the file to be exported to the local disk through the processing end, and sending the file to be exported to the cloud storage. The invention also discloses a distributed file batch processing device, equipment and a storage medium. The invention replaces NAS disc with database to realize distributed data exchange.
Description
Technical Field
The present invention relates to the field of financial technology (Fintech), and in particular, to a distributed file batch processing method, apparatus, device, and readable storage medium.
Background
In recent years, with the development of financial technology (Fintech), particularly internet finance, data processing technology has been introduced into daily services of financial institutions such as banks. In the daily service process of a financial institution, running batch work is often needed, and the running batch mainly comprises generation of a general account, checking of the general account or mass transaction, such as settlement, counting, paying, etc.; or generating a report, exporting stream data and the like, namely, how to batch the distributed files from each extension mechanism.
In the prior art, a shared NAS disk (Network Attached Storage, network memory) is adopted to store data, so as to realize distributed architecture data exchange, specifically, a local machine downloads a file from cloud storage to the local disk, the local machine writes the file into the NAS disk, a processing system reads the file from the NAS disk and processes the file, and finally the processing system writes the processed file into the NAS disk again and synchronizes the processed file into the cloud storage, so as to realize data exchange.
However, the current batch processing method for distributed files depends on NAS disk implementation, and requires higher hardware cost, and NAS performance is slow to write, especially when a large number of small files are written. Thus, the prior art is still susceptible to improvement in the batch processing of distributed files.
Disclosure of Invention
The invention mainly aims to provide a distributed file batch processing method, a device, equipment and a readable storage medium, aiming at improving the efficiency of processing distributed files.
In order to achieve the above object, the present invention provides a distributed file batch processing method, which includes the following steps:
when a file downloading notification is detected, determining a downloading end corresponding to the file downloading notification, downloading a file corresponding to the file downloading notification from a corresponding cloud storage to a local disk corresponding to the downloading end through the downloading end;
reading the file content of the file in the local disk through the downloading end, and importing the file into a database corresponding to the downloading end through the downloading end based on the file content;
processing the files in the database through the processing end corresponding to the database to obtain files to be exported;
Exporting the file to be exported to the local disk through the processing end, and sending the file to be exported to the cloud storage.
Preferably, when the file download notification is detected, determining a download end corresponding to the file download notification, and downloading the file corresponding to the file download notification from the corresponding cloud storage to the local disk corresponding to the download end through the download end, where the step of downloading the file from the corresponding cloud storage to the local disk corresponding to the download end includes:
when a file downloading notification is detected, determining a downloading end corresponding to the file downloading notification;
the downloading end notifies the file downloading state in the corresponding file list to be the file to be downloaded, and downloads the file to the local disk corresponding to the downloading end from the corresponding cloud storage;
and changing the file state of the file into downloading or downloading by the downloading end, and updating the file list according to the current file state.
Preferably, the step of reading, by the downloading end, the file content of the file in the local disc, and importing, by the downloading end, the file into a database corresponding to the downloading end based on the file content includes:
reading a receiving and transmitting configuration table of the file in the local disk through the downloading end, and determining whether the file needs to be imported or not based on the receiving and transmitting configuration table;
If so, determining the file type of the file through the processing end based on the receiving and transmitting configuration table, and determining the import type of the file based on the file type;
and importing the file into a database corresponding to the downloading end through the downloading end based on the importing type.
Preferably, if the import type is single-threaded import, the step of importing, by the downloading end, the file into a database corresponding to the downloading end based on the import type includes:
determining whether a failure record corresponding to the file exists in the database;
if yes, cleaning the failure record, and reading the file configuration of the file through the downloading end to determine whether the file head of the file needs to be skipped;
if so, importing the file skipped over the file header into a database corresponding to the downloading end through the downloading end.
Preferably, if the import type is multithreaded import, the step of importing, by the downloading end, the file into a database corresponding to the downloading end based on the import type includes:
determining whether a failure record corresponding to the file exists in the database;
If yes, clearing the failure record, and splitting the file into a first split file with a first preset number through the downloading end;
and the first split file is parallelly imported into a database corresponding to the downloading end through the downloading end.
Preferably, the step of exporting, by the processing end, the file to be exported to the local disk, and sending, by the processing end, the file to be exported to the cloud storage includes:
acquiring a transceiving configuration table corresponding to the file to be exported through the processing end, and determining the initialization state of the file to be exported based on the transceiving configuration table corresponding to the file to be exported;
based on the initialization state, determining whether the file to be exported needs to be exported or not through the processing end;
if so, determining, by the processing end, that the file to be exported is in the local disk and a first directory corresponding to the cloud storage, splitting the file to be exported into a second split file with a second preset number, and exporting the second split file to the first directory;
and merging the second split files into merged files in the first catalogue, and merging the merged files into a second catalogue corresponding to the file to be exported.
Preferably, the step of determining, by the processing end, that the file to be exported is in the local disk and in the first directory corresponding to the cloud storage, splitting the file to be exported into a second split file with a second preset number, and exporting the second split file to the first directory includes:
if so, determining that the file to be exported is stored in the local disk and the cloud storage corresponding first catalogue, and determining whether the first catalogue has residual files or not;
if the files do not exist, splitting the files to be exported into second split files with a second preset number through the processing end, and exporting the second split files to the first directory;
if the file exists, deleting the residual file, splitting the file to be exported into a second split file with a second preset number through the processing end, and exporting the second split file to the first directory.
In addition, in order to achieve the above object, the present invention also provides a distributed file batch processing apparatus, including:
the downloading module is used for determining a downloading end corresponding to the file downloading notification when the file downloading notification is detected, downloading the file corresponding to the file downloading notification from the corresponding cloud storage to the local disk corresponding to the downloading end through the downloading end;
The importing module is used for reading the file content of the file from the local disk through the downloading end and importing the file into a database corresponding to the downloading end through the downloading end based on the file content;
the processing module is used for processing the files in the database through the processing end corresponding to the database so as to obtain files to be exported;
and the export module is used for exporting the file to be exported to the local disk through the processing end and sending the file to be exported to the cloud storage.
Preferably, the download module is further configured to:
when a file downloading notification is detected, determining a downloading end corresponding to the file downloading notification;
the downloading end notifies the file downloading state in the corresponding file list to be the file to be downloaded, and downloads the file to the local disk corresponding to the downloading end from the corresponding cloud storage;
and changing the file state of the file into downloading or downloading by the downloading end, and updating the file list according to the current file state.
Preferably, the importing module is further configured to:
reading a receiving and transmitting configuration table of the file in the local disk through the downloading end, and determining whether the file needs to be imported or not based on the receiving and transmitting configuration table;
If so, determining the file type of the file through the processing end based on the receiving and transmitting configuration table, and determining the import type of the file based on the file type;
and importing the file into a database corresponding to the downloading end through the downloading end based on the importing type.
Preferably, if the import type is single-threaded import, the import module is further configured to:
determining whether a failure record corresponding to the file exists in the database;
if yes, cleaning the failure record, and reading the file configuration of the file through the downloading end to determine whether the file head of the file needs to be skipped;
if so, importing the file skipped over the file header into a database corresponding to the downloading end through the downloading end.
Preferably, if the import type is multithreaded import, the import module is further configured to:
determining whether a failure record corresponding to the file exists in the database;
if yes, clearing the failure record, and splitting the file into a first split file with a first preset number through the downloading end;
and the first split file is parallelly imported into a database corresponding to the downloading end through the downloading end.
Preferably, the deriving module is further configured to:
acquiring a transceiving configuration table corresponding to the file to be exported through the processing end, and determining the initialization state of the file to be exported based on the transceiving configuration table corresponding to the file to be exported;
based on the initialization state, determining whether the file to be exported needs to be exported or not through the processing end;
if so, determining, by the processing end, that the file to be exported is in the local disk and a first directory corresponding to the cloud storage, splitting the file to be exported into a second split file with a second preset number, and exporting the second split file to the first directory;
and merging the second split files into merged files in the first catalogue, and merging the merged files into a second catalogue corresponding to the file to be exported.
Preferably, the deriving module is further configured to:
if so, determining that the file to be exported is stored in the local disk and the cloud storage corresponding first catalogue, and determining whether the first catalogue has residual files or not;
if the files do not exist, splitting the files to be exported into second split files with a second preset number through the processing end, and exporting the second split files to the first directory;
If the file exists, deleting the residual file, splitting the file to be exported into a second split file with a second preset number through the processing end, and exporting the second split file to the first directory.
In addition, to achieve the above object, the present invention also provides a distributed file batch processing apparatus, including: the system comprises a memory, a processor and a distributed file batch program stored on the memory and capable of running on the processor, wherein the distributed file batch program realizes the steps of the distributed file batch method when being executed by the processor.
In addition, in order to achieve the above object, the present invention also provides a computer-readable storage medium having stored thereon a distributed file batch processing program which, when executed by a processor, implements the steps of the distributed file batch processing method as described above.
When a file downloading notification is detected, determining a downloading end corresponding to the file downloading notification, downloading a file corresponding to the file downloading notification from a corresponding cloud storage to a local disk corresponding to the downloading end through the downloading end; reading the file content of the file in the local disk through the downloading end, and importing the file into a database corresponding to the downloading end through the downloading end based on the file content; processing the files in the database through the processing end corresponding to the database to obtain files to be exported; exporting the file to be exported to the local disk through the processing end, and sending the file to be exported to the cloud storage. The invention replaces NAS disc with database to realize distributed data exchange.
Drawings
FIG. 1 is a schematic diagram of a device architecture of a hardware operating environment according to an embodiment of the present invention;
FIG. 2 is a flow chart of a first embodiment of a distributed file batch processing method according to the present invention;
FIG. 3 is a flowchart illustrating a second embodiment of a distributed file batch processing method according to the present invention.
The achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.
Detailed Description
It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
Referring to fig. 1, fig. 1 is a schematic device structure of a hardware running environment according to an embodiment of the present invention.
The terminal of the embodiment of the invention can be a PC or a server device.
As shown in fig. 1, the terminal may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, a communication bus 1002. Wherein the communication bus 1002 is used to enable connected communication between these components. The user interface 1003 may include a Display, an input unit such as a Keyboard (Keyboard), and the optional user interface 1003 may further include a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface). The memory 1005 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also optionally be a storage device separate from the processor 1001 described above.
It will be appreciated by those skilled in the art that the device structure shown in fig. 1 is not limiting of the device and may include more or fewer components than shown, or may combine certain components, or a different arrangement of components.
As shown in FIG. 1, an operating system, network communications modules, user interface modules, and distributed file batching programs may be included in memory 1005, which is a type of computer storage medium.
The operating system is a program for managing and controlling the distributed file batch processing equipment and software resources and supports the operation of a network communication module, a user interface module, the distributed file batch processing program and other programs or software; the network communication module is used to manage and control the network interface 1002; the user interface module is used to manage and control the user interface 1003.
In the distributed file batch processing apparatus shown in fig. 1, the distributed file batch processing apparatus calls a distributed file batch processing program stored in a memory 1005 through a processor 1001 and performs operations in various embodiments of the distributed file batch processing method described below.
Based on the hardware structure, the embodiment of the distributed file batch processing method is provided.
Referring to fig. 2, fig. 2 is a flow chart of a first embodiment of a distributed file batch processing method according to the present invention, where the method includes:
step S10, when a file downloading notification is detected, determining a downloading end corresponding to the file downloading notification, and downloading a file corresponding to the file downloading notification from a corresponding cloud storage to a local disk corresponding to the downloading end through the downloading end;
step S20, reading file content of the file in the local disk through the downloading end, and importing the file into a database corresponding to the downloading end through the downloading end based on the file content;
step S30, processing the files in the database through the processing end corresponding to the database to obtain files to be exported;
and step S40, exporting the file to be exported to the local disk through the processing end, and sending the file to be exported to the cloud storage.
According to the embodiment, a database is used for replacing an NAS disc, files are downloaded to a local disc through a downloading end and imported into the database, then the files are read from the database through a processing end and processed, and finally the processed files are exported to the local disc and synchronized to the database, so that batch processing of the distributed files is realized.
The following will explain each step in detail:
step S10, when a file downloading notification is detected, determining a downloading end corresponding to the file downloading notification, and downloading a file corresponding to the file downloading notification from a corresponding cloud storage to a local disk corresponding to the downloading end through the downloading end.
The distributed file batch processing method is applied to distributed file batch processing equipment of financial institutions such as financial institutions or banking systems, and is hereinafter referred to as processing equipment for convenience of description, wherein the processing equipment comprises a plurality of downloading ends, a plurality of processing ends, a database and the like, file downloading notification is issued by a message bus, any downloading end can receive the file downloading notification, and the downloading end receiving the file downloading notification is responsible for downloading the file corresponding to the file downloading notification to a database.
Specifically, when a file download notification issued by a message bus is detected, a download end receiving the file download notification is determined, and a corresponding file is downloaded from a cloud storage corresponding to the file download notification to a local disk corresponding to the download end through the download end.
When determining the downloading end receiving the file downloading notification, the IP information of the downloading end needs to be recorded, and when downloading the corresponding file to the local disk, the recorded IP information is associated with the current file, i.e. the downloading end from which the current file is downloaded is recorded.
Further, step S10 includes:
step a, when a file downloading notification is detected, determining a downloading end corresponding to the file downloading notification;
in this step, to prevent a single point of failure, any downloading end may receive a file downloading notification, specifically, initiate a polling process, and each downloading end monitors the file status of each file in the file list in a round-robin manner. When the file downloading notification is detected, the downloading end in the monitoring state is determined to be the downloading end, and the IP information of the downloading end is recorded.
Step b, the downloading end downloads the file state in the file list corresponding to the file downloading notification as the file to be downloaded from the corresponding cloud storage to the local disk corresponding to the downloading end;
in this step, the file list has file states of each file to indicate whether each file needs to be downloaded, specifically, the downloading end scans the file states marked in the file list corresponding to the file download notification, and downloads the file states as files to be downloaded from the corresponding cloud storage to the local disk corresponding to the downloading end.
And c, changing the file state of the file into downloading or downloading by the downloading end, and updating the file list according to the current file state.
In the step, in the process of downloading the current file by the downloading end, changing the file state of the current file into downloading, after the downloading is completed, changing the file state of the current file into downloaded, and updating the file state of the current file into a file list in real time so as to update the file list.
It can be understood that in this embodiment, the downloading end only downloads the file state as the file to be downloaded, and when the file state of the current file is in the downloading process, even if the downloading is not completed in the polling interval, the next downloading end does not need to download the file when detecting the file downloading notification.
Of course, in order to avoid that the downloading end is down or the downloading process is interrupted, so that the downloading end is always in a downloading state, the downloading end currently executing the downloading action not only downloads the files to be downloaded in the file list to the local disk, but also detects whether the downloading end corresponding to the files in the downloading state is normal, and particularly can determine the IP information associated with the files in the downloading state, and sends a survival detection packet to the downloading end corresponding to the IP information through the current downloading end, so as to determine whether a corresponding response packet is received in a first preset time; or in the second preset time, determining whether the downloading progress of the file in the downloading state changes or not, if so, indicating that the downloading end corresponding to the file in the downloading state is normal, and not needing to adjust the file in the downloading state; if not, determining that the downloading end corresponding to the file in the downloading state is down or the downloading process is interrupted, at the moment, executing the switching action, specifically switching to the current downloading end to execute the downloading operation, and changing the IP information associated with the corresponding file in the file list into the IP information of the current downloading end.
And step S20, reading the file content of the file in the local disk through the downloading end, and importing the file into a database corresponding to the downloading end through the downloading end based on the file content.
In this embodiment, after downloading a file to a local disk, reading, by a downloading end, file content of a current file in the local disk, where the file content includes IP information associated with the current file, determining, based on the file content, whether the current file is downloaded by the current downloading end, and if not, not processing the current file; if yes, the current file is imported into a database corresponding to the downloading end. The current downloading end can only import the current file into the database, and the current downloading end does not need to import the file into the database when the current file is in the state of being downloaded or in the downloading process or the files downloaded by other downloading ends.
Further, after the file content of the file is read, the importing time is determined based on the file content, it can be understood that some files do not need urgent processing, so that the downloaded files can be divided into urgent files and non-urgent files in advance, the urgent files can be imported into the database in real time, and for the non-urgent files, batch processing can be performed, namely, the currently downloaded non-urgent files are reserved, and when the number of the non-urgent files reaches the preset number, all non-urgent files are imported into the database in one block, or non-urgent files in the interval time are imported into the database in one block based on the preset interval time.
In the process of importing the current file into the database, the file state of the current file is marked as imported, and after the current file is imported into the database, the file state of the current file is marked as imported, so that repeated importing is avoided.
And step S30, processing the files in the database through the processing end corresponding to the database to obtain files to be exported.
In this embodiment, a processing end corresponding to the database is adopted to read a file in the database and process the file, and a specific processing process is determined according to a service type of the file, if the current file is a payment and settlement file and belongs to an accounting class, accounting is performed on a payment account and a payment number in the payment and settlement file so as to deduct an amount corresponding to the payment account and settle the deducted amount; if the current file is an interest file and belongs to the interest class, acquiring the interest rate of each account, the deposit amount of each account and the storage time in the interest file, and calculating the interest of each account based on the interest rate, the amount and the storage time; if the current file is a running file and belongs to a running account, the running data is exported, and a running report is generated.
The method comprises the steps of adopting a processing end to read a current file, determining the service type of the current file, and processing the current file by adopting a corresponding processing mode based on the service type to obtain a file to be exported.
And step S40, exporting the file to be exported to the local disk through the processing end, and sending the file to be exported to the cloud storage.
In this embodiment, after the file to be exported is obtained, the file to be exported is exported to the local disk through the processing end, so as to update the corresponding file in the local disk, and the file to be exported is sent to the cloud storage, so as to update the corresponding file in the cloud storage.
Further, the files to be exported can be reserved, after a third preset time is set, all the files to be exported in the third preset time are exported to the local disk, and all the files to be exported are synchronously sent to the cloud storage.
Further, the files to be exported can be reserved, when the files to be exported reach the preset quantity, all the files to be exported are exported to the local disk in one block, and all the files to be exported are synchronously sent to the cloud storage.
When a file downloading notification is detected, determining a downloading end corresponding to the file downloading notification, and downloading a file corresponding to the file downloading notification from a corresponding cloud storage to a local disk corresponding to the downloading end through the downloading end; reading the file content of the file in the local disk through the downloading end, and importing the file into a database corresponding to the downloading end through the downloading end based on the file content; processing the files in the database through the processing end corresponding to the database to obtain files to be exported; exporting the file to be exported to the local disk through the processing end, and sending the file to be exported to the cloud storage. According to the invention, the target file is downloaded by the downloading end and imported into the database through the database instead of the NAS disk, the corresponding processing end processes the data of the target file, and after the processing, the processed target file is exported to the local disk and synchronized to cloud storage, so that distributed data exchange is realized.
Further, based on the first embodiment of the distributed file batch processing method of the present invention, a second embodiment of the distributed file batch processing method of the present invention is provided.
The second embodiment of the distributed file batch processing method is different from the first embodiment of the distributed file batch processing method in that, referring to fig. 3, step S20 includes:
step S21, reading a receiving and transmitting configuration table of the file in the local disk through the downloading end, and determining whether the file needs to be imported or not based on the receiving and transmitting configuration table;
step S22, if necessary, determining the file type of the file by the processing end based on the transceiving configuration table, and determining the import type of the file based on the file type;
step S23, based on the import type, importing the file into a database corresponding to the downloading end through the downloading end.
When the downloaded file is imported into the database, the file type and the import type of the file are required to be judged, so that the correct file is smoothly imported into the database, and the intelligence of file import is improved.
The following will explain each step in detail:
and S21, reading a receiving and transmitting configuration table of the file in the local disk through the downloading end, and determining whether the file needs to be imported or not based on the receiving and transmitting configuration table.
In this embodiment, each file corresponds to a transceiving configuration table, and the transceiving configuration table is set with the import and export conditions of the corresponding file in advance, which specifically includes that the database needs to be imported, the database needs to be exported, the corresponding file type, and the like.
Step S22, if necessary, determining the file type of the file by the processing end based on the transceiving configuration table, and determining the import type of the file based on the file type.
In this embodiment, if it is determined that the current file needs to be imported into the database, the transceiving configuration table corresponding to the current file is further read, and the file type of the current file is determined, where the file type includes a fixed-length file and a non-fixed-length file, it may be understood that if it is determined that the current file is a fixed-length file, a fixed-length character corresponding to the fixed-length file needs to be further determined, so that when the current file is imported into the database, the current file is imported according to the determined fixed-length character.
After determining the file type of the current file, determining the import type according to the file type, specifically, if the file type of the current file is a fixed-length file, the corresponding import type is multithreaded import; if the file type of the current file is a file with a non-fixed length, the corresponding import type is single-thread import.
Step S23, based on the import type, importing the file into a database corresponding to the downloading end through the downloading end.
In this embodiment, according to the determined import type and the corresponding import manner, the current file is imported into the database corresponding to the downloading end through the downloading end.
Specifically, if the import type is single-threaded import, step S23 specifically includes:
determining whether a failure record corresponding to the file exists in the database;
in this step, it is first determined whether there is a failure record corresponding to the current file in the database, and it can be understood that a failure such as downtime or process interruption is likely to occur in the process of importing the file into the database, so that the file is imported and failed, and the data remaining in the database is the failure record, so that in order to avoid that the current file is the file that is imported and failed before, the data is repeated, so that too much space of the database is occupied, it is necessary to first determine whether there is a failure record corresponding to the current file in the database.
If yes, cleaning the failure record, and reading the file configuration of the file through the downloading end to determine whether the file head of the file needs to be skipped;
In this step, if it is determined that there is a failure record corresponding to the current file in the database, the failure record is cleaned, and the file configuration of the current file is read through the download end, it can be understood that when the file is imported into the database, the files of the same type have the same file header, so that some files can determine in advance whether the file header needs to be imported according to actual conditions, if the number of lines currently imported is the first line of the current file, and when the file configuration is performed to skip the file header, the file header of the current file is not imported into the database, otherwise, the file header is imported into the database.
It can be appreciated that, if not, the step of reading the file configuration of the file by the downloading end is directly performed to determine whether the file header of the file needs to be skipped.
If so, importing the file skipped over the file header into a database corresponding to the downloading end through the downloading end.
In the step, if the need is determined, the downloading end skips the file header, and the files except the file header are imported into the database corresponding to the downloading end.
It will be appreciated that importing the current file into the corresponding database is performed directly, if not required.
If the import type is multithreaded import, the step S23 specifically includes:
determining whether a failure record corresponding to the file exists in the database;
in this step, it is first determined whether there is a failure record corresponding to the current file in the database, and it can be understood that a failure such as downtime or process interruption is likely to occur in the process of importing the file into the database, so that the file is imported and failed, and the data remaining in the database is the failure record, so that in order to avoid that the current file is the file that is imported and failed before, the data is repeated, so that too much space of the database is occupied, it is necessary to first determine whether there is a failure record corresponding to the current file in the database.
If yes, clearing the failure record, and splitting the file into a first split file with a first preset number through the downloading end;
in the step, if the failure record corresponding to the current file exists in the database, the failure record is cleaned, and the current file is split into a first split file with a first preset number through the downloading end, wherein the first preset number can be set according to the actual situation.
Further, when splitting the current file into the first split file, the corresponding splitting number can be determined through the fixed-length character corresponding to the current file and the file content of the current file, and the current file is split into split files corresponding to the splitting number according to the splitting number.
And the first split file is parallelly imported into a database corresponding to the downloading end through the downloading end.
In the step, the split first split files are parallelly imported into the database corresponding to the downloading end through the downloading end, namely, a plurality of threads are adopted, and simultaneously the split first split files are parallelly imported into the database.
And finally, generating a corresponding file record, and particularly marking the file state of the current file as imported.
When the downloaded file is imported into the database, the file type and the import type of the file are required to be judged, so that the correct file is smoothly imported into the database, and the intelligence of file import is improved.
Further, based on the first and second embodiments of the distributed file batch processing method of the present invention, a third embodiment of the distributed file batch processing method of the present invention is provided.
The third embodiment of the distributed file batch processing method is different from the first and second embodiments of the distributed file batch processing method in that step S40 includes:
acquiring a transceiving configuration table corresponding to the file to be exported through the processing end, and determining the initialization state of the file to be exported based on the transceiving configuration table corresponding to the file to be exported;
Based on the initialization state, determining whether the file to be exported needs to be exported or not through the processing end;
if so, determining, by the processing end, that the file to be exported is in the local disk and a first directory corresponding to the cloud storage, splitting the file to be exported into a second split file with a second preset number, and exporting the second split file to the first directory;
and merging the second split files into merged files in the first catalogue, and merging the merged files into a second catalogue corresponding to the file to be exported.
In this embodiment, when the file to be exported is exported to the local disk and cloud storage, further judgment needs to be performed on the file to be exported, that is, not all the files to be exported need to be exported, but only the files to be exported meeting the conditions are exported, and when the files to be exported are exported, the correct export mode is selected, so that the export of the files is faster and more convenient.
The following will explain each step in detail:
and acquiring a transceiving configuration table corresponding to the file to be exported through the processing end, and determining the initialization state of the file to be exported based on the transceiving configuration table corresponding to the file to be exported.
In this embodiment, when the file to be exported is exported, the processing end obtains the transceiving configuration table corresponding to the file to be exported, and it can be understood that each file has a transceiving configuration table all the way, and the transceiving configuration table is always carried in the process of downloading the file to the local or importing the file into the database or processing the file to generate the file to be exported. The transceiving configuration table comprises an initialization state of a file to be exported. Thus, the initialization state of the file to be exported can be determined by acquiring the transceiving configuration table of the file to be exported.
And determining whether the file to be exported needs to be exported or not through the processing end based on the initialization state.
In this embodiment, the initialization state is specifically the export state, so it can be determined whether the current file to be exported needs to be exported by reading the initialization state of the file to be exported, that is, the initialization state is that the exported file to be exported needs to be exported.
Specifically, the initialization state of the file to be exported is read through the processing end, and the file to be exported, of which the initialization state is the export state, is determined to be exported. It can be understood that the number of the processing ends can be one or more, and in the case that the number of the processing ends is more than one, a polling mechanism can also be adopted to poll and read the initialization state of the file to be exported, and the initialization state is the export state, and the file to be exported processed by the processing end is determined to be exported. In the process that the processing end processes the current file to generate the file to be exported, the processing end also needs to record the IP information of the processing end so as to determine which files to be exported are processed locally through the IP information.
If so, determining, by the processing end, that the file to be exported is in the local disk and in the first directory corresponding to the cloud storage, splitting the file to be exported into a second split file with a second preset number, and exporting the second split file to the first directory.
In this embodiment, if it is determined by the processing end that the current file to be exported needs to be exported, it is determined that the file to be exported is located on the local disk and a first directory corresponding to the file to be exported in the cloud storage, that is, it is determined to which location the file to be exported is about to be exported, and the file to be exported is split into a second preset number of second split files, where the second preset number can be set according to the actual situation, and if the preset number is 7, the file to be exported is split into 7 split files.
Further, the corresponding splitting number can be determined through the fixed-length character corresponding to the file to be exported and the file content of the file to be exported, and the file to be exported is split into split files corresponding to the splitting number according to the splitting number.
After splitting the file to be exported into a plurality of split files, starting multithreading parallel export, and exporting the plurality of split files to a determined first directory.
Further, the step of exporting the second split file to the first directory includes:
if so, determining that the file to be exported is stored in the local disk and the cloud storage corresponding first catalogue, and determining whether the first catalogue has residual files or not;
If the files do not exist, splitting the files to be exported into second split files with a second preset number through the processing end, and exporting the second split files to the first directory;
if the file exists, deleting the residual file, splitting the file to be exported into a second split file with a second preset number through the processing end, and exporting the second split file to the first directory.
In the step, whether the first catalog has a residual file is determined, if so, the residual file is deleted, and the second split file is exported to the first catalog through a processing end; if the second split file does not exist, the second split file is directly exported to the first directory through the processing end.
It can be understood that when the file to be exported is exported, a fault such as downtime or process interruption may occur, in this case, a file that is not exported completely, that is, a residual file, is necessarily present in the corresponding directory, so that the residual file needs to be deleted, the second split file is exported by the processing end, so that repeated exporting is avoided, and if the residual file does not exist in the corresponding directory, the second split file is exported directly by the processing end.
And merging the second split files into merged files in the first catalogue, and merging the merged files into a second catalogue corresponding to the file to be exported.
In this embodiment, after the second split file is split and exported to the first directory, the second split file in the first directory is merged into the merged file and merged into the second directory, which specifically means that when the file to be exported is split into the second split file, the split files may be numbered, for example, part1, part2, part3, etc., and after the second split file is exported to the first directory, the split files are merged into the merged file one by one according to the number of the split file, so far, the file to be exported is synchronously exported to the local disk and the cloud storage.
In this embodiment, when the file to be exported is exported to the local disk and cloud storage, further judgment needs to be performed on the file to be exported, that is, not all the files to be exported need to be exported, but only the files to be exported meeting the conditions are exported, and when the files to be exported are exported, the correct export mode is selected, so that the export of the files is faster and more convenient.
In the above embodiment, if the downloading end and/or the processing end is down or the process is interrupted, the fault file corresponding to the fault and the fault action corresponding to the fault file are recorded, the standby end corresponding to the current downloading end and/or the processing end is switched, and the execution is continued according to the recorded fault file and fault action.
The invention also provides a distributed file batch processing device. The distributed file batch processing device of the invention comprises:
the downloading module is used for determining a downloading end corresponding to the file downloading notification when the file downloading notification is detected, downloading the file corresponding to the file downloading notification from the corresponding cloud storage to the local disk corresponding to the downloading end through the downloading end;
the importing module is used for reading the file content of the file from the local disk through the downloading end and importing the file into a database corresponding to the downloading end through the downloading end based on the file content;
the processing module is used for processing the files in the database through the processing end corresponding to the database so as to obtain files to be exported;
and the export module is used for exporting the file to be exported to the local disk through the processing end and sending the file to be exported to the cloud storage.
Further, the download module is further configured to:
when a file downloading notification is detected, determining a downloading end corresponding to the file downloading notification;
the downloading end notifies the file downloading state in the corresponding file list to be the file to be downloaded, and downloads the file to the local disk corresponding to the downloading end from the corresponding cloud storage;
And changing the file state of the file into downloading or downloading by the downloading end, and updating the file list according to the current file state.
Further, the importing module is further configured to:
reading a receiving and transmitting configuration table of the file in the local disk through the downloading end, and determining whether the file needs to be imported or not based on the receiving and transmitting configuration table;
if so, determining the file type of the file through the processing end based on the receiving and transmitting configuration table, and determining the import type of the file based on the file type;
and importing the file into a database corresponding to the downloading end through the downloading end based on the importing type.
Further, if the import type is single-threaded import, the import module is further configured to:
determining whether a failure record corresponding to the file exists in the database;
if yes, cleaning the failure record, and reading the file configuration of the file through the downloading end to determine whether the file head of the file needs to be skipped;
if so, importing the file skipped over the file header into a database corresponding to the downloading end through the downloading end.
Further, if the import type is multithreaded import, the import module is further configured to:
determining whether a failure record corresponding to the file exists in the database;
if yes, clearing the failure record, and splitting the file into a first split file with a first preset number through the downloading end;
and the first split file is parallelly imported into a database corresponding to the downloading end through the downloading end.
Further, the deriving module is further configured to:
acquiring a transceiving configuration table corresponding to the file to be exported through the processing end, and determining the initialization state of the file to be exported based on the transceiving configuration table corresponding to the file to be exported;
based on the initialization state, determining whether the file to be exported needs to be exported or not through the processing end;
if so, determining, by the processing end, that the file to be exported is in the local disk and a first directory corresponding to the cloud storage, splitting the file to be exported into a second split file with a second preset number, and exporting the second split file to the first directory;
and merging the second split files into merged files in the first catalogue, and merging the merged files into a second catalogue corresponding to the file to be exported.
Further, the deriving module is further configured to:
if so, determining that the file to be exported is stored in the local disk and the cloud storage corresponding first catalogue, and determining whether the first catalogue has residual files or not;
if the files do not exist, splitting the files to be exported into second split files with a second preset number through the processing end, and exporting the second split files to the first directory;
if the file exists, deleting the residual file, splitting the file to be exported into a second split file with a second preset number through the processing end, and exporting the second split file to the first directory.
The invention also provides a computer readable storage medium.
The computer readable storage medium of the present invention has stored thereon a distributed file batch processing program which when executed by a processor implements the steps of the distributed file batch processing method as described above.
The method implemented when the distributed file batch processing program running on the processor is executed may refer to various embodiments of the distributed file batch processing method of the present invention, which are not described herein.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.
The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.
From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) as described above, comprising instructions for causing a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to perform the method according to the embodiments of the present invention.
The foregoing description is only of the preferred embodiments of the present invention, and is not intended to limit the scope of the invention, but rather is intended to cover any equivalents of the structures or equivalent processes disclosed herein, or any application, directly or indirectly, in the field of other related technology.
Claims (14)
1. The distributed file batch processing method is characterized by comprising the following steps of:
when a file downloading notification is detected, determining a downloading end corresponding to the file downloading notification, downloading a file corresponding to the file downloading notification from a corresponding cloud storage to a local disk corresponding to the downloading end through the downloading end;
reading a receiving and transmitting configuration table of the file in the local disk through the downloading end, and determining whether the file needs to be imported or not based on the receiving and transmitting configuration table;
if so, determining the file type of the file through a processing end based on the transceiving configuration table, and determining the import type of the file based on the file type;
based on the importing type, importing the file into a database corresponding to the downloading end through the downloading end;
processing the files in the database through the processing end corresponding to the database to obtain files to be exported;
exporting the file to be exported to the local disk through the processing end, and sending the file to be exported to the cloud storage.
2. The distributed file batch processing method according to claim 1, wherein the step of determining a download end corresponding to the file download notification when the file download notification is detected, and downloading the file corresponding to the file download notification from the corresponding cloud storage to the local disk corresponding to the download end through the download end comprises:
When a file downloading notification is detected, determining a downloading end corresponding to the file downloading notification;
the downloading end notifies the file downloading state in the corresponding file list to be the file to be downloaded, and downloads the file to the local disk corresponding to the downloading end from the corresponding cloud storage;
and changing the file state of the file into downloading or downloading by the downloading end, and updating the file list according to the current file state.
3. The method of claim 1, wherein if the import type is single-threaded import, the step of importing, by the download terminal, the file into a database corresponding to the download terminal based on the import type includes:
determining whether a failure record corresponding to the file exists in the database;
if yes, cleaning the failure record, and reading the file configuration of the file through the downloading end to determine whether the file head of the file needs to be skipped;
if so, importing the file skipped over the file header into a database corresponding to the downloading end through the downloading end.
4. The method of claim 1, wherein if the import type is a multi-thread import, the step of importing, by the download terminal, the file into a database corresponding to the download terminal based on the import type includes:
Determining whether a failure record corresponding to the file exists in the database;
if yes, clearing the failure record, and splitting the file into a first split file with a first preset number through the downloading end;
and the first split file is parallelly imported into a database corresponding to the downloading end through the downloading end.
5. The method for batch processing of distributed files according to any one of claims 1 to 4, wherein the step of exporting, by the processing end, the file to be exported to the local disk and sending the file to be exported to the cloud storage includes:
acquiring a transceiving configuration table corresponding to the file to be exported through the processing end, and determining the initialization state of the file to be exported based on the transceiving configuration table corresponding to the file to be exported;
based on the initialization state, determining whether the file to be exported needs to be exported or not through the processing end;
if so, determining, by the processing end, that the file to be exported is in the local disk and a first directory corresponding to the cloud storage, splitting the file to be exported into a second split file with a second preset number, and exporting the second split file to the first directory;
And merging the second split files into merged files in the first catalogue, and merging the merged files into a second catalogue corresponding to the file to be exported.
6. The method of claim 5, wherein the step of determining, by the processing side, that the file to be exported is stored in the local disk and in the cloud in a first directory corresponding to the file to be exported, splitting the file to be exported into a second split file with a second preset number, and exporting the second split file to the first directory if the file to be exported is needed includes:
if so, determining that the file to be exported is stored in the local disk and the cloud storage corresponding first catalogue, and determining whether the first catalogue has residual files or not;
if the files do not exist, splitting the files to be exported into second split files with a second preset number through the processing end, and exporting the second split files to the first directory;
if the file exists, deleting the residual file, splitting the file to be exported into a second split file with a second preset number through the processing end, and exporting the second split file to the first directory.
7. A distributed document batch processing apparatus, the distributed document batch processing apparatus comprising:
The downloading module is used for determining a downloading end corresponding to the file downloading notification when the file downloading notification is detected, downloading the file corresponding to the file downloading notification from the corresponding cloud storage to the local disk corresponding to the downloading end through the downloading end;
the importing module is used for reading the file content of the file from the local disk through the downloading end and importing the file into a database corresponding to the downloading end through the downloading end based on the file content;
the processing module is used for processing the files in the database through the processing end corresponding to the database so as to obtain files to be exported;
the export module is used for exporting the file to be exported to the local disk through the processing end and sending the file to be exported to the cloud storage;
the import module is further configured to:
reading a receiving and transmitting configuration table of the file in the local disk through the downloading end, and determining whether the file needs to be imported or not based on the receiving and transmitting configuration table;
if so, determining the file type of the file through the processing end based on the receiving and transmitting configuration table, and determining the import type of the file based on the file type;
And importing the file into a database corresponding to the downloading end through the downloading end based on the importing type.
8. The distributed file batch processing apparatus of claim 7, wherein the download module is further configured to:
when a file downloading notification is detected, determining a downloading end corresponding to the file downloading notification;
the downloading end notifies the file downloading state in the corresponding file list to be the file to be downloaded, and downloads the file to the local disk corresponding to the downloading end from the corresponding cloud storage;
and changing the file state of the file into downloading or downloading by the downloading end, and updating the file list according to the current file state.
9. The distributed file batch processing apparatus of claim 7, wherein if the import type is single-threaded import, the import module is further configured to:
determining whether a failure record corresponding to the file exists in the database;
if yes, cleaning the failure record, and reading the file configuration of the file through the downloading end to determine whether the file head of the file needs to be skipped;
if so, importing the file skipped over the file header into a database corresponding to the downloading end through the downloading end.
10. The distributed file batch processing apparatus of claim 7, wherein if the import type is a multi-threaded import, the import module is further configured to:
determining whether a failure record corresponding to the file exists in the database;
if yes, clearing the failure record, and splitting the file into a first split file with a first preset number through the downloading end;
and the first split file is parallelly imported into a database corresponding to the downloading end through the downloading end.
11. A distributed file batching apparatus according to any one of claims 7 to 10, wherein the export module is further adapted to:
acquiring a transceiving configuration table corresponding to the file to be exported through the processing end, and determining the initialization state of the file to be exported based on the transceiving configuration table corresponding to the file to be exported;
based on the initialization state, determining whether the file to be exported needs to be exported or not through the processing end;
if so, determining, by the processing end, that the file to be exported is in the local disk and a first directory corresponding to the cloud storage, splitting the file to be exported into a second split file with a second preset number, and exporting the second split file to the first directory;
And merging the second split files into merged files in the first catalogue, and merging the merged files into a second catalogue corresponding to the file to be exported.
12. The distributed file batching apparatus according to claim 11, wherein the export module is further configured to:
if so, determining that the file to be exported is stored in the local disk and the cloud storage corresponding first catalogue, and determining whether the first catalogue has residual files or not;
if the files do not exist, splitting the files to be exported into second split files with a second preset number through the processing end, and exporting the second split files to the first directory;
if the file exists, deleting the residual file, splitting the file to be exported into a second split file with a second preset number through the processing end, and exporting the second split file to the first directory.
13. A distributed file batching apparatus, the distributed file batching apparatus comprising: a memory, a processor, and a distributed file batch program stored on the memory and executable on the processor, which when executed by the processor, implements the steps of the distributed file batch method of any one of claims 1 to 6.
14. A computer readable storage medium, wherein a distributed file batch processing program is stored on the computer readable storage medium, which when executed by a processor, implements the steps of the distributed file batch processing method of any one of claims 1 to 6.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910465966.5A CN110191182B (en) | 2019-05-30 | 2019-05-30 | Distributed file batch processing method, device, equipment and readable storage medium |
| PCT/CN2020/092139 WO2020238860A1 (en) | 2019-05-30 | 2020-05-25 | Distributed file batch processing method and apparatus, and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910465966.5A CN110191182B (en) | 2019-05-30 | 2019-05-30 | Distributed file batch processing method, device, equipment and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110191182A CN110191182A (en) | 2019-08-30 |
| CN110191182B true CN110191182B (en) | 2023-04-21 |
Family
ID=67719137
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910465966.5A Active CN110191182B (en) | 2019-05-30 | 2019-05-30 | Distributed file batch processing method, device, equipment and readable storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN110191182B (en) |
| WO (1) | WO2020238860A1 (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110191182B (en) * | 2019-05-30 | 2023-04-21 | 深圳前海微众银行股份有限公司 | Distributed file batch processing method, device, equipment and readable storage medium |
| CN113722277A (en) * | 2020-05-25 | 2021-11-30 | 中兴通讯股份有限公司 | Data import method, device, service platform and storage medium |
| CN112597228B (en) * | 2020-12-26 | 2024-06-07 | 中国农业银行股份有限公司 | File processing method and system |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102307219A (en) * | 2011-03-18 | 2012-01-04 | 北京神州数码思特奇信息技术股份有限公司 | File access system, file uploading method and file downloading method |
| CN108830715A (en) * | 2018-05-30 | 2018-11-16 | 平安科技（深圳）有限公司 | Batch file part disk returning processing method and system |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103379149A (en) * | 2012-04-19 | 2013-10-30 | 北京智慧风云科技有限公司 | Cloud service system providing function of processing files according to received commands |
| CN104580437A (en) * | 2014-12-30 | 2015-04-29 | 创新科存储技术（深圳）有限公司 | Cloud storage client and high-efficiency data access method thereof |
| US10152489B2 (en) * | 2015-07-24 | 2018-12-11 | Salesforce.Com, Inc. | Synchronize collaboration entity files |
| CN105956481B (en) * | 2015-09-17 | 2019-05-10 | 中国银联股份有限公司 | A kind of data processing method and its device |
| CN107870814B (en) * | 2016-09-23 | 2022-02-22 | 伊姆西Ip控股有限责任公司 | Method and apparatus for content management batching |
| CN107967347B (en) * | 2017-12-07 | 2021-01-22 | 湖北三新文化传媒有限公司 | Batch data processing method, server, system and storage medium |
| CN110191182B (en) * | 2019-05-30 | 2023-04-21 | 深圳前海微众银行股份有限公司 | Distributed file batch processing method, device, equipment and readable storage medium |
-
2019
- 2019-05-30 CN CN201910465966.5A patent/CN110191182B/en active Active
-
2020
- 2020-05-25 WO PCT/CN2020/092139 patent/WO2020238860A1/en active Application Filing
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102307219A (en) * | 2011-03-18 | 2012-01-04 | 北京神州数码思特奇信息技术股份有限公司 | File access system, file uploading method and file downloading method |
| CN108830715A (en) * | 2018-05-30 | 2018-11-16 | 平安科技（深圳）有限公司 | Batch file part disk returning processing method and system |
Non-Patent Citations (1)
| Title |
|---|
| 基于SpringBatch框架的数据批量处理在决策分析系统中的应用;徐杨等;《电子测量技术》;20160415(第04期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110191182A (en) | 2019-08-30 |
| WO2020238860A1 (en) | 2020-12-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110191182B (en) | Distributed file batch processing method, device, equipment and readable storage medium | |
| US8707194B1 (en) | System and method for decentralized performance monitoring of host systems | |
| CN103227812B (en) | Smart machine is supported method for down loading and the device of breakpoint transmission | |
| CN111866099B (en) | Method, device, system, equipment and storage medium for downloading mirror image file | |
| CN110764881A (en) | Distributed system background retry method and device | |
| CN109634730A (en) | Method for scheduling task, device, computer equipment and storage medium | |
| CN109561117A (en) | Collecting method and device | |
| CN103268183A (en) | Processing method and device for information report | |
| CN105847231A (en) | Service publishing method, device and system | |
| CN108667740B (en) | Flow control method, device and system | |
| WO2019140829A1 (en) | Electronic device, method for releasing application update version, and storage medium | |
| CN107277022B (en) | Process marking method and device | |
| CN115277566A (en) | Load balancing method and device for data access, computer equipment and medium | |
| CN103425486A (en) | Remote card content management method and system using synchronous server-side scripting | |
| CN115268854A (en) | Service expansion implementation method, system, electronic equipment and readable storage medium | |
| CN110532066A (en) | A kind of realization method and system of transaction consistency | |
| US7197533B2 (en) | Non-persistent service support in transactional application support environments | |
| CN110727523A (en) | Service request processing method, processing device and readable storage medium | |
| CN106034148B (en) | Rapid information interaction method, local server, remote server and system | |
| CN110046100A (en) | A kind of method, electronic equipment and the medium of grouping test | |
| CN109218338B (en) | Information processing system, method and device | |
| CN107438097B (en) | Network request processing method and device | |
| US11861214B2 (en) | Memory device forensics and preparation | |
| CN111741080A (en) | Network file distribution method and device | |
| CN110764911A (en) | Resource scheduling method, device and control system based on order |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |