CN115442361A - Synchronous transmission method, system, equipment and medium for large-batch files - Google Patents
Synchronous transmission method, system, equipment and medium for large-batch files Download PDFInfo
- Publication number
- CN115442361A CN115442361A CN202211051029.3A CN202211051029A CN115442361A CN 115442361 A CN115442361 A CN 115442361A CN 202211051029 A CN202211051029 A CN 202211051029A CN 115442361 A CN115442361 A CN 115442361A
- Authority
- CN
- China
- Prior art keywords
- file
- data
- files
- synchronous
- list
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000001360 synchronised effect Effects 0.000 title claims abstract description 122
- 230000005540 biological transmission Effects 0.000 title claims abstract description 54
- 238000000034 method Methods 0.000 title claims abstract description 36
- 238000012550 audit Methods 0.000 claims abstract description 41
- 230000004044 response Effects 0.000 claims abstract description 8
- 230000001172 regenerating effect Effects 0.000 claims description 13
- 238000012545 processing Methods 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 7
- 238000000605 extraction Methods 0.000 claims description 7
- 230000008569 process Effects 0.000 claims description 6
- 230000037211 monthly cycles Effects 0.000 claims description 4
- 238000012795 verification Methods 0.000 claims description 4
- 230000006835 compression Effects 0.000 claims description 3
- 238000007906 compression Methods 0.000 claims description 3
- 238000004806 packaging method and process Methods 0.000 claims description 3
- 239000000126 substance Substances 0.000 claims 1
- 230000006870 function Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method, a system, equipment and a medium for synchronously transmitting large-batch files, wherein the method comprises the following steps: a data provider generates a file list; generating a synchronous file under an FTP/SFTP directory of a data receiving party according to the file list; the data receiver downloads the synchronous file to the local part from the corresponding FTP/SFTP directory, and if the downloading is successful, the state of the updated file list is that the downloading is successful; if the downloading fails, updating the file list state into a downloading failure and generating a receipt file; the data provider regenerates a new file list according to the receipt file; the data receiver audits the downloaded synchronous files, and if the audit is successful, the state of the updated file list is successful; if the audit fails, updating the list state of the file list to be the audit failure and generating a response piece file; putting the generated receipt file into a specified receipt directory; warehousing the synchronous files into a data warehouse tool or a file system; the synchronous reliable transmission of large batch of files is realized.
Description
Technical Field
The invention belongs to the technical field of file synchronous transmission, and particularly relates to a method, a system, equipment and a medium for synchronously transmitting large-batch files.
Background
With the advent of the big data era, each business line of an enterprise produces massive data, the data forms detailed and index data after being subjected to unified ETL (extraction, conversion and processing) in an enterprise central data warehouse, and in order to unify the aperture and increase the utilization rate of the data, providing the data for other departments of the enterprise or big data projects according to a certain period is a very common application scenario, but two problems are brought: first, in the face of large-scale data, the data provider provides the data in a reliable and efficient manner; second, how to let the data receiver correctly receive the data to ensure that the data is complete and accurate. The existing method is generally suitable for the transmission of small data volume of a relational database in a synchronization scheme in the field of synchronous transmission, is somewhat careless in mass data transmission, and has no effective guarantee on the completeness and accuracy of data.
Disclosure of Invention
The invention mainly aims to overcome the defects of the prior art and provide a method, a system, equipment and a medium for synchronously transmitting mass files.
In order to achieve the purpose, the invention adopts the following technical scheme:
in one aspect, the present invention provides a method for synchronously transmitting a large number of files, comprising the following steps:
the data provider generates a file list according to the transmission requirement of the data receiver and stores the file list in a relational database;
according to the file list, the data provider circularly generates synchronous files under the FTP/SFTP directory of the data receiver, and if the synchronous files are successfully generated, the file list is updated to be in a state of successfully generating the files and named according to a naming rule; if the generation of the synchronous file fails, updating the file list state to be the file generation failure and regenerating the synchronous file;
the data receiver circularly downloads the synchronous file to the local in a concurrent mode under the corresponding FTP/SFTP directory according to the file list, and if the downloading is successful, the state of the file list is updated to be the downloading success; if the downloading fails, updating the file list state to be downloading failure and generating a receipt file; regenerating a new file list by the data provider receipt file;
after the data receiver downloads the synchronous files, auditing the downloaded synchronous files, and if the auditing is successful, updating the file list state to be the auditing success; if the audit fails, updating the file list state to be the audit failure and generating a response piece file; the data provider performs corresponding processing according to the receipt file;
after the audit is finished, the generated receipt files are put into a specified receipt directory to describe the transmission completion condition of each synchronous file;
and the data receiver stores the synchronous files after the audit is finished into a data warehouse tool or a file system.
As a preferred technical scheme, the periodically generated file list is divided into a periodically generated file list and a quasi-real-time generated file list;
the regular generation of the file list refers to the generation of the file list by extracting data according to a daily cycle or a monthly cycle, and the extraction mode is incremental or full extraction;
the quasi-real-time generation of the file list refers to the generation of the file list according to hours or minutes;
the file list comprises time, data names, synchronous files needing to be transmitted, states and interface file states;
the file list is as follows: naming in a manner of list;
the data application type is named according to the data application type;
the periods include a daily period d and a monthly period m;
the generation time refers to the generation time of the file list and is expressed as YYYYMMDD.
As a preferred technical scheme, the synchronization file comprises a data file, an MD5 check file and a CHK check file;
splitting the data file according to the data volume and the set data record number, subpackaging and compressing;
the MD5 verification file is an MD5 verification code of the data file; each data file corresponds to an MD5 check file;
the content of the CHK check file comprises a data file name, a data file record number and a data file size;
the size of the data file is the size of the data file before compression, and the unit is BYTE.
As a preferred technical solution, the naming rule is: data application type _ interface sequence number _ cycle _ file time flag _ packet number;
the data application type is named according to the data application type;
the interface serial number represents the serial number of the interface in the subject domain;
the periods include a daily period d and a monthly period m;
the file time mark is determined according to a period, and if the file time mark is a day period, the file time mark is represented as YYYYMMDD; if the month period is the period, the file time mark is represented as YYYYMM;
the sub-package number refers to a file number after sub-packaging;
the file extension comprises a data file AVL, an MD5 check file MD5, a CHK check file CHK, a normal return receipt file RR and a problem return receipt file RE.
As a preferred technical solution, the content in the receipt file includes a file number, an error code, a file name and detailed error information;
the error codes include 101, 102, 103, 104, 105, 106, and 107;
when the error code is 101, the synchronous file is not uploaded within the specified time;
when the error code is 102, the number of data files is not matched with the specified number;
when the error code is 103, the CHK check file is not matched with the specification;
when the error code is 104, the record number of the data file is inconsistent with the record number of the data file in the CHK check file;
when the error code is 105, the data file is represented as an empty file;
when the error code is 106, the CHK check file is not empty;
when the error code is 107, it indicates that the interface file fluctuation exceeds the threshold.
As a preferred technical solution, the auditing downloaded synchronous files includes auditing data integrity, consistency and accuracy, and specifically includes:
checking the integrity of data in the downloading process of the synchronous file by downloading the MD5 check file in the synchronous file; if the data file does not meet the integrity, writing the receipt file in the corresponding directory, and updating the interface file state of the file list into integrity check failure;
if the data file meets the integrity, checking the consistency of the data file by downloading a CHK check file in the synchronous file; if the data files do not meet the consistency, writing the receipt files in the corresponding directories, and updating the interface file state of the file list into consistency audit failure;
if the data files meet the consistency, checking the accuracy of the data files by downloading the CHK check files in the synchronous files; if the data files are not full, the accuracy is that the receipt files are written in the corresponding directories, and the interface file state of the file list is updated to be failure in accuracy audit; if the data file meets the accuracy, the audit is successful.
On the other hand, the invention provides a synchronous transmission system of large-batch files, which is applied to the synchronous transmission method of the large-batch files and comprises a list generation module, a file downloading module, a file auditing module, a receipt storage module and a file warehousing module;
the list generation module is used for generating a file list by a data provider according to the transmission requirement of a data receiver and storing the file list in a relational database;
the file generation module is used for circularly generating synchronous files under the FTP/SFTP directory of the data receiving party by the data providing party according to the file list, and if the synchronous files are successfully generated, the file list state is updated to be the state that the files are successfully generated and named according to the naming rule; if the synchronous file is failed to generate, updating the file list state to be the file generation failure and regenerating the synchronous file;
the file downloading module is used for circularly downloading the synchronous files to the local in a concurrent mode under the corresponding FTP/SFTP directory by the data receiving party according to the file list, and if the downloading is successful, the state of updating the file list is that the downloading is successful; if the downloading fails, updating the file list state into a downloading failure and generating a receipt file; regenerating a new file list by the data provider receipt file;
the file auditing module is used for auditing the downloaded synchronous files after the synchronous files are downloaded by the data receiving party, and if the auditing is successful, the state of updating the file list is that the auditing is successful; if the audit fails, updating the list state of the file list to be the audit failure and generating a response piece file; the data provider performs corresponding processing according to the receipt file;
the receipt storage module is used for placing the generated receipt files into a specified receipt directory after the completion of the audit and describing the transmission completion condition of each synchronous file;
and the file warehousing module is used for warehousing the synchronous files after the audit is finished into a data warehouse tool or a file system by the data receiving party.
In yet another aspect, the present invention also provides an electronic device, including:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein, the first and the second end of the pipe are connected with each other,
the memory stores computer program instructions executable by the at least one processor, the computer program
The instructions are executable by the at least one processor to enable the at least one processor to perform a method of synchronous transmission of bulk files as described above.
In still another aspect, the present invention provides a computer-readable storage medium storing a program, which when executed by a processor, implements a method for synchronous transmission of bulk files as described above.
Compared with the prior art, the invention has the following advantages and beneficial effects:
when data synchronization is carried out, massive data can be transmitted in a mode of staging, classifying and subpackaging data files, and the high efficiency of the transmission process is ensured; through the receipt files, the specific transmission condition of each file is known, and problematic data in the transmission process is quickly positioned and processed, so that the reliability of the transmission process is ensured; the MD5 check file and the CHK check file are used for auditing and checking, so that data is not lost in the transmission process, and the completeness and accuracy of the data are guaranteed.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the description below are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.
FIG. 1 is a flowchart illustrating a method for synchronously transmitting a large number of files according to an embodiment of the present invention;
fig. 2 is a block diagram of a system for synchronously transmitting a large batch of files according to an embodiment of the present invention.
Fig. 3 is a block diagram of an electronic device according to an embodiment of the invention.
Detailed Description
In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. It is to be understood that the embodiments described are only a few embodiments of the present application and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the specification. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.
Referring to fig. 1, in an embodiment of the present application, a method for synchronous transmission of a large batch of files is provided, including the following steps:
s1, a data provider generates a file list according to transmission requirements of a data receiver and stores the file list in a relational database periodically;
according to the transmission requirement of a data receiver, a data provider generates a file list according to the period, and the file list is divided into a period generated file list and a quasi-real-time generated file list; the regular generation of the file list refers to the generation of the file list by extracting data according to a daily cycle or a monthly cycle; the extraction mode is incremental or full extraction;
generating the file list in quasi-real time refers to generating the file list according to hours or minutes, and extracting and transmitting changed data in time;
in the invention, the content of the file list comprises time, data name, synchronous file to be transmitted, state and interface file state; the file list is as follows: naming in a list mode according to the data application type _ period _ generation time; the data application type is named according to the data application type; the period comprises a day period d and a month period m; the generation time refers to the generation time of the file list and is expressed as YYYYMMDD; the file list of the driving day data generated by 20220408 is represented by a file list drive _ d _20220408. List.
S2, according to the file list, the data provider circularly generates a synchronous file under the FTP/SFTP directory of the data receiver, and if the synchronous file is successfully generated, the file list state is updated to be the file generation success and named according to the naming rule; if the synchronous file is failed to generate, updating the file list state to be the file generation failure and regenerating the synchronous file;
in the invention, the synchronous file comprises a data file, an MD5 check file and a CHK check file; the data files are split according to the data volume and the set data record number, then are packaged and compressed, and the data transmission efficiency is guaranteed; the MD5 check file is an MD5 check code of the data file; each data file must correspond to an MD5 check file; the content of the CHK check file comprises a data file name, a data file record number and a data file size; the data file size is the size of the data file before compression, in units of BYTE. In this embodiment, the splitting is performed every 500 ten thousand data records.
The generated files are named according to a naming rule, wherein the naming rule is ' data application type _ interface sequence number _ period _ file time mark _ sub-packet number ' file extension name ';
the data application types are named according to the data application types, such as driving data named as "drive" and vehicle condition data named as "car-condition";
the interface serial number represents the serial number of the interface in the subject domain, such as 00001;
the cycle includes a daily cycle "d" and a monthly cycle "m";
determining the file time mark according to the period, and if the file time mark is the day period, representing the file time mark as YYYYMMDD; if the month period is the period, the file time mark is represented as YYYYMM;
the sub-package number refers to the number of the files after sub-packaging;
the file extension comprises a data file AVL, an MD5 check file MD5, a CHK check file CHK, a normal return receipt file RR and a problem return receipt file RE.
For example: a package number of 01 for a file with a file name of drive _00001_d _20220408_01.Avl, which represents a data file of daily data with a billing period of 20220408 with a driving data number of 00001;
for a file with a file name of drive _00001_d _20220408_01.Md5, which represents an MD5 check file of day data with an account period of 20220408 with a driving data number of 00001, a package number of 01;
for a file with a file name of drive _00001_d _20220408_01.CHK, which represents a CHK check file of daily data with an account period of 20220408 with a driving data number of 00001, a package number of 01;
for a file with a file name of drive _00001_d _20220408_01.RR, the file represents a receipt file of the day data with the account period of 20220408 and the divided data warehouse number of 00001, and the sub-package number of 01 represents that an interface file is normal;
for a file with a file name of drive _00001_d _20220408_01.Re, which represents a receipt file of day data with an account period of 20220408 with a divided data warehouse number of 00001, a package number of 01 represents an interface file exception.
S3, the data receiver circularly downloads the synchronous files to the local in a concurrent mode under the corresponding FTP/SFTP directory according to the file list, and if the downloading is successful, the state of the file list is updated to be the downloading success; if the downloading fails, updating the file list state into a downloading failure and generating a receipt file; the data provider regenerates a new file list according to the receipt file;
s4, after the data receiver downloads the synchronous files, auditing the downloaded synchronous files, and if the auditing is successful, updating the file list state to be successful; if the audit fails, updating the list state of the file list to be the audit failure and generating a response piece file; the data provider performs corresponding processing according to the receipt file;
auditing the integrity, consistency and accuracy of the downloaded synchronous files, specifically comprising the following steps:
checking the integrity of data in the downloading process of the synchronous file by downloading the MD5 check file in the synchronous file; if the data file does not meet the integrity, writing a receipt file in a corresponding directory, and updating the interface file state of the file list into integrity check failure;
if the data file meets the integrity, checking the consistency of the data file by downloading a CHK check file in the synchronous file; if the data files do not meet the consistency, the receipt files are written in the corresponding directories, and the interface file state of the file list is updated to be consistency audit failure;
if the data files meet the consistency, checking the accuracy of the data files by downloading CHK check files in the synchronous files; if the data files are not full, the accuracy is that the receipt files are written in the corresponding directories, and the interface file state of the file list is updated to be failure in accuracy audit; if the data file meets the accuracy, the audit is successful.
S5, after the audit is finished, putting the generated receipt files into a designated receipt directory to describe the transmission completion condition of each synchronous file;
when the data file is inconsistent with the check file data, error information is embodied in the receipt content of the data file, and the content in the receipt file comprises a file number, an error code, a file name and detailed error information;
error codes include 101, 102, 103, 104, 105, 106, and 107; when the error code is 101, the synchronous file is not uploaded within the specified time; when the error code is 102, the number of data files is not matched with the specified number; when the error code is 103, the CHK check file is not matched with the specification; when the error code is 104, the record number of the data file is inconsistent with the record number of the data file in the CHK check file; when the error code is 105, the data file is represented as an empty file; when the error code is 106, the CHK check file is not empty; when the error code is 107, it indicates that the interface file fluctuation exceeds the threshold.
And S6, the data receiver stores the synchronous files after the audit is finished into a data warehouse tool or a file system.
It should be noted that, for the sake of simplicity, the foregoing method embodiments are described as a series of acts or combinations, but those skilled in the art should understand that the present invention is not limited by the described order of acts, as some steps may be performed in other orders or simultaneously according to the present invention.
Based on the same idea as the synchronous transmission method of the large-batch files in the embodiment, the invention also provides a synchronous transmission system of the large-batch files, which can be used for executing the synchronous transmission method of the large-batch files. For convenience of illustration, the structure diagram of an embodiment of a system for synchronous transmission of large-batch files only shows a part related to the embodiment of the present invention, and those skilled in the art will understand that the illustrated structure does not constitute a limitation to the apparatus, and may include more or less components than those illustrated, or combine some components, or arrange different components.
Referring to fig. 2, in another embodiment of the present application, a system for synchronously transmitting a large number of documents is provided, which includes a list generating module, a document downloading module, a document auditing module, a receipt storing module, and a document warehousing module;
the list generation module is used for generating a file list by a data provider according to the transmission requirement of a data receiver and storing the file list in a relational database;
the file generation module is used for generating a synchronous file under the FTP/SFTP directory of the data receiving party circularly by the data providing party according to the file list, and if the synchronous file is generated successfully, the file list state is updated to be the file generation success and the file is named according to the naming rule; if the synchronous file is failed to generate, updating the file list state to be the file generation failure and regenerating the synchronous file;
the file downloading module is used for circularly downloading the synchronous files to the local in a concurrent mode under the corresponding FTP/SFTP directory by the data receiving party according to the file list, and if the downloading is successful, the state of updating the file list is successful; if the downloading fails, updating the file list state to be downloading failure and generating a receipt file; regenerating a new file list by the data provider receipt file;
the file auditing module is used for auditing the downloaded synchronous files after the synchronous files are downloaded by the data receiving party, and if the auditing is successful, the state of the updated file list is that the auditing is successful; if the audit fails, updating the list state of the file list to be the audit failure and generating a response piece file; the data provider performs corresponding processing according to the receipt file;
the receipt storage module is used for placing the generated receipt files into a specified receipt directory after the audit is finished and describing the transmission completion condition of each synchronous file;
and the file warehousing module is used for warehousing the synchronous files after the audit is finished into a data warehouse tool or a file system by the data receiving party.
It should be noted that, a synchronous transmission system for large-batch files of the present invention corresponds to a synchronous transmission method for large-batch files of the present invention one-to-one, and the technical features and the advantageous effects described in the embodiment of the synchronous transmission method for large-batch files are all applicable to the embodiment of the synchronous transmission system for large-batch files.
In addition, in the implementation of the synchronous transmission system for large-batch files in the above embodiment, the logical division of each program module is only an example, and in practical applications, the above function distribution may be performed by different program modules according to needs, for example, due to the configuration requirement of corresponding hardware or the convenience of implementation of software, that is, the internal structure of the synchronous transmission system for large-batch files is divided into different program modules to perform all or part of the above described functions.
Referring to fig. 3, in an embodiment, an electronic device for implementing a method for synchronously transmitting a large batch of files is provided, where the electronic device may include a first processor, a first memory, a bus, and a computer program stored in the first memory and operable on the first processor, such as a program for synchronously transmitting a large batch of files.
Wherein the first memory comprises at least one type of readable storage medium, the readable storage medium comprises flash memory, a removable hard disk, a multimedia card, a card type memory (e.g., SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, etc. The first memory may in some embodiments be an internal storage unit of the electronic device, for example a removable hard disk of the electronic device. The first memory may also be an external storage device of the electronic device in other embodiments, such as a plug-in removable hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like provided on the electronic device. Further, the first memory may also include both an internal storage unit and an external storage device of the electronic device. The first memory may be used not only to store application software installed in the electronic device and various types of data, such as codes of a synchronous transmission program of a large-volume file, but also to temporarily store data that has been output or is to be output.
The first processor may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or may be composed of a plurality of integrated circuits packaged with the same function or different functions, including one or more Central Processing Units (CPUs), microprocessors, digital Processing chips, graphics processors, and combinations of various control chips. The first processor is a Control Unit (Control Unit) of the electronic device, connects various components of the whole electronic device by using various interfaces and lines, and executes various functions and processes data of the electronic device by running or executing programs or modules (such as a synchronous transmission program of a large batch of files, etc.) stored in the first memory and calling data stored in the first memory.
Fig. 3 shows only an electronic device having components, and those skilled in the art will appreciate that the structure shown in fig. 3 does not constitute a limitation of the electronic device, and may include fewer or more components than those shown, or some components may be combined, or a different arrangement of components.
The synchronous transmission program of the large batch of files stored in the first memory in the electronic device is a combination of a plurality of instructions, and when the synchronous transmission program runs in the first processor, the synchronous transmission program can realize that:
the data provider generates a file list according to the transmission requirement of the data receiver and stores the file list in a relational database;
according to the file list, the data provider circularly generates synchronous files under the FTP/SFTP directory of the data receiver, and if the synchronous files are successfully generated, the state of the file list is updated to be that the files are successfully generated and named according to the naming rule; if the synchronous file is failed to generate, updating the file list state to be the file generation failure and regenerating the synchronous file;
the data receiver circularly downloads the synchronous file to the local in a concurrent mode under the corresponding FTP/SFTP directory according to the file list, and if the downloading is successful, the state of the file list is updated to be the downloading success; if the downloading fails, updating the file list state into a downloading failure and generating a receipt file; regenerating a new file list by the data provider receipt file;
after the data receiver downloads the synchronous files, auditing the downloaded synchronous files, and if the auditing is successful, updating the file list state to be the auditing success; if the audit fails, updating the file list state to be the audit failure and generating a response piece file; the data provider performs corresponding processing according to the receipt file;
after the audit is finished, the generated receipt files are put into a specified receipt directory to describe the transmission completion condition of each synchronous file;
and the data receiver stores the synchronous files after the audit is finished into a data warehouse tool or a file system.
Further, the electronic device integrated module/unit, if implemented in the form of a software functional unit and sold or used as a separate product, may be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying said computer program code, recording medium, U-disk, removable hard disk, magnetic disk, optical disk, computer Memory, read-Only Memory (ROM).
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a non-volatile computer-readable storage medium, and can include the processes of the embodiments of the methods described above when the program is executed. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), rambus (Rambus) direct RAM (RDRAM), direct Rambus Dynamic RAM (DRDRAM), and Rambus Dynamic RAM (RDRAM), among others.
All possible combinations of the technical features in the above embodiments may not be described for the sake of brevity, but should be considered as being within the scope of the present disclosure as long as there is no contradiction between the combinations of the technical features.
The above embodiments are preferred embodiments of the present invention, but the present invention is not limited to the above embodiments, and any other changes, modifications, substitutions, combinations, and simplifications which do not depart from the spirit and principle of the present invention should be construed as equivalents thereof, and all such changes, modifications, substitutions, combinations, and simplifications are intended to be included in the scope of the present invention.
Claims (9)
1.A synchronous transmission method for large-batch files is characterized by comprising the following steps:
the data provider generates a file list according to the transmission requirement of the data receiver and stores the file list in a relational database;
according to the file list, the data provider circularly generates synchronous files under the FTP/SFTP directory of the data receiver, and if the synchronous files are successfully generated, the state of the file list is updated to be that the files are successfully generated and named according to the naming rule; if the generation of the synchronous file fails, updating the file list state to be the file generation failure and regenerating the synchronous file;
the data receiver circularly downloads the synchronous file to the local in a concurrent mode under the corresponding FTP/SFTP directory according to the file list, and if the downloading is successful, the state of the file list is updated to be the downloading success; if the downloading fails, updating the file list state to be downloading failure and generating a receipt file; regenerating a new file list by the data provider receipt file;
after the data receiver downloads the synchronous files, auditing the downloaded synchronous files, and if the auditing is successful, updating the file list state to be the auditing success; if the audit fails, updating the list state of the file list to be the audit failure and generating a response piece file; the data provider performs corresponding processing according to the receipt file;
after the audit is finished, the generated receipt files are put into a specified receipt directory to describe the transmission completion condition of each synchronous file;
and the data receiving party puts the synchronous files after the audit is finished into a data warehouse tool or a file system.
2. The method for synchronously transmitting the files in the large batch according to the claim 1, wherein the periodically generating the file list is divided into periodically generating the file list and generating the file list in a quasi-real time manner;
the regular generation of the file list refers to the generation of the file list by extracting data according to a daily cycle or a monthly cycle, and the extraction mode is incremental or full extraction;
the quasi-real-time generation of the file list refers to the generation of the file list according to hours or minutes;
the content of the file list comprises time, data name, synchronous files needing to be transmitted, state and interface file state;
the file list is as follows: naming in a manner of list;
the data application type is named according to the data application type;
the periods include a daily period d and a monthly period m;
the generation time refers to the generation time of the file list and is represented as YYYYMMDD.
3. The method for synchronously transmitting the bulk files according to claim 2, wherein the synchronous files comprise data files, MD5 check files and CHK check files;
splitting the data file according to the data volume and the set data record number, subpackaging and compressing;
the MD5 verification file is an MD5 verification code of the data file; each data file corresponds to an MD5 check file;
the content of the CHK check file comprises a data file name, a data file record number and a data file size;
the size of the data file is the size of the data file before compression, and the unit is BYTE.
4. The method for synchronously transmitting the large batch of files according to claim 3, wherein the naming rule is as follows: data application type _ interface sequence number _ cycle _ file time flag _ packet number;
the data application type is named according to the data application type;
the interface serial number represents the serial number of the interface in the subject domain;
the periods include a daily period d and a monthly period m;
the file time mark is determined according to a period, and if the file time mark is a day period, the file time mark is represented as YYYYMMDD; if the file is a month cycle, the file time mark is represented as YYYYMM;
the sub-package number refers to the number of the files after sub-packaging;
the file extension comprises a data file AVL, an MD5 check file MD5, a CHK check file CHK, a normal receipt file RR and a problem receipt file RE.
5. The method for synchronously transmitting the bulk files according to claim 4, wherein the content in the receipt file comprises a file number, an error code, a file name and detailed error information;
the error codes include 101, 102, 103, 104, 105, 106, and 107;
when the error code is 101, the synchronous file is not uploaded within the specified time;
when the error code is 102, the number of data files is not matched with the specified number;
when the error code is 103, the CHK check file is not matched with the specification;
when the error code is 104, the record number of the data file is inconsistent with the record number of the data file in the CHK check file;
when the error code is 105, the data file is represented as an empty file;
when the error code is 106, the CHK check file is not empty;
when the error code is 107, it indicates that the interface file fluctuation exceeds the threshold.
6. The method as claimed in claim 5, wherein auditing the downloaded synchronization files includes auditing data integrity, consistency and accuracy, and specifically comprises:
checking the integrity of data in the downloading process of the synchronous file by downloading the MD5 check file in the synchronous file; if the data file does not meet the integrity, writing a receipt file in a corresponding directory, and updating the interface file state of the file list into integrity check failure;
if the data file meets the integrity, checking the consistency of the data file by downloading a CHK check file in the synchronous file; if the data files do not meet the consistency, writing the receipt files in the corresponding directories, and updating the interface file state of the file list into consistency audit failure;
if the data files meet the consistency, checking the accuracy of the data files by downloading CHK check files in the synchronous files; if the data files are not full, the accuracy is that the receipt files are written in the corresponding directory, and the interface file state of the file list is updated to be the accuracy audit failure; if the data file meets the accuracy, the audit is successful.
7. A synchronous transmission system of bulk files is characterized in that the synchronous transmission system is applied to the synchronous transmission method of bulk files according to any one of claims 1 to 6, and comprises a list generation module, a file downloading module, a file auditing module, a receipt storage module and a file warehousing module;
the list generation module is used for generating a file list by a data provider according to the transmission requirement of a data receiver and storing the file list in a relational database;
the file generation module is used for generating a synchronous file under the FTP/SFTP directory of the data receiving party circularly by the data providing party according to the file list, and if the synchronous file is generated successfully, the file list state is updated to be the file generation success and named according to the naming rule; if the synchronous file is failed to generate, updating the file list state to be the file generation failure and regenerating the synchronous file;
the file downloading module is used for circularly downloading the synchronous files to the local in a concurrent mode under the corresponding FTP/SFTP directory by the data receiving party according to the file list, and if the downloading is successful, the state of updating the file list is that the downloading is successful; if the downloading fails, updating the file list state to be downloading failure and generating a receipt file; regenerating a new file list by the data provider receipt file;
the file auditing module is used for auditing the downloaded synchronous files after the synchronous files are downloaded by the data receiving party, and if the auditing is successful, the state of updating the file list is that the auditing is successful; if the audit fails, updating the list state of the file list to be the audit failure and generating a response piece file; the data provider performs corresponding processing according to the receipt file;
the receipt storage module is used for placing the generated receipt files into a specified receipt directory after the completion of the audit and describing the transmission completion condition of each synchronous file;
and the file warehousing module is used for warehousing the synchronous files after the audit is finished into a data warehouse tool or a file system by the data receiving party.
8. An electronic device, characterized in that the electronic device comprises:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores computer program instructions executable by the at least one processor, the computer program
The instructions are executable by the at least one processor to enable the at least one processor to perform a method of synchronous transmission of bulk files as claimed in any one of claims 1 to 6.
9. A computer-readable storage medium storing a program which, when executed by a processor, implements a method for synchronous transmission of bulk files according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211051029.3A CN115442361B (en) | 2022-08-30 | 2022-08-30 | Synchronous transmission method, system, equipment and medium for large-batch files |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211051029.3A CN115442361B (en) | 2022-08-30 | 2022-08-30 | Synchronous transmission method, system, equipment and medium for large-batch files |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115442361A true CN115442361A (en) | 2022-12-06 |
| CN115442361B CN115442361B (en) | 2024-06-04 |
Family
ID=84244758
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211051029.3A Active CN115442361B (en) | 2022-08-30 | 2022-08-30 | Synchronous transmission method, system, equipment and medium for large-batch files |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115442361B (en) |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101094026A (en) * | 2006-06-19 | 2007-12-26 | 上海全成通信技术有限公司 | Method for synchronizing, transmitting mass data, and inputting mass data to database |
| US20100050017A1 (en) * | 2008-08-25 | 2010-02-25 | Saudi Arabian Oil Company | Intelligent Field Oil and Gas Field Data Acquisition, Delivery, Control, and Retention Based Apparatus, Program Product and Related Methods |
| CN103067483A (en) * | 2012-12-25 | 2013-04-24 | 广东邮电职业技术学院 | Remote data increment synchronization method and device based on data package |
| CN103873517A (en) * | 2012-12-14 | 2014-06-18 | 中兴通讯股份有限公司 | Method, device and system for data synchronization |
| WO2015010272A1 (en) * | 2013-07-22 | 2015-01-29 | 烽火通信科技股份有限公司 | Method and system for transmission management of full configuration synchronization between eml-nml |
| CN104361128A (en) * | 2014-12-05 | 2015-02-18 | 河海大学 | Data synchronization method of PC (Personnel Computer) end and mobile terminal based on hydraulic polling business |
| US20190042303A1 (en) * | 2015-09-24 | 2019-02-07 | Wangsu Science & Technology Co.,Ltd. | Distributed storage-based file delivery system and method |
| CN110704448A (en) * | 2019-09-29 | 2020-01-17 | 上海易点时空网络有限公司 | Data synchronization method and device and server |
| CN111654522A (en) * | 2020-04-27 | 2020-09-11 | 平安证券股份有限公司 | File synchronization method, file synchronization server and storage medium |
| CN114003659A (en) * | 2021-11-05 | 2022-02-01 | 中国工商银行股份有限公司 | Data synchronization method, data synchronization device, electronic equipment, storage medium and program product |
-
2022
- 2022-08-30 CN CN202211051029.3A patent/CN115442361B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101094026A (en) * | 2006-06-19 | 2007-12-26 | 上海全成通信技术有限公司 | Method for synchronizing, transmitting mass data, and inputting mass data to database |
| US20100050017A1 (en) * | 2008-08-25 | 2010-02-25 | Saudi Arabian Oil Company | Intelligent Field Oil and Gas Field Data Acquisition, Delivery, Control, and Retention Based Apparatus, Program Product and Related Methods |
| CN103873517A (en) * | 2012-12-14 | 2014-06-18 | 中兴通讯股份有限公司 | Method, device and system for data synchronization |
| CN103067483A (en) * | 2012-12-25 | 2013-04-24 | 广东邮电职业技术学院 | Remote data increment synchronization method and device based on data package |
| WO2015010272A1 (en) * | 2013-07-22 | 2015-01-29 | 烽火通信科技股份有限公司 | Method and system for transmission management of full configuration synchronization between eml-nml |
| CN104361128A (en) * | 2014-12-05 | 2015-02-18 | 河海大学 | Data synchronization method of PC (Personnel Computer) end and mobile terminal based on hydraulic polling business |
| US20190042303A1 (en) * | 2015-09-24 | 2019-02-07 | Wangsu Science & Technology Co.,Ltd. | Distributed storage-based file delivery system and method |
| CN110704448A (en) * | 2019-09-29 | 2020-01-17 | 上海易点时空网络有限公司 | Data synchronization method and device and server |
| CN111654522A (en) * | 2020-04-27 | 2020-09-11 | 平安证券股份有限公司 | File synchronization method, file synchronization server and storage medium |
| CN114003659A (en) * | 2021-11-05 | 2022-02-01 | 中国工商银行股份有限公司 | Data synchronization method, data synchronization device, electronic equipment, storage medium and program product |
Non-Patent Citations (2)
| Title |
|---|
| 임민규: "《A File Synchronization Framework Based on Rsync Protocol for Cloud Storage Services》", 《 전기학회논문지》, 25 August 2022 (2022-08-25) * |
| 吴文怡: "《基于广电融媒云平台的专业级摄像机移动快传系统设计与应用》", 《广播与电视技术》, 15 May 2022 (2022-05-15) * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115442361B (en) | 2024-06-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106933703B (en) | Database data backup method and device and electronic equipment | |
| CN111444196B (en) | Method, device and equipment for generating Hash of global state in block chain type account book | |
| CN111026767B (en) | Block chain data storage method and device and hardware equipment | |
| JP2006236328A (en) | File system item and serialization of relevant entity | |
| CN107992492B (en) | Data block storage method, data block reading method, data block storage device, data block reading device and block chain | |
| CN113495903B (en) | Electric power time sequence database caching method, system, equipment and readable storage medium | |
| US7770065B2 (en) | File data restoring system and method of computer operating system and software thereof | |
| CN113312338A (en) | Data consistency checking method, device, equipment, medium and program product | |
| CN113468118B (en) | File increment storage method, device and storage medium based on blockchain | |
| CN114117543A (en) | PDF document multi-page batch signature verification method and device based on shared signature value mode | |
| CN113220777B (en) | Service data processing method, device, computer equipment and storage medium | |
| CN115442361A (en) | Synchronous transmission method, system, equipment and medium for large-batch files | |
| CN116644122A (en) | Data transaction processing method, device, computer equipment and storage medium | |
| CN111159985A (en) | Data export method, data export device, computer equipment and computer-readable storage medium | |
| CN110866068A (en) | Announcement data storage method and device based on HDFS | |
| CN114090547A (en) | Data migration method and device, computer equipment and storage medium | |
| CN112070610B (en) | Method, node, system, electronic device and medium for verifying validity of block | |
| CN115018569A (en) | Invoice issuing method, invoice reimbursement device and equipment based on block chain | |
| CN113342774A (en) | Data processing method and device, computer readable medium and electronic equipment | |
| CN113791922B (en) | Exception handling method, system and device for distributed storage system | |
| CN113313540B (en) | Contract generation method and device, electronic equipment and readable storage medium | |
| CN117591577B (en) | Nuclear power historical data comparison method and system based on file storage | |
| CN117009143A (en) | Object storage data backup method and device and storage medium | |
| CN115357388A (en) | Large-data-volume Excel file export method and system based on streaming reading | |
| CN117290415A (en) | Data loading method, device, computer equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |