CN115442361B - Synchronous transmission method, system, equipment and medium for large-batch files - Google Patents
Synchronous transmission method, system, equipment and medium for large-batch files Download PDFInfo
- Publication number
- CN115442361B CN115442361B CN202211051029.3A CN202211051029A CN115442361B CN 115442361 B CN115442361 B CN 115442361B CN 202211051029 A CN202211051029 A CN 202211051029A CN 115442361 B CN115442361 B CN 115442361B
- Authority
- CN
- China
- Prior art keywords
- file
- data
- files
- list
- synchronous
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000001360 synchronised effect Effects 0.000 title claims abstract description 116
- 230000005540 biological transmission Effects 0.000 title claims abstract description 44
- 238000000034 method Methods 0.000 title claims abstract description 36
- 238000012545 processing Methods 0.000 claims description 11
- 238000000605 extraction Methods 0.000 claims description 9
- 230000000737 periodic effect Effects 0.000 claims description 8
- 238000012550 audit Methods 0.000 claims description 7
- 238000004590 computer program Methods 0.000 claims description 7
- 230000001172 regenerating effect Effects 0.000 claims description 7
- 230000008569 process Effects 0.000 claims description 6
- 238000012546 transfer Methods 0.000 claims description 4
- 230000006835 compression Effects 0.000 claims description 3
- 238000007906 compression Methods 0.000 claims description 3
- 230000037211 monthly cycles Effects 0.000 claims description 3
- 238000004806 packaging method and process Methods 0.000 claims description 3
- 230000006870 function Effects 0.000 description 4
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000005856 abnormality Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a synchronous transmission method, a system, equipment and a medium for a large number of files, wherein the method comprises the following steps: the data provider generates a file list; generating a synchronous file under the FTP/SFTP catalog of the data receiver according to the file list; the data receiver downloads the synchronous file to the local in parallel to the corresponding FTP/SFTP catalogue, and if the downloading is successful, the file list state is updated to be successful; if the downloading fails, updating the file list state as the downloading fails and generating a receipt file; the data provider regenerates a new file list according to the receipt file; the data receiver carries out auditing on the downloaded synchronous files, and if the auditing is successful, the file list state is updated as the auditing is successful; if the auditing fails, updating the file list state as auditing fails and generating a receipt file; placing the generated receipt file into a designated receipt directory; the synchronous files are put into a data warehouse tool or a file system; the synchronous and reliable transmission of a large number of files is realized.
Description
Technical Field
The invention belongs to the technical field of file synchronous transmission, and particularly relates to a method, a system, equipment and a medium for synchronously transmitting a large number of files.
Background
With the advent of the big data age, business lines of enterprises produce massive data, the data form details and index data after unified ETL (extraction, conversion and processing) in an enterprise center data warehouse, and in order to unify caliber and increase the utilization rate of the data, providing data for other departments or big data items of the enterprises according to a certain period is a very common application scene, but the two problems are brought: first, in the face of large-scale data, the data provider provides the data in what reliable and efficient manner; second, how to let the data receiver correctly receive the data to ensure that the data is complete and accurate. In the synchronous scheme in the synchronous transmission field, the conventional method is generally suitable for the transmission of small data volume of a relational database, is somewhat inexperienced in mass data transmission, and has no effective guarantee on the integrity and accuracy of data.
Disclosure of Invention
The invention mainly aims to overcome the defects and shortcomings of the prior art and provide a method, a system, equipment and a medium for synchronously transmitting a large number of files.
In order to achieve the above purpose, the present invention adopts the following technical scheme:
in one aspect, the invention provides a method for synchronously transmitting a large number of files, which comprises the following steps:
The data provider generates a file list according to the transmission requirement of the data receiver and periodically and stores the file list in a relational database;
Generating synchronous files under the FTP/SFTP catalogues of the data receiver in a circulating way by the data provider according to the file list, and updating the file list state to be successful in generating the files and naming the files according to naming rules if the synchronous files are generated successfully; if the synchronous file generation fails, updating the file list state to be the file generation failure and regenerating the synchronous file;
According to the file list, the data receiver circularly downloads the synchronous file to the local in a concurrent mode under the corresponding FTP/SFTP directory, and if the downloading is successful, the file list state is updated to be successful; if the downloading fails, updating the file list state as the downloading fails and generating a receipt file; the receipt file of the data provider regenerates a new file list;
After the data receiver downloads the synchronous file, auditing the downloaded synchronous file, and updating the list state of the file list to be successful if auditing is successful; if the auditing fails, updating the file list state as auditing fails and generating a receipt file; the data provider carries out corresponding processing according to the receipt file;
After the auditing is finished, the generated receipt files are put into a designated receipt directory, and the transmission completion condition of each synchronous file is described;
And the data receiver stores the synchronized file after the auditing is completed into a data warehouse tool or a file system.
As an optimal technical scheme, the periodically generated file list is divided into a periodically generated file list and a quasi-real-time generated file list;
The periodic generation of the file list means that the file list is generated by extracting data according to a daily cycle or a monthly cycle, and the extraction mode is divided into increment or total extraction;
generating a file list in near real time means generating the file list according to hours or minutes;
The file list comprises time, data name, synchronous files to be transmitted, states and interface file states;
the file list is as follows: naming the data application type_cycle_generation time in a list mode;
The data application type is named according to the data application type;
the period comprises a daily period d and a monthly period m;
The generation time refers to that the generation time of the file list is denoted YYYYMMDD.
As a preferable technical scheme, the synchronous file comprises a data file, an MD5 check file and a CHK check file;
splitting the data file according to the data quantity and the set data record number, and then sub-packaging and compressing the data file;
the MD5 check file is an MD5 check code of the data file; each data file corresponds to an MD5 check file;
The CHK check file comprises a data file name, a data file record number and a data file size;
The size of the data file is the size of the data file before compression, and the unit is BYTE.
As a preferable technical solution, the naming rule is: file extension;
The data application type is named according to the data application type;
The interface serial number represents the serial number of the interface in the theme zone;
the period comprises a daily period d and a monthly period m;
the file time mark is determined according to the period, and if the file time mark is a daily period, the file time mark is expressed as YYYYMMDD; if the time is a month period, the file time mark is expressed as YYYYMM;
the sub-package number refers to the file number after sub-package;
The file extension comprises data files AVL, MD5 check files MD5, CHK check files CHK, normal receipt files RR and problem receipt files RE.
As a preferable technical scheme, the content in the receipt file comprises a file number, an error code, a file name and detailed error information;
The error code includes 101, 102, 103, 104, 105, 106, and 107;
When the error code is 101, the synchronous file is not uploaded in a specified time;
when the error code is 102, the number of the data files is not matched with the stipulation;
When the error code is 103, the CHK check file is not matched with the stipulation;
When the error code is 104, the record number of the data file is inconsistent with the record number of the data file in the CHK check file;
when the error code is 105, the data file is represented as an empty file;
when the error code is 106, the CHK check file is not empty;
When the error code is 107, it indicates that the interface file fluctuates beyond a threshold.
As an optimized technical scheme, the auditing of the downloaded synchronous file comprises auditing the integrity, consistency and accuracy of data, and specifically comprises the following steps:
checking the data integrity in the downloading process of the synchronous file by downloading an MD5 check file in the synchronous file; if the data file does not meet the integrity, writing a receipt file under the corresponding directory, and updating the interface file state of the file list to be an integrity audit failure;
If the data file meets the integrity, checking the consistency of the data file by downloading a CHK check file in the synchronous file; if the data files do not meet the consistency, writing the receipt files under the corresponding catalogues, and updating the interface file states of the file list to be consistency auditing failure;
If the data files meet the consistency, checking the accuracy of the data files by downloading CHK check files in the synchronous files; if the data file is not full, writing a receipt file under the corresponding directory, and updating the interface file state of the file list to be an accuracy audit failure; if the data file meets the accuracy, the auditing is successful.
On the other hand, the invention provides a synchronous transmission system of a large number of files, which is applied to the synchronous transmission method of the large number of files, and comprises a list generation module, a file downloading module, a file auditing module, a receipt storage module and a file warehousing module;
The list generation module is used for generating a file list according to the transmission requirement of the data receiver by the data provider according to the period and storing the file list in the relational database;
the file generation module is used for generating synchronous files under the FTP/SFTP catalogue of the data receiver in a circulating way according to the file list, and updating the file list state to be successful in generating the files and naming the files according to naming rules if the synchronous files are generated successfully; if the synchronous file generation fails, updating the file list state to be the file generation failure and regenerating the synchronous file;
The file downloading module is used for the data receiver to download the synchronous file to the local in a concurrent mode from the corresponding FTP/SFTP directory according to the file list, and if the downloading is successful, the file list state is updated to be successful; if the downloading fails, updating the file list state as the downloading fails and generating a receipt file; the receipt file of the data provider regenerates a new file list;
The file auditing module is used for auditing the downloaded synchronous file after the synchronous file is downloaded by the data receiver, and updating the list state of the file list to be successful if the auditing is successful; if the auditing fails, updating the file list state as auditing fails and generating a receipt file; the data provider carries out corresponding processing according to the receipt file;
the receipt storage module is used for placing the generated receipt files into a designated receipt directory after the completion of auditing, and describing the transmission completion condition of each synchronous file;
The file storage module is used for storing the synchronous files after the auditing is completed to a data warehouse tool or a file system by the data receiver.
In still another aspect, the present invention further provides an electronic device, including:
at least one processor; and
A memory communicatively coupled to the at least one processor; wherein,
The memory stores computer program instructions executable by the at least one processor, the computer program
Instructions are executed by the at least one processor to enable the at least one processor to perform a method of simultaneous transfer of bulk files as described above.
In yet another aspect, the present invention provides a computer readable storage medium storing a program which, when executed by a processor, implements a method for synchronously transferring a large number of files as described above.
Compared with the prior art, the invention has the following advantages and beneficial effects:
When the data synchronization is carried out, the method can realize the transmission of mass data by means of data file staging, classification and sub-packaging, and ensure the high efficiency of the transmission process; the specific transmission condition of each file is known through the receipt files, and the data with problems in the transmission process are rapidly positioned and processed, so that the reliability of the transmission process is ensured; through the audit inspection of the MD5 check file and the CHK check file, the data is not lost in the transmission process, and the integrity and accuracy of the data are ensured.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flowchart of a method for synchronously transmitting a large number of files according to an embodiment of the present invention;
fig. 2 is a block diagram of a synchronous transmission system for a large number of files according to an embodiment of the present invention.
Fig. 3 is a block diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to enable those skilled in the art to better understand the present application, the following description will make clear and complete descriptions of the technical solutions according to the embodiments of the present application with reference to the accompanying drawings. It will be apparent that the described embodiments are only some, but not all, embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of skill in the art will explicitly and implicitly appreciate that the described embodiments of the application may be combined with other embodiments.
Referring to fig. 1, in one embodiment of the present application, a method for synchronously transmitting a large number of files is provided, including the following steps:
s1, a data provider generates a file list according to a period and stores the file list in a relational database according to transmission requirements of a data receiver;
According to the transmission requirement of a data receiver, a data provider generates a file list according to a period, and the file list is divided into a periodically generated file list and a quasi-real-time generated file list; the periodic generation of the file list means that the file list is generated by extracting data according to a daily period or a monthly period; the extraction mode is divided into increment or total extraction;
Generating a file list in quasi-real time means generating the file list according to hours or minutes, and timely extracting and transmitting changed data;
In the invention, the contents of the file list comprise time, data name, synchronous file to be transmitted, state and interface file state; the file list is as follows: naming the data application type_cycle_generation time in a list mode; the data application type is named according to the data application type; the period comprises a day period d and a month period m; the generation time refers to that the generation time of the file list is YYYYMMDD; the file list of driving date data generated by 20220408 is indicated as file list drive_d_20220408. List.
S2, generating synchronous files under the FTP/SFTP catalogues of the data receiver in a circulating mode by the data provider according to the file list, and updating the file list state to be successful in generating the files and naming the files according to naming rules if the synchronous files are generated successfully; if the synchronous file generation fails, updating the file list state to be the file generation failure and regenerating the synchronous file;
In the invention, the synchronous file comprises a data file, an MD5 check file and a CHK check file; the data file is split according to the data quantity and the set data record number, and then is sub-packaged and compressed, so that the data transmission efficiency is ensured; the MD5 check file is an MD5 check code of the data file; each data file must correspond to an MD5 check file; the CHK check file comprises a data file name, a data file record number and a data file size; the data file size is the size of the data file before compression in BYTE. In this embodiment, splitting is performed according to the number of data records per 500 ten thousand.
The generated file is named according to a naming rule, wherein the naming rule is' data application type_interface serial number_cycle_file time mark_sub packet number;
The data application type is named according to the data application type, for example, driving data is named as drive, and vehicle condition data is named as car-condition;
The interface serial number represents the serial number of the interface in the subject domain, such as 00001;
The period includes a daily period "d" and a monthly period "m";
the file time mark is determined according to the period, and if the file time mark is a daily period, the file time mark is expressed as YYYYMMDD; if the time is a month period, the file time mark is expressed as YYYYMM;
The subcontracting number refers to the file number after subcontracting;
The file extension includes data file AVL, MD5 check file MD5, CHK check file CHK, normal receipt file RR, and problem receipt file RE.
For example: for a file with a file name of drive_00001_d_20220408_01.Avl, a data file representing day data with a billing period 20220408 for driving data number 00001, and a packet number 01;
for a file with a file name drive_00001_d_20220408_01.md5, which represents an MD5 check file of day data with a billing period 20220408 for driving data number 00001, a packet number 01;
For a file with a file name of drive_00001_d_20220408_01.CHK, which represents a CHK check file of day data with a billing period 20220408 for driving data number 00001, a packet number is 01;
For a file with a file name of drive_00001_d_20220408_01.Rr, it represents a receipt file of day data with a billing period 20220408 of a data warehouse number 00001, and a package number of 01 represents that the interface file is normal;
The file named drive_00001_d_20220408_01.Re is a receipt file of day data with a billing period 20220408 of a data warehouse number 00001, and a package number 01 is an interface file abnormality.
S3, the data receiver downloads the synchronous file to the local in a concurrent mode under the corresponding FTP/SFTP directory circularly according to the file list, and if the downloading is successful, the file list state is updated as the downloading is successful; if the downloading fails, updating the file list state as the downloading fails and generating a receipt file; the data provider regenerates a new file list according to the receipt file;
s4, after the data receiver downloads the synchronous file, auditing the downloaded synchronous file, and updating the list state of the file list to be successful if the auditing is successful; if the auditing fails, updating the file list state as auditing fails and generating a receipt file; the data provider carries out corresponding processing according to the receipt file;
the downloaded synchronous file is subjected to data integrity, consistency and accuracy auditing, and the method specifically comprises the following steps:
checking the data integrity in the downloading process of the synchronous file by downloading an MD5 check file in the synchronous file; if the data file does not meet the integrity, writing a receipt file under the corresponding directory, and updating the interface file state of the file list to be an integrity audit failure;
If the data file meets the integrity, checking the consistency of the data file by downloading a CHK check file in the synchronous file; if the data files do not meet the consistency, writing the receipt files under the corresponding catalogues, and updating the interface file states of the file list to be consistency auditing failure;
If the data files meet the consistency, checking the accuracy of the data files by downloading CHK check files in the synchronous files; if the data file is not full, writing a receipt file under the corresponding directory, and updating the interface file state of the file list to be an accuracy audit failure; if the data file meets the accuracy, the auditing is successful.
S5, after the auditing is completed, the generated receipt files are put into a designated receipt directory, and the transmission completion condition of each synchronous file is described;
when the data file is inconsistent with the check file data, the error information is embodied in the receipt content of the data file, and the content in the receipt file comprises a file number, an error code, a file name and detailed error information;
Error codes include 101, 102, 103, 104, 105, 106, and 107; when the error code is 101, the synchronous file is not uploaded in a specified time; when the error code is 102, the number of the data files is not matched with the stipulation; when the error code is 103, the CHK check file is not matched with the stipulation; when the error code is 104, the record number of the data file is inconsistent with the record number of the data file in the CHK check file; when the error code is 105, the data file is represented as an empty file; when the error code is 106, the CHK check file is not empty; when the error code is 107, it indicates that the interface file fluctuates beyond a threshold.
S6, the data receiver stores the synchronized file after the auditing is completed into a data warehouse tool or a file system.
It should be noted that, for the sake of simplicity of description, the foregoing method embodiments are all expressed as a series of combinations of actions, but it should be understood by those skilled in the art that the present invention is not limited by the order of actions described, as some steps may be performed in other order or simultaneously in accordance with the present invention.
Based on the same ideas of the method for synchronously transmitting the large-batch files in the embodiment, the invention also provides a system for synchronously transmitting the large-batch files, which can be used for executing the method for synchronously transmitting the large-batch files. For ease of illustration, only those portions of an embodiment of a synchronous transfer system for bulk documents are shown in the schematic configuration of an embodiment of the present invention, and those skilled in the art will appreciate that the illustrated configuration is not limiting of the apparatus and may include more or fewer components than illustrated, or may combine certain components, or a different arrangement of components.
Referring to fig. 2, in another embodiment of the present application, a synchronous transmission system for a large number of files is provided, where the system includes a manifest generating module, a file downloading module, a file auditing module, a receipt storing module, and a file warehousing module;
The list generation module is used for generating a file list according to the transmission requirement of the data receiver by the data provider according to the period, and storing the file list in the relational database;
The file generation module is used for generating synchronous files under the FTP/SFTP catalogue of the data receiver in a circulating way according to the file list, and updating the file list state to be successful in generating the files and naming the files according to naming rules if the synchronous files are successful in generating the synchronous files; if the synchronous file generation fails, updating the file list state to be the file generation failure and regenerating the synchronous file;
the file downloading module is used for the data receiver to download the synchronous file to the local in a concurrent mode under the corresponding FTP/SFTP directory circularly according to the file list, and if the downloading is successful, the file list state is updated as the downloading is successful; if the downloading fails, updating the file list state as the downloading fails and generating a receipt file; the receipt file of the data provider regenerates a new file list;
The file auditing module is used for auditing the downloaded synchronous file after the synchronous file is downloaded by the data receiver, and updating the list state of the file list to be successful after the auditing is successful; if the auditing fails, updating the file list state as auditing fails and generating a receipt file; the data provider carries out corresponding processing according to the receipt file;
The receipt storage module is used for placing the generated receipt files into a designated receipt directory after the completion of auditing, and describing the transmission completion condition of each synchronous file;
the file storage module is used for the data receiver to store the synchronized file after the auditing is completed into a data warehouse tool or a file system.
It should be noted that, the technical features and the beneficial effects described in the embodiments of the method for synchronously transmitting a large number of files are applicable to the embodiments of the system for synchronously transmitting a large number of files, and specific content can be seen from the description in the embodiments of the method of the present invention, which is not repeated here.
In addition, in the implementation of the system for synchronous transmission of bulk files according to the foregoing embodiment, the logic division of each program module is merely illustrative, and in practical application, the allocation of the functions may be performed by different program modules according to needs, for example, in view of configuration requirements of corresponding hardware or convenience of implementation of software, that is, the internal structure of the system for synchronous transmission of bulk files is divided into different program modules, so as to perform all or part of the functions described above.
Referring to fig. 3, in one embodiment, an electronic device for implementing a method for synchronously transferring a large batch of files is provided, where the electronic device may include a first processor, a first memory, and a bus, and may further include a computer program stored in the first memory and executable on the first processor, such as a synchronous transfer program of a large batch of files.
Wherein the first memory comprises at least one type of readable storage medium including flash memory, a removable hard disk, a multimedia card, a card memory (e.g., SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, etc. The first memory may in some embodiments be an internal storage unit of the electronic device, such as a mobile hard disk of the electronic device. The first memory may in other embodiments also be an external storage device of the electronic device, such as a plug-in mobile hard disk, a smart memory card (SMART MEDIA CARD, SMC), a secure digital (SecureDigital, SD) card, a flash memory card (FLASH CARD) or the like, provided on the electronic device. Further, the first memory may also include both an internal memory unit and an external memory device of the electronic device. The first memory may be used not only to store application software installed in an electronic device and various types of data, such as codes of a synchronous transmission program of a large-batch file, but also to temporarily store data that has been output or is to be output.
The first processor may be comprised of integrated circuits in some embodiments, for example, a single packaged integrated circuit, or may be comprised of multiple integrated circuits packaged with the same or different functions, including one or more central processing units (Central Processing unit, CPU), microprocessors, digital processing chips, graphics processors, combinations of various control chips, and the like. The first processor is a Control Unit (Control Unit) of the electronic device, connects respective components of the entire electronic device using various interfaces and lines, executes or executes programs or modules (e.g., a synchronous transfer program of a large-batch file, etc.) stored in the first memory, and invokes data stored in the first memory to perform various functions of the electronic device and process the data.
Fig. 3 shows only an electronic device with components, and it will be understood by those skilled in the art that the structure shown in fig. 3 is not limiting of the electronic device and may include fewer or more components than shown, or may combine certain components, or a different arrangement of components.
The synchronous transmission program of the large-batch files stored in the first memory in the electronic device is a combination of a plurality of instructions, and when running in the first processor, the synchronous transmission program can realize:
the data provider generates a file list according to the transmission requirement of the data receiver and stores the file list in a relational database according to the period;
Generating synchronous files under the FTP/SFTP catalogues of the data receiver in a circulating way by the data provider according to the file list, and updating the file list state to be successful in generating the files and naming the files according to naming rules if the synchronous files are generated successfully; if the synchronous file generation fails, updating the file list state to be the file generation failure and regenerating the synchronous file;
According to the file list, the data receiver circularly downloads the synchronous file to the local in a concurrent mode under the corresponding FTP/SFTP directory, and if the downloading is successful, the file list state is updated to be successful; if the downloading fails, updating the file list state as the downloading fails and generating a receipt file; the receipt file of the data provider regenerates a new file list;
After the data receiver downloads the synchronous file, auditing the downloaded synchronous file, and updating the list state of the file list to be successful if auditing is successful; if the auditing fails, updating the file list state as auditing fails and generating a receipt file; the data provider carries out corresponding processing according to the receipt file;
After the auditing is finished, the generated receipt files are put into a designated receipt directory, and the transmission completion condition of each synchronous file is described;
And the data receiver stores the synchronized file after the auditing is completed into a data warehouse tool or a file system.
Further, the electronic device integrated modules/units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a non-volatile computer readable storage medium. The computer readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM).
Those skilled in the art will appreciate that all or part of the processes in the methods of the above embodiments may be implemented by a computer program for instructing relevant hardware, where the program may be stored in a non-volatile computer readable storage medium, and where the program, when executed, may include processes in the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in embodiments provided herein may include non-volatile and/or volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (SYNCHLINK) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), among others.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above examples are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above examples, and any other changes, modifications, substitutions, combinations, and simplifications that do not depart from the spirit and principle of the present invention should be made in the equivalent manner, and the embodiments are included in the protection scope of the present invention.
Claims (8)
1. The synchronous transmission method of the large-batch files is characterized by comprising the following steps of:
The data provider generates a file list according to the transmission requirement of the data receiver and periodically and stores the file list in a relational database;
The periodic file list generation list is divided into a periodic file list generation list and a quasi-real-time file list generation list;
The periodic generation of the file list means that the file list is generated by extracting data according to a daily cycle or a monthly cycle, and the extraction mode is divided into increment or total extraction;
generating a file list in near real time means generating the file list according to hours or minutes;
The contents of the file list comprise time, data name, synchronous files to be transmitted, states and interface file states;
the file list is as follows: naming the data application type_cycle_generation time in a list mode;
The data application type is named according to the data application type;
the period comprises a daily period d and a monthly period m;
the generation time refers to that the generation time of the file list is YYYYMMDD;
Generating synchronous files under the FTP/SFTP catalogues of the data receiver in a circulating way by the data provider according to the file list, and updating the file list state to be successful in generating the files and naming the files according to naming rules if the synchronous files are generated successfully; if the synchronous file generation fails, updating the file list state to be the file generation failure and regenerating the synchronous file;
According to the file list, the data receiver circularly downloads the synchronous file to the local in a concurrent mode under the corresponding FTP/SFTP directory, and if the downloading is successful, the file list state is updated to be successful; if the downloading fails, updating the file list state as the downloading fails and generating a receipt file; the receipt file of the data provider regenerates a new file list;
After the data receiver downloads the synchronous file, auditing the downloaded synchronous file, and updating the list state of the file list to be successful if auditing is successful; if the auditing fails, updating the file list state as auditing fails and generating a receipt file; the data provider carries out corresponding processing according to the receipt file;
After the auditing is finished, the generated receipt files are put into a designated receipt directory, and the transmission completion condition of each synchronous file is described;
And the data receiver stores the synchronized file after the auditing is completed into a data warehouse tool or a file system.
2. The method for simultaneous transmission of bulk files according to claim 1, wherein said simultaneous files comprise data files, MD5 check files and CHK check files;
splitting the data file according to the data quantity and the set data record number, and then sub-packaging and compressing the data file;
the MD5 check file is an MD5 check code of the data file; each data file corresponds to an MD5 check file;
The CHK check file comprises a data file name, a data file record number and a data file size;
The size of the data file is the size of the data file before compression, and the unit is BYTE.
3. The method for synchronously transmitting a plurality of files according to claim 2, wherein the naming rule is: file extension;
The data application type is named according to the data application type;
The interface serial number represents the serial number of the interface in the theme zone;
the period comprises a daily period d and a monthly period m;
the file time mark is determined according to the period, and if the file time mark is a daily period, the file time mark is expressed as YYYYMMDD; if the time is a month period, the file time mark is expressed as YYYYMM;
the sub-package number refers to the file number after sub-package;
The file extension comprises data files AVL, MD5 check files MD5, CHK check files CHK, normal receipt files RR and problem receipt files RE.
4. A method for synchronizing transmission of a plurality of documents according to claim 3, wherein the contents of the receipt document include a document number, an error code, a document name and detailed error information;
The error code includes 101, 102, 103, 104, 105, 106, and 107;
When the error code is 101, the synchronous file is not uploaded in a specified time;
when the error code is 102, the number of the data files is not matched with the stipulation;
When the error code is 103, the CHK check file is not matched with the stipulation;
When the error code is 104, the record number of the data file is inconsistent with the record number of the data file in the CHK check file;
when the error code is 105, the data file is represented as an empty file;
when the error code is 106, the CHK check file is not empty;
When the error code is 107, it indicates that the interface file fluctuates beyond a threshold.
5. The method for synchronously transmitting a large number of files according to claim 4, wherein auditing the downloaded synchronous files includes auditing data integrity, consistency and accuracy, specifically:
checking the data integrity in the downloading process of the synchronous file by downloading an MD5 check file in the synchronous file; if the data file does not meet the integrity, writing a receipt file under the corresponding directory, and updating the interface file state of the file list to be an integrity audit failure;
If the data file meets the integrity, checking the consistency of the data file by downloading a CHK check file in the synchronous file; if the data files do not meet the consistency, writing the receipt files under the corresponding catalogues, and updating the interface file states of the file list to be consistency auditing failure;
If the data files meet the consistency, checking the accuracy of the data files by downloading CHK check files in the synchronous files; if the data file is not full, writing a receipt file under the corresponding directory, and updating the interface file state of the file list to be an accuracy audit failure; if the data file meets the accuracy, the auditing is successful.
6. The synchronous transmission system of a large number of files is characterized in that the synchronous transmission method applied to the large number of files in any one of claims 1-5 comprises a list generation module, a file download module, a file auditing module, a receipt storage module and a file warehousing module;
The list generation module is used for generating a file list according to the transmission requirement of the data receiver by the data provider according to the period and storing the file list in the relational database;
The periodic file list generation list is divided into a periodic file list generation list and a quasi-real-time file list generation list;
The periodic generation of the file list means that the file list is generated by extracting data according to a daily cycle or a monthly cycle, and the extraction mode is divided into increment or total extraction;
generating a file list in near real time means generating the file list according to hours or minutes;
The contents of the file list comprise time, data name, synchronous files to be transmitted, states and interface file states;
the file list is as follows: naming the data application type_cycle_generation time in a list mode;
The data application type is named according to the data application type;
the period comprises a daily period d and a monthly period m;
the generation time refers to that the generation time of the file list is YYYYMMDD;
the file generation module is used for generating synchronous files under the FTP/SFTP catalogue of the data receiver in a circulating way according to the file list, and updating the file list state to be successful in generating the files and naming the files according to naming rules if the synchronous files are generated successfully; if the synchronous file generation fails, updating the file list state to be the file generation failure and regenerating the synchronous file;
The file downloading module is used for the data receiver to download the synchronous file to the local in a concurrent mode from the corresponding FTP/SFTP directory according to the file list, and if the downloading is successful, the file list state is updated to be successful; if the downloading fails, updating the file list state as the downloading fails and generating a receipt file; the receipt file of the data provider regenerates a new file list;
The file auditing module is used for auditing the downloaded synchronous file after the synchronous file is downloaded by the data receiver, and updating the list state of the file list to be successful if the auditing is successful; if the auditing fails, updating the file list state as auditing fails and generating a receipt file; the data provider carries out corresponding processing according to the receipt file;
the receipt storage module is used for placing the generated receipt files into a designated receipt directory after the completion of auditing, and describing the transmission completion condition of each synchronous file;
The file storage module is used for storing the synchronous files after the auditing is completed to a data warehouse tool or a file system by the data receiver.
7. An electronic device, the electronic device comprising:
at least one processor; and
A memory communicatively coupled to the at least one processor; wherein,
The memory stores computer program instructions executable by the at least one processor, the computer program
Instructions to be executed by the at least one processor to enable the at least one processor to perform a method of simultaneous transfer of bulk files according to any one of claims 1-5.
8. A computer-readable storage medium storing a program, wherein the program, when executed by a processor, implements a method for simultaneous transfer of bulk files according to any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211051029.3A CN115442361B (en) | 2022-08-30 | 2022-08-30 | Synchronous transmission method, system, equipment and medium for large-batch files |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211051029.3A CN115442361B (en) | 2022-08-30 | 2022-08-30 | Synchronous transmission method, system, equipment and medium for large-batch files |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115442361A CN115442361A (en) | 2022-12-06 |
| CN115442361B true CN115442361B (en) | 2024-06-04 |
Family
ID=84244758
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211051029.3A Active CN115442361B (en) | 2022-08-30 | 2022-08-30 | Synchronous transmission method, system, equipment and medium for large-batch files |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115442361B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101094026A (en) * | 2006-06-19 | 2007-12-26 | 上海全成通信技术有限公司 | Method for synchronizing, transmitting mass data, and inputting mass data to database |
| CN103067483A (en) * | 2012-12-25 | 2013-04-24 | 广东邮电职业技术学院 | Remote data increment synchronization method and device based on data package |
| CN103873517A (en) * | 2012-12-14 | 2014-06-18 | 中兴通讯股份有限公司 | Method, device and system for data synchronization |
| WO2015010272A1 (en) * | 2013-07-22 | 2015-01-29 | 烽火通信科技股份有限公司 | Method and system for transmission management of full configuration synchronization between eml-nml |
| CN104361128A (en) * | 2014-12-05 | 2015-02-18 | 河海大学 | Data synchronization method of PC (Personnel Computer) end and mobile terminal based on hydraulic polling business |
| CN110704448A (en) * | 2019-09-29 | 2020-01-17 | 上海易点时空网络有限公司 | Data synchronization method and device and server |
| CN111654522A (en) * | 2020-04-27 | 2020-09-11 | 平安证券股份有限公司 | File synchronization method, file synchronization server and storage medium |
| CN114003659A (en) * | 2021-11-05 | 2022-02-01 | 中国工商银行股份有限公司 | Data synchronization method, data synchronization device, electronic equipment, storage medium and program product |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2385396B1 (en) * | 2008-08-25 | 2013-01-09 | Saudi Arabian Oil Company | Data acquisition in an intelligent oil and gas field |
| CN105162878B (en) * | 2015-09-24 | 2018-08-31 | 网宿科技股份有限公司 | Document distribution system based on distributed storage and method |
-
2022
- 2022-08-30 CN CN202211051029.3A patent/CN115442361B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101094026A (en) * | 2006-06-19 | 2007-12-26 | 上海全成通信技术有限公司 | Method for synchronizing, transmitting mass data, and inputting mass data to database |
| CN103873517A (en) * | 2012-12-14 | 2014-06-18 | 中兴通讯股份有限公司 | Method, device and system for data synchronization |
| CN103067483A (en) * | 2012-12-25 | 2013-04-24 | 广东邮电职业技术学院 | Remote data increment synchronization method and device based on data package |
| WO2015010272A1 (en) * | 2013-07-22 | 2015-01-29 | 烽火通信科技股份有限公司 | Method and system for transmission management of full configuration synchronization between eml-nml |
| CN104361128A (en) * | 2014-12-05 | 2015-02-18 | 河海大学 | Data synchronization method of PC (Personnel Computer) end and mobile terminal based on hydraulic polling business |
| CN110704448A (en) * | 2019-09-29 | 2020-01-17 | 上海易点时空网络有限公司 | Data synchronization method and device and server |
| CN111654522A (en) * | 2020-04-27 | 2020-09-11 | 平安证券股份有限公司 | File synchronization method, file synchronization server and storage medium |
| CN114003659A (en) * | 2021-11-05 | 2022-02-01 | 中国工商银行股份有限公司 | Data synchronization method, data synchronization device, electronic equipment, storage medium and program product |
Non-Patent Citations (2)
| Title |
|---|
| 《A File Synchronization Framework Based on Rsync Protocol for Cloud Storage Services》;임민규;《 전기학회논문지》;20220825;全文 * |
| 《基于广电融媒云平台的专业级摄像机移动快传系统设计与应用》;吴文怡;《广播与电视技术》;20220515;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115442361A (en) | 2022-12-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111078140B (en) | Nuclear power station file uploading management method and device, terminal equipment and medium | |
| US20130103982A1 (en) | Log file compression | |
| CN107992492B (en) | Data block storage method, data block reading method, data block storage device, data block reading device and block chain | |
| CN111026767B (en) | Block chain data storage method and device and hardware equipment | |
| CN113535856A (en) | Data synchronization method and system | |
| CN106445815B (en) | A kind of automated testing method and device | |
| CN115442361B (en) | Synchronous transmission method, system, equipment and medium for large-batch files | |
| CN114611153A (en) | Electronic receipt pushing system based on bank | |
| CN112905676A (en) | Data file importing method and device | |
| CN113220777B (en) | Service data processing method, device, computer equipment and storage medium | |
| CN110019347B (en) | Data processing method and device of block chain and terminal equipment | |
| CN111143463B (en) | Construction method and device of bank data warehouse based on topic model | |
| CN110866068A (en) | Announcement data storage method and device based on HDFS | |
| CN110837537A (en) | Order data synchronization method and device, computer readable storage medium and server | |
| CN114168661A (en) | Task dependence method and device based on data | |
| CN113392085A (en) | Distributed file batch processing method and platform | |
| CN108874592B (en) | Data cold standby method and system for Log-structured storage engine | |
| CN102081596B (en) | Conversion method for file format of circuit board view program | |
| CN112364611A (en) | High-performance table importing method | |
| CN118410085A (en) | Data table transmission method and device, storage medium and electronic equipment | |
| CN115174368B (en) | Method and system for generating and acquiring logs of terminal equipment of Internet of vehicles | |
| CN118012958B (en) | Method and system for controlling uplink and downlink interoperation based on blockchain | |
| CN118276784A (en) | Recording data migration method, device, equipment, storage medium and product | |
| CN115687521A (en) | Data synchronization method and device and computer equipment | |
| CN112579683A (en) | Method for efficiently accessing Tbase data in batches |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |