CN115442363A - Data file downloading and pushing method and device - Google Patents
Data file downloading and pushing method and device Download PDFInfo
- Publication number
- CN115442363A CN115442363A CN202211071954.2A CN202211071954A CN115442363A CN 115442363 A CN115442363 A CN 115442363A CN 202211071954 A CN202211071954 A CN 202211071954A CN 115442363 A CN115442363 A CN 115442363A
- Authority
- CN
- China
- Prior art keywords
- file
- task
- downloading
- target
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Transfer Between Computers (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a method and a device for downloading and pushing a data file, which can be applied to the technical field of big data or the financial field. When the method is executed, a file downloading request interface is called first, and whether a file is uploaded to an object to be stored or not is judged according to parameters of the file downloading request interface; if yes, inserting the information of the file into a downloading task table according to the parameters; when the task state in the downloading task table is a first state, downloading the task; after the task is downloaded successfully, inserting the information of the file into a push task table; and when the task state in the pushing task table is the second state, pushing the task. The method and the device have the advantages that the file downloading request interface is called, the downloading task and the pushing task are automatically stored according to the parameters of the file downloading request interface, and the task is automatically downloaded or pushed when the state is the preparation state, so that the problems of high manual operation and maintenance cost and high time cost in the scheme of responding to the file application requirements of all downstream systems in the prior art are solved.
Description
Technical Field
The application relates to the technical field of big data, in particular to a method and a device for downloading and pushing data files.
Background
The data lake receives data files of all source systems every day, and uploads the received source files to the object storage for subsequent downloading. At present, historical data files from various source systems downloaded by an audit data platform and subsequent monthly data files are newly added to a data lake, the data are uploaded to an object for storage, the subsequent audit data platform and various downstream systems apply for use at any time, and the files are downloaded from the object storage and pushed to different systems. In the face of increasing file application requirements of various downstream systems, a set of methods for realizing functions of receiving different system requests, downloading files and pushing the files to different target systems by a data lake is needed.
In the prior art, file extraction programs need to be written for different file extraction requirements respectively, or files are stored and downloaded by an operation and maintenance object to be placed under appointed directories of various systems, so that the manual operation and maintenance cost and the time cost are high in the face of increasing file application requirements of various downstream systems.
Disclosure of Invention
In view of this, the present application provides a method and an apparatus for downloading and pushing a data file, and aims to solve the problem that the manual operation and maintenance cost and the time cost are high in the prior art in response to the file application requirement of each downstream system.
In a first aspect, the present application provides a method for downloading and pushing a data file, where the method includes:
calling a file downloading request interface, and judging whether a target file is uploaded to an object for storage or not according to the parameters of the file downloading request interface;
if the target file is uploaded to the object storage, inserting the information of the target file into a downloading task table according to the parameters of the file downloading request interface;
when the target task state in the download task table is a first state, downloading the target task;
after the target task is downloaded successfully, inserting the information of the target file into a push task table;
and when the target task state in the push task table is a second state, pushing the target task.
Optionally, the method further includes:
establishing a file downloading request interface; the parameters of the interface at least comprise an application file system name, a file type and a source file name.
Optionally, before the information of the target file is inserted into the download task table, the method further includes:
and establishing a downloading task table, and recording related information of the task to be downloaded by using the downloading task table.
Optionally, before the inserting the information of the target file into the push task table, the method further includes:
and establishing a push task table, and recording related information of the task to be pushed by using the push task table.
Optionally, before the determining whether the target file is uploaded to the object storage, the method further includes:
synchronizing file information in the object storage to a first database by taking date as a unit;
and mapping the data in the first database to a second database, wherein the second database is used for inquiring whether the target file information exists in the second database when the target file information is not inquired in the first database.
In a second aspect, the present application provides a data file downloading and pushing device, where the device includes: the device comprises a judging module, a first inserting module, a first downloading module, a second inserting module and a second downloading module;
the judging module is used for calling a file downloading request interface and judging whether a target file is uploaded to an object for storage or not according to the parameters of the file downloading request interface;
the first inserting module is used for inserting the information of the target file into a downloading task table according to the parameters of the file downloading request interface under the condition that the target file is uploaded to the object storage;
the first downloading module is used for downloading the target task when the target task state in the downloading task table is a first state;
the second inserting module is used for inserting the information of the target file into a push task table after the target task is downloaded successfully;
and the second downloading module is used for pushing the target task when the target task state in the task pushing table is the second state.
Optionally, the apparatus further includes an establishing module, where the establishing module is specifically configured to:
establishing a file downloading request interface; the parameters of the interface at least comprise an application file system name, a file type and a source file name.
Optionally, before the information of the target file is inserted into the download task table, the establishing module is further configured to:
and establishing a downloading task table, and recording related information of the task to be downloaded by using the downloading task table.
Optionally, before the inserting the information of the target file into the push task table, the establishing module is further configured to:
and establishing a push task table, and recording related information of the task to be pushed by using the push task table.
Optionally, the apparatus further includes a synchronization module and a mapping module; before the judgment of whether the target file is uploaded to the object storage, the synchronization module is specifically configured to synchronize file information in the object storage to a first database in a date unit; the mapping module is specifically configured to map data in the first database to a second database, where the second database is configured to query whether target file information exists in the second database when the target file information is not queried in the first database.
The application provides a data file downloading and pushing method and device. When the method is executed, a file downloading request interface is called first, and whether a target file is uploaded to an object to be stored or not is judged according to parameters of the file downloading request interface; if the target file is uploaded to the object storage, inserting the information of the target file into a downloading task table according to the parameters of the file downloading request interface; when the target task state in the downloading task table is a first state, downloading the target task; after the target task is downloaded successfully, inserting the information of the target file into a push task table; and when the target task state in the push task table is a second state, pushing the target task. The method and the device have the advantages that the file downloading request interface is called, the downloading task and the pushing task are automatically stored according to the parameters of the file downloading request interface, and the task is automatically downloaded or pushed when the state is the preparation state, so that the problems of high manual operation and maintenance cost and high time cost in the scheme responding to the file application requirements of all downstream systems in the prior art are solved.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments or the prior art, the drawings used in the embodiments or the prior art descriptions will be briefly described below, and obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a flowchart of a data file downloading and pushing method according to an embodiment of the present disclosure;
fig. 2 is a schematic structural diagram of a data file downloading and pushing device according to an embodiment of the present disclosure.
Detailed Description
The data lake receives data files of all source systems every day, and uploads the received source files to the object storage for subsequent downloading. At present, historical data files from various source systems downloaded by an audit data platform and subsequent monthly data files are newly added to a data lake, the data are uploaded to an object for storage, the subsequent audit data platform and various downstream systems apply for use at any time, and the files are downloaded from the object storage and pushed to different systems. In the face of increasing file application requirements of various downstream systems, a set of methods for realizing functions of receiving different system requests, downloading files and pushing the files to different target systems by a data lake is needed.
In the prior art, file extraction programs need to be written for different file extraction requirements respectively, or files are stored and downloaded by an operation and maintenance object to be placed under specified directories of various systems, so that the manual operation and maintenance cost and the time cost are higher in the face of increasing file application requirements of various downstream systems.
In view of this, the inventor of the present application considers that if a common file download request interface can be developed and provided for each data user to call, by calling the file download interface, the data lake receives parameters transmitted by each system, queries related information of a file to be downloaded and inserts the information into a download task table, develops a related program to monitor and scan the download task table, stores the downloaded file from an object to a service processing directory, services the file, generates a push task and transfers the push task to a push directory, and a push thread performs file push to a designated directory of each data user. Therefore, automatic downloading and pushing of files can be achieved, and the problems that in the prior art, file extraction programs need to be written for different file extraction requirements respectively, or the files are stored and downloaded by an operation and maintenance object and are placed under appointed directories of systems, and accordingly manual operation and maintenance cost and time cost are high are solved.
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Referring to fig. 1, fig. 1 is a flowchart of a data file downloading and pushing method provided in an embodiment of the present application, including:
s101, calling a file downloading request interface, and judging whether a target file is uploaded to an object to be stored or not according to the parameters of the file downloading request interface.
And each data user calls a file application interface, and the microservice ms firstly searches an object storage directory obs dir and an uploading state obs status corresponding to the src file name in a file uploading object storage record table dbus obs upload info of the first database according to each system parameter value transmitted by the file downloading request interface, including an application file system name sub product name, a source file name src file name and a file Type data Type. If the query is not received, the second database is used for querying, if the query is not received, the object storage is indicated that no target file exists, and the interface returns prompt information; if the target file can be inquired, the target file exists in the object storage. The target file is a file required by the current downstream system.
And S102, if the target file is uploaded to the object storage, inserting the information of the target file into a download task table according to the parameters of the file download request interface.
If the target file information is inquired in the database and the obs status is the SUC success state, writing the src file name and the obs dir information into a dbus desktop pulldata online downloading task table of the udp dbus service library according to the parameter of the file downloading request interface, so that the service monitors, downloads and pushes the file.
S103, when the target task state in the downloading task table is the first state, downloading the target task.
The service monitors and scans the download task table, and executes the task with the task status of the first status, where the first status may be rdy (ready), and stores the download file from the object to the service processing directory, and if the download is successful, updates the task status of the table to be suc, and if the download is failed, updates the task status to be err.
And S104, inserting the information of the target file into a push task table after the target task is downloaded successfully.
And when the file is successfully downloaded from the object storage to the service processing directory, transferring the file to the push directory, inserting a file record into the push table of the dbus tasklist file, and recording the information of the src file name, the file path, the sub product name and the like.
And S105, when the target task state in the push task table is a second state, pushing the target task.
The service monitors and scans the push task table, executes a task with a push status in a second status, wherein the second status can be an rdy (ready) status, and pushes files in a push directory to each downstream system specified directory, if the pushing is successful, the push status push _ status of the table is updated to be Suc, and if the pushing is failed, the push status push _ status of the table is updated to be err.
The embodiment of the application provides a data file downloading and pushing method. When the method is executed, a file downloading request interface is called first, and whether a target file is uploaded to an object for storage is judged according to parameters of the file downloading request interface; if the target file is uploaded to the object storage, inserting the information of the target file into a downloading task table according to the parameters of the file downloading request interface; when the target task state in the download task table is a first state, downloading the target task; after the target task is downloaded successfully, inserting the information of the target file into a push task table; and when the target task state in the push task table is a second state, pushing the target task. The method comprises the steps of calling a file downloading request interface, automatically storing a downloading task and a pushing task according to parameters of the file downloading request interface, and automatically downloading or pushing the task when the state is a preparation state, so that the problems of high manual operation and maintenance cost and high time cost in a scheme responding to file application requirements of various downstream systems in the prior art are solved.
Further, in an optional embodiment of the present application, a file interface is established, which is provided for each data user to call, and supports the hbase database query function. The interface parameters include: sub product name: the file system name is applied. And finding information such as the address ftp ip, the user name ftp user, the password ftp password and the file receiving path of the file receiving server of the system from the ftp pushing configuration table according to the parameter value, and pushing the file to the file. dataType: and according to the value, designating the name of the database to which the file belongs, inquiring an object storage directory obs dir and an uploading state obs _ status corresponding to the src file name in a file uploading object storage record table dbus obs upload info of the database, and if the obs status is the SUC success state, writing the information of the src file name and the obs dir into a dbus tasklist download task table of the udp dbus service library for monitoring, downloading and pushing of the file by a service. src file name: the source file name, the file name value of each data system according to the uniform standard format.
Further, in an optional embodiment of the present application, before the information of the target file is inserted into the download task table, a download task table may be further established to record relevant information of the file to be downloaded, when the microservice ms receives a file download request of each system, the upload state of the file in the object storage is queried in the dbus obsupload info table of the specified database according to the dataType and src file name parameter values transmitted by the interface, if the obs status state is successful, a file record is inserted into the dbus tasklist download data online table, and information such as the src file name, obs dir, sub product name, and the like is recorded.
Further, in an optional embodiment of the present application, before the information of the file is inserted into the push task table, a push task table may be further established to record relevant information of the file to be pushed. And when the file is successfully downloaded from the object storage to the service processing directory, transferring the file to the push directory, inserting a file record into the dbus tasklist file push table, and recording information such as the src file name, the file path, the sub product name and the like.
Further, in an optional embodiment of the present application, before determining whether the target file is uploaded to the object storage, a mysql-to-hive synchronization function of the Tencent TBDS platform may be further used to synchronize mysql table information to a hive table on a daily basis. And (3) newly building a workflow, scheduling according to the day, and synchronizing the daily increment of the data in the mysql table dbus obs upload info into the hive table, wherein the hive table is strong in storage capacity but not suitable for query, so that the data is mapped into the hbase table, when the ms receives a request, firstly, the file records are queried in the mysql table dbus obs upload info, and if the file records cannot be queried, the file records are queried in the hbase table.
The foregoing provides some specific implementation manners of a method for downloading and pushing a data file according to an embodiment of the present application, and based on this, the present application also provides a corresponding device for downloading and pushing a data file. The device provided by the embodiment of the present application will be described in terms of functional modularity.
Referring to fig. 2, fig. 2 is a schematic structural diagram of a data file downloading and pushing device provided in an embodiment of the present application, where the device includes: the system comprises a judging module 201, a first inserting module 202, a first downloading module 203, a second inserting module 204 and a second downloading module 205;
the judging module 201 is configured to invoke a file downloading request interface, and judge whether a target file is uploaded to an object for storage according to interface parameters;
the first inserting module 202 is configured to insert information of a target file into a download task table according to a parameter of the file download request interface and the queried obs _ dir information when the target file is uploaded to the object storage;
the first downloading module 203 is configured to, when a target task state in the downloading task table is a first state, download the target task;
the second inserting module 204 is configured to insert the information of the target file into a push task table after the target task is successfully downloaded;
the second downloading module 205 is configured to, when the target task state in the push task table is a second state, push the target task.
The application provides a data file downloading and pushing device which is used for executing a corresponding data file downloading and pushing method. When the method is executed, a file downloading request interface is called first, and whether a target file is uploaded to an object for storage is judged according to parameters of the file downloading request interface; if the target file is uploaded to the object storage, inserting the information of the target file into a downloading task table according to the parameters of the file downloading request interface; when the target task state in the download task table is a first state, downloading the target task; after the target task is downloaded successfully, inserting the information of the target file into a push task table; and when the target task state in the push task table is a second state, pushing the target task. The method and the device have the advantages that the file downloading request interface is called, the downloading task and the pushing task are automatically stored according to the parameters of the file downloading request interface, and the task is automatically downloaded or pushed when the state is the preparation state, so that the problems of high manual operation and maintenance cost and high time cost in the scheme responding to the file application requirements of all downstream systems in the prior art are solved.
Further, in an optional embodiment of the present application, the apparatus further includes an establishing module, where the establishing module is specifically configured to:
establishing a file downloading request interface; the parameters of the interface at least comprise an application file system name, a file type and a source file name.
Further, in an optional embodiment of the present application, before the information of the target file is inserted into the download task table, the establishing module is further configured to:
and establishing a downloading task table, and recording related information of the task to be downloaded by using the downloading task table.
Further, in an optional embodiment of the present application, before the inserting the information of the target file into the push task table, the establishing module is further configured to:
and establishing a push task table, and recording related information of the task to be pushed by using the push task table.
Further, in an optional embodiment of the present application, the apparatus further includes a synchronization module and a mapping module; before the judgment of whether the target file is uploaded to the object storage, the synchronization module is specifically configured to synchronize file information in the object storage to a first database in a date unit; the mapping module is specifically configured to map data in the first database (periodic cleaning) to a second database (persistent storage), where the second database is configured to query whether target file information exists in the second database when the target file information is not queried in the first database.
In the embodiments of the present application, the names "first" and "second" in the names "first state" and "second state" are used merely as name labels, and do not represent the first and second in sequence.
As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the above embodiment methods can be implemented by software plus a general hardware platform. Based on such understanding, the technical solution of the present application may be embodied in the form of a software product, which may be stored in a storage medium, such as a read-only memory (ROM)/RAM, a magnetic disk, an optical disk, or the like, and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network communication device such as a router) to execute the method according to the embodiments or some parts of the embodiments of the present application.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the apparatus embodiment, since it is substantially similar to the method embodiment, it is relatively simple to describe, and reference may be made to some descriptions of the method embodiment for relevant points. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement without inventive effort.
The above description is only an exemplary embodiment of the present application, and is not intended to limit the scope of the present application.
It should be noted that the data file downloading and pushing method and device provided by the invention can be applied to the technical field of big data or the financial field. The foregoing is only an example, and does not limit the application field of the method and apparatus for downloading and pushing a data file provided by the present invention.
Claims (10)
1. A data file downloading and pushing method is characterized by comprising the following steps:
calling a file downloading request interface, and judging whether a target file is uploaded to an object for storage or not according to parameters of the file downloading request interface;
if the target file is uploaded to the object storage, inserting the information of the target file into a downloading task table according to the parameters of the file downloading request interface;
when the target task state in the downloading task table is a first state, downloading the target task;
after the target task is downloaded successfully, inserting the information of the target file into a push task table;
and when the target task state in the push task table is a second state, pushing the target task.
2. The method of claim 1, further comprising:
establishing a file downloading request interface; the parameters of the interface at least comprise an application file system name, a file type and a source file name.
3. The method of claim 1, wherein before the information of the target file is inserted into a download task table, the method further comprises:
and establishing a downloading task table, and recording related information of the task to be downloaded by using the downloading task table.
4. The method of claim 1, wherein prior to said inserting information of the target file into a push task table, the method further comprises:
and establishing a push task table, and recording related information of the task to be pushed by using the push task table.
5. The method of claim 1, wherein prior to said determining whether the target file is uploaded to the object store, the method further comprises:
synchronizing file information in the object storage to a first database by taking date as a unit;
and mapping the data in the first database to a second database, wherein the second database is used for inquiring whether the target file information exists in the second database when the target file information is not inquired in the first database.
6. A data file downloading and pushing device is characterized in that the device comprises: the device comprises a judging module, a first inserting module, a first downloading module, a second inserting module and a second downloading module;
the judging module is used for calling a file downloading request interface and judging whether a target file is uploaded to an object for storage or not according to the parameters of the file downloading request interface;
the first inserting module is used for inserting the information of the target file into a downloading task table according to the parameters of the file downloading request interface under the condition that the target file is uploaded to the object storage;
the first downloading module is used for downloading the target task when the target task state in the downloading task table is a first state;
the second inserting module is used for inserting the information of the target file into a push task table after the target task is downloaded successfully;
and the second downloading module is used for pushing the target task when the target task state in the task pushing table is the second state.
7. The apparatus according to claim 6, further comprising an establishment module, the establishment module being specifically configured to:
establishing a file downloading request interface; the parameters of the interface at least comprise an application file system name, a file type and a source file name.
8. The apparatus of claim 6, wherein before the information of the target file is inserted into the download task table, the establishing module is further configured to:
and establishing a downloading task table, and recording related information of the task to be downloaded by using the downloading task table.
9. The apparatus of claim 6, wherein before the inserting the information of the target file into the push task table, the establishing module is further configured to:
and establishing a push task table, and recording related information of the task to be pushed by using the push task table.
10. The apparatus of claim 6, further comprising a synchronization module and a mapping module; before the target file is uploaded to the object storage, the synchronization module is specifically configured to synchronize file information in the object storage to a first database in units of dates; the mapping module is specifically configured to map data in the first database to a second database, where the second database is configured to query whether target file information exists in the second database when the target file information is not queried in the first database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211071954.2A CN115442363A (en) | 2022-09-02 | 2022-09-02 | Data file downloading and pushing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211071954.2A CN115442363A (en) | 2022-09-02 | 2022-09-02 | Data file downloading and pushing method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115442363A true CN115442363A (en) | 2022-12-06 |
Family
ID=84247163
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211071954.2A Pending CN115442363A (en) | 2022-09-02 | 2022-09-02 | Data file downloading and pushing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115442363A (en) |
-
2022
- 2022-09-02 CN CN202211071954.2A patent/CN115442363A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102868731B (en) | Method and device for software online updating and downloading acceleration | |
| CN107566463B (en) | Multi-cloud storage management system for improving storage availability | |
| CN107105050B (en) | Storage and downloading method and system for service objects | |
| CN103019757A (en) | Method, device and system for synchronizing software among multiple clients | |
| US20110138079A1 (en) | Information processing apparatus, information processing method, data management server and data synchronization system | |
| CN111200657A (en) | Method for managing resource state information and resource downloading system | |
| CN105912424A (en) | Cloud architecture based terminal program quick backup and recovery method | |
| CN104699499A (en) | Heterogeneous terminal software upgrading method | |
| CN108614750A (en) | The restoration methods and device of delay machine data | |
| EP1096769A2 (en) | Methods and systems for building and distributing audio packages | |
| CN110602232A (en) | Terminal system version downloading method, device and system based on peer-to-peer network idea | |
| CN115442363A (en) | Data file downloading and pushing method and device | |
| CN111064798B (en) | Node network access method, device, client node and storage medium | |
| CN108763471B (en) | Method and system for deploying HTTP file server in cluster | |
| CN111400276A (en) | Real-time synchronous database migration device and method | |
| CN115629784A (en) | Method, system, equipment and computer readable storage medium for updating machine file | |
| CN115455121A (en) | Real-time reliable data synchronous transmission method, equipment and medium | |
| CN103795810A (en) | Data distribution system and method and central server of data distribution system | |
| CN112738206B (en) | Butt joint method and assembly of distributed cluster and cloud platform | |
| CN112153093B (en) | Cluster-based task scheduling method, device, equipment and readable storage medium | |
| CN100561921C (en) | The system and method for client and server quick exchange data | |
| EP3780625A1 (en) | Recorded data processing method and relevant device | |
| WO2019179050A1 (en) | Call log management method and system, computer device and storage medium | |
| CN113794574B (en) | Method and device for generating and transmitting ticket | |
| KR20180120499A (en) | Method for synchronizing of database and private branch exchange for the same |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |