WO2016165482A1

WO2016165482A1 - File transfer method and apparatus

Info

Publication number: WO2016165482A1
Application number: PCT/CN2016/074278
Authority: WO
Inventors: 吴孝鹏; 尤元建; 黄增建
Original assignee: 中兴通讯股份有限公司
Priority date: 2015-08-07
Filing date: 2016-02-22
Publication date: 2016-10-20
Also published as: CN106445951A; CN106445951B

Abstract

A data transfer method and apparatus. The data transfer method comprises: generating a configuration file according to input of a user, wherein the configuration file comprises a file transfer path and a transfer rule corresponding to the file transfer path, and the file transfer path comprises a source address and a destination address of a file to be transferred; and searching, according to the configuration file, for a file satisfying the transfer rule in the source address of the file transfer path, and transferring the found file to the destination address, wherein the transferred file comprises a formatted data file and/or a non-formatted data file. The technical solution solves the problems of a complex file transfer operation mode and low efficiency in the mode of file transfer with a Hadoop system.

Description

File transmission method and device

Technical field

This document relates to, but is not limited to, the field of data storage technology, and more particularly to a file transmission method and apparatus.

Background technique

With the development of the Internet, the Internet of Things developed on the basis of the Internet has been widely used in information exchange and data transmission. In order to satisfy users' storage of massive data, the distributed system infrastructure developed by the Apache Foundation is generally used. The Hadoop system stores massive amounts of data.

The Hadoop system includes a distributed file system (Hadoop Distributed File System, HDFS) and a distributed database (Hbase). The data storage method in the Hadoop system storage includes structured data files or files stored in HDFS in file storage mode. Structured data files, for unstructured column mode data, can be queried by Hbase; Hadoop system is a platform for massive data storage. Users can use the command line to upload or download files through terminal devices, for example, through PUT command. The files stored in the terminal device are uploaded to the HDFS of the Hadoop system. For example, the files stored in the HDFS of the Hadoop system are downloaded to the memory of the terminal device or the relational database of the Hadoop system in the peer database by using the GET command. Each execution of the command line can only support the transmission of a single file, and the data format of each transmitted file is fixed. If you need to transfer multiple files and files of different data formats, you need to manually enter the command line multiple times. The way can be achieved.

However, in the method of file transfer with the Hadoop system, the related art of the file transfer, that is, each command line can only support the transmission of a single file, and the data format of each transmitted file is fixed, resulting in a file. The operation of the transmission is complicated and the efficiency is low.

Summary of the invention

The following is an overview of the topics detailed in this document. This Summary is not intended to limit the scope of the claims.

Embodiments of the present invention provide a file transmission method and apparatus, which can be avoided in dealing with Hadoop. In the way that the system performs file transfer, the file transfer operation mode is complicated and the efficiency is low.

In a first aspect, an embodiment of the present invention provides a file transmission method, including:

And generating, by the user input, a configuration file, where the configuration file includes a file transmission path and a transmission rule corresponding to the file transmission path, where the file transmission path includes a source address and a target address of the file to be transmitted;

Determining, according to the configuration file, a file that meets the transmission rule in a source address of the file transmission path, and transmitting the found file to the target address, where the transmitted file includes a formatted data file and / or unformatted data files.

In a first possible implementation manner of the first aspect, the transmission rule includes a file format, where a file that meets the transmission rule is searched for in a source address of the file transmission path, and the file that is found is found Transfer to the target address, including:

Finding a file matching the file format in a source address of the file transmission path, and generating a file transfer list;

And correspondingly transferring the corresponding file in the source address to the target address according to the file transfer list.

According to a first possible implementation manner of the first aspect, in a second possible implementation manner, the file transmission path includes a first path and a second path, where the first path and the second path are both included At least one of the file transmission paths, the source address of the file transmission path is searched for a file matching the file format, and a file transfer list is generated; and the source address is sequentially selected according to the file transfer list The corresponding file is transferred to the target address, including:

And searching for a file matching the file format corresponding to the current file transmission path in the source addresses of the first path and the second path, generating a current file transfer list, and sequentially performing the according to the current file transfer list. A corresponding file in the source address of the current file transfer path is transferred to the target address.

In a third possible implementation manner of the first aspect, the configuration file further includes a verification mode, where the transmission rule includes an index verification file, and the searching in the source address of the file transmission path is consistent with the Transfer the file of the rule and transfer the found file to the target address, including:

When the verification mode is an open check, searching for a file corresponding to an index entry of the index check file in a source address of the file transfer path, and transmitting the found file to the target address in;

After the transferring the file to the target address, the method further includes:

When the verification mode is the file processing, the index verification file is stored in a preset path of the network end to which the target address belongs;

And when the verification mode is to close the file processing, deleting the index verification file or storing the index verification file in a preset path of the network end of the source address.

According to a third possible implementation manner of the first aspect, in a fourth possible implementation, the transmission rule further includes a file format, where the searching in the source address of the file transmission path matches the transmission rule The file and transfer the found file to the target address, including:

When the verification mode is a shutdown check, searching for a file matching the file format in a source address of the file transmission path, generating a file transfer list, and sequentially, according to the file transfer list, the source address The corresponding file in the file is transferred to the target address.

According to the first aspect, any one of the first to fourth possible implementation manners of the first aspect, in a fifth possible implementation, the configuration file further includes a compressed mode, where the Transfer the file to the target address, including:

When the compression mode is on, compressing the searched file, and transmitting the compressed file to the target address;

The searched file is transferred to the target address when the compressed mode is off.

According to the first aspect, any one of the first to fourth possible implementation manners of the first aspect, in a sixth possible implementation, the configuration file further includes a backup mode and a backup address, where After the found file is transferred to the target address, it also includes:

When the backup mode is off, deleting a file that has been transmitted to the target address in a source address of the file transfer path;

When the backup mode is enabled, the file that has been transferred to the target address in the source address of the file transmission path is backed up to the backup address of the network end to which the source address belongs.

According to the first aspect, the first to the fourth possible implementation manner of the first aspect, in a seventh possible implementation, the configuration file further includes a scanning time interval, where After searching for the file that meets the transmission rule in the source address of the file transmission path, and transferring the found file to the target address, the method further includes:

When the aging time reaches the scanning time interval, searching for a file that meets the transmission rule of the current configuration file in the source address of the file transmission path of the current configuration file according to the current configuration file, and transmitting the found file to The destination address of the file transfer path of the current configuration file.

In a second aspect, an embodiment of the present invention provides a file transmission apparatus, including:

a configuration module, configured to generate a configuration file according to a user input, where the configuration file includes a file transmission path and a transmission rule corresponding to the file transmission path, where the file transmission path includes a source address and a destination address of the file to be transmitted ;

a transmission module, configured to search for a file that meets the transmission rule in a source address of the file transmission path according to a configuration file generated by the configuration module, and transmit the found file to the target address, where The transferred files include formatted data files and/or unformatted data files.

In a first possible implementation manner of the second aspect, the transmission rule includes a file format, and the transmission module includes: a searching unit configured to search for a source format of the file transmission path and the file format Match the files and generate a file transfer list;

And a transmitting unit, configured to sequentially transmit the corresponding file in the source address to the target address according to the file transfer list generated by the searching unit.

According to a first possible implementation manner of the second aspect, in a second possible implementation, the file transmission path includes a first path and a second path, where the first path and the second path are both included At least one of the file transmission paths, the searching unit is configured to sequentially search for a file matching the file format corresponding to the current file transmission path in the source addresses of the first path and the second path, and generate a current file File transfer list;

And the transmitting unit is configured to sequentially transmit the corresponding file in the source address of the current file transmission path to the target address according to the current file transmission list generated by the searching unit.

In a third possible implementation manner of the second aspect, the configuration file further includes a verification mode, the transmission rule includes an index verification file, and the transmission module includes: a determining unit, configured to determine the verification State of the mode;

The searching unit is further configured to: when the determining unit determines that the verification mode is the on check, searching for a file corresponding to the index entry of the index check file in the source address of the file transfer path;

a transmission unit, configured to transmit the file found by the searching unit to the target address;

The file transfer module further includes: a processing unit configured to: after the file unit transmits the file to the target address, when the determining unit determines that the check mode is open file processing, the index check file is Stored in a preset path to the network end to which the target address belongs;

When the determining unit determines that the verification mode is closed file processing, deleting the index verification file or storing the index verification file in a preset path of the network end of the source address.

According to a third possible implementation of the second aspect, in a fourth possible implementation, the transmission rule further includes a file format, and the searching unit is further configured to determine, at the determining unit, the school When the verification mode is off verification, a file matching the file format is searched for in the source address of the file transmission path to generate a file transfer list.

According to the second aspect, any one of the first to fourth possible implementation manners of the second aspect, in the fifth possible implementation, the configuration file further includes a compressed mode, where the transmission module includes: a determining unit, a compressing unit, and a transmitting unit; the determining unit is configured to determine a state of the compressed mode;

The compression unit is configured to perform compression processing on the found file when the determining unit determines that the compression mode is on;

The transmission module is configured to transmit the compressed file of the compression unit to the target address; and when the determining unit determines that the compression mode is off, transmit the found file to In the target address.

According to the second aspect, any one of the first to fourth possible implementations of the second aspect, In a sixth possible implementation, the configuration file further includes a backup mode and a backup address, the transmission module further includes: a determining unit and a processing unit; the determining unit is configured to be in the transmitting unit After the found file is transferred to the target address, the state of the backup mode is determined;

The processing unit is configured to, when the determining unit determines that the backup mode is off, delete a file that has been transmitted to the target address in a source address of the file transmission path;

When the determining unit determines that the backup mode is on, back up the file that has been transmitted to the target address in the source address of the file transmission path to the backup address of the network end to which the source address belongs.

According to the second aspect, any one of the first to fourth possible implementation manners of the second aspect, in a seventh possible implementation, the configuration file further includes a scanning time interval, the file transmission device The method further includes: a timing module configured to perform a timing operation after the transmission module transmits the found file to the target address;

The transmission module is further configured to: when the timing time of the timing module reaches the scanning time interval, searching for the current configuration file according to the current configuration file in the source address of the file transmission path of the current configuration file. Transfer the file of the rule and transfer the found file to the destination address of the file transfer path of the current profile.

The embodiment of the invention further provides a computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions are used to execute the above method.

A file transmission method and apparatus according to an embodiment of the present invention generates a configuration file including a file transmission path and a transmission rule according to a user input, where the file transmission path includes a source address and a destination address of the file to be transmitted, according to the configuration file, Find the file that meets the transmission rule in the source address of the file transmission path, and transfer the found file to the target address. In this embodiment, by setting the configuration file, the configuration file can be used only with the Hadoop system. Mass transfer of mass files is performed, and the plurality of files transferred may be formatted data files and/or unformatted data files; the embodiment of the present invention solves the operation mode of file transfer in the manner of file transfer with the Hadoop system. Complex and less efficient problems.

Other aspects will be apparent upon reading and understanding the drawings and detailed description.

BRIEF abstract

FIG. 1 is a flowchart of a file transmission method according to an embodiment of the present invention;

2 is a flowchart of another file transmission method according to an embodiment of the present invention;

FIG. 3 is a flowchart of still another file transmission method according to an embodiment of the present invention;

4 is a flowchart of still another file transmission method according to an embodiment of the present invention;

FIG. 5 is a schematic structural diagram of a file transmission apparatus according to an embodiment of the present disclosure;

FIG. 6 is a schematic structural diagram of another file transmission apparatus according to an embodiment of the present invention.

Embodiments of the invention

Embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be noted that, in the case of no conflict, the features in the embodiments and the embodiments in the present application may be arbitrarily combined with each other.

The steps illustrated in the flowchart of the figures may be executed in a computer system such as a set of computer executable instructions. Also, although logical sequences are shown in the flowcharts, in some cases the steps shown or described may be performed in a different order than the ones described herein.

In the related art, the application scenario of the data exchange between the terminal device and the HDFS of the Hadoop system generally includes: data transmission between the memory of the terminal device and the HDFS, and data transmission between the relational database of the Hadoop system and the HDFS in the network. The above two data transmission methods are all performed by the terminal device. In the file transmission described in the following embodiments of the present invention, the actual content transmitted is different types of data, and the data exists in different network ends in the form of files, and The two ends of the application scenario can be uploaded to the HDFS or downloaded to the local terminal device or the relational database. The storage capacity of the network can be: HDFS and local terminal devices. Store structured data files and unstructured data files, while structured data files can only be stored in relational databases.

It should be noted that the terminal device that performs the file transmission method in the following embodiments is usually a local server. Since the terminal device in the embodiments of the present invention needs to transmit massive data with the Hadoop system, the Linux-based operating system can usually be used. Server. Specifically by The following is a detailed description of the technical solutions of the present invention, and the following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be described in some embodiments.

FIG. 1 is a flowchart of a file transmission method according to an embodiment of the present invention. In the case where the file transfer method provided in this embodiment is applicable to file transfer with the HDFS of the Hadoop system, the method may be performed by a data transfer device, which is usually implemented in hardware and software, and the device may be integrated in The processor of the terminal device is used by the processor to call. As shown in FIG. 1, the method in this embodiment may include:

S110: Generate a configuration file according to the input of the user, where the configuration file includes a file transmission path and a transmission rule corresponding to the file transmission path, where the file transmission path includes a source address and a target address of the file to be transmitted.

In this embodiment, the startup process of the terminal device can be started by starting a loader program on the terminal device, and the configuration interface is accessed after the heartbeat service is started, and the configuration template file for file transfer is provided on the terminal device, and the configuration template file is provided. Including various configuration items that need to be set, the terminal device can visually display the items to be set in the configuration template file to the user through a graphical user interface (GUI), so that the user can configure the template file in the configuration template file. The configuration file is required to generate a configuration file. The name of the configuration template file in this embodiment is, for example, template.transferConfig.xml, and the name of the configuration file is, for example, transferConfig.xml.

In a specific implementation, the content of the configuration file generated according to the user-defined input generally includes a file transmission path and a corresponding transmission rule, where the file transmission path is the two ends of the application scenario, one end is HDFS in the Hadoop system, and the other end is The terminal device or the relational database, that is, the file transmission path can represent the transmission direction of the massive file to be transmitted and the start point and the end point, that is, the file transmission path includes the source address and the target address of the file to be transmitted.

It should be noted that the file transfer in the embodiments of the present invention may be uploaded to the HDFS by the memory or the relational database of the terminal device, or may be downloaded from the HDFS to the memory of the terminal device or the relational database, or may be the above. The combination of the two transmission directions, that is, the embodiments of the present invention do not limit the number of file transmission paths, and also does not limit the source address of the different file transmission paths and the network end to which the target address belongs.

S120. Search for a file conforming to the transmission rule in the source address of the file transmission path according to the configuration file, and transfer the found file to the target address, where the transmitted file includes the formatted data file and/or the unformatted data. file.

In this embodiment, the file transmission path defines the source address and the destination address of the mass file to be transmitted. The number of files in the source file in the configuration file is usually very large, so it is necessary to further filter the files to be transmitted, that is, by transmitting. The rule filters out the files that need to be transferred. For example, you can transfer all the images with the suffix ".jpg" in the source address to the destination address, or you can transfer all the files with the suffix ".doc" in the source address to the destination address. In the target address.

It should be noted that the foregoing transmission rules in this embodiment are in one-to-one correspondence with the file transmission path, and different file transmission paths may have different transmission rules; and the embodiment does not limit the data format of the files transmitted according to the configuration file. It can be a formatted data file, an unformatted data file, or a combination of the above two data formats. In addition, in this embodiment, the Hadoop system performs a mass file transfer process, and only needs to set a configuration file once, and all the massive files conforming to the transmission rule in the path defined by the configuration file are transmitted from the source address to the target address, and executed. Compared with the way the file is transmitted by the command line, it greatly simplifies the operation mode and improves the efficiency of file transfer.

The file transmission method provided in this embodiment generates a configuration file including a file transmission path and a transmission rule according to a user input, where the file transmission path includes a source address and a destination address of the file to be transmitted, and the file is transmitted according to the configuration file. Find the file that meets the transmission rule in the source address of the path, and transfer the found file to the destination address. In this embodiment, by setting the configuration file, it is possible to perform massively with the Hadoop system only through the configuration file set by the user. Bulk transfer of files, and the plurality of files transferred may be formatted data files and/or unformatted data files; the method provided in this embodiment solves the operation mode of file transfer in the manner of file transfer with the Hadoop system. Complex and less efficient problems.

Optionally, FIG. 2 is a flowchart of another file transmission method according to an embodiment of the present invention. In this embodiment, the transmission rule corresponding to the file transmission path is, for example, a file format. On the basis of the embodiment shown in FIG. 1 , S120 in this embodiment may include: S121, according to the configuration file, in the file transmission path. Finding a file matching the file format in the source address, and generating a file transfer list; S122, sequentially transferring the corresponding file in the source address to the target address according to the file transfer list. The specific manner of performing massive file transfer in this embodiment is: after all the files matching the file format are found, the files in the file transfer list need to be sequentially transferred to the target address, and the traversal file transfer list can usually be used. Transfer, optionally, traversing to the current file in the file transfer list, determining whether the current file is being transferred, and if the current file is not the file being transferred, opening an HDFS file push thread for the current file, and placing the thread Waiting to execute the scheduling in the thread pool, and then determining whether the current file is the last file. If the current file is the last file, the current file is transferred and the configuration file is transferred. If the current file is not the last file, continue. The file in the file transfer list is traversed for transmission; in addition, when it is determined that the current file is the file being transferred, it is also possible to further determine whether the current file is the last file, and perform subsequent operations according to the judgment result.

In a specific implementation, the file transmission path in this embodiment may be one or more. On the one hand, if only one file transmission path is included in the configuration file, only all files in the path that meet the transmission rule are transmitted, and the terminal is The process and thread configuration requirements of the device are not high. On the other hand, if the configuration file includes multiple file transmission paths and requires simultaneous transmission of files conforming to the transmission rules in each path, the configuration of the processes and threads of the terminal device is relatively high, and usually requires more and more. Thread configuration to meet the requirements for parallel file transfer. On the other hand, if the configuration file includes multiple file transmission paths, the file transmission path in the configuration file may be traversed in sequence, that is, the files in the partial path are transmitted first, and then the other file transmission paths in the configuration file are traversed. The search and the transmission operation are performed multiple times. The specific implementation manner is that the file transmission path includes, for example, a first path and a second path, and the first path and the second path both include at least one file transmission path, as shown in FIG. S120 in the embodiment may be replaced by: sequentially searching for a file matching the file format corresponding to the current file transmission path in the source addresses of the first path and the second path, generating a current file transfer list, and sequentially according to the current file transfer list. Transfer the corresponding file in the source address of the current file transfer path to the destination address. It should be noted that the number of file transmission paths traversed each time in this embodiment may be determined according to the thread configuration of the terminal device. For example, when the terminal device is in a single-process single-thread mode, one file transmission path may be traversed each time to execute. The file transmission in the path is the same as that in the foregoing embodiment, and therefore will not be described here.

FIG. 3 is a flowchart of still another file transmission method according to an embodiment of the present invention. In another possible implementation manner of the embodiment shown in FIG. 1, the configuration file further includes a verification mode, and For the transmission rule corresponding to the file transmission path, for example, the index verification file, on the basis of the embodiment shown in FIG. 1 , S120 in this embodiment includes: S121, and the state of the verification mode is determined according to the configuration file; If the mode is to enable the verification, then S122 is performed; if the verification mode is the closed verification, then S123 is executed.

S122. Search for a file corresponding to the index entry of the index check file in the source address of the file transfer path, and transfer the found file to the target address.

It should be noted that the transmission rule in this embodiment may further include a file format, and S123 may be: searching for a file matching the file format in the source address of the file transmission path, generating a file transmission list, and according to the file transmission list. The corresponding file in the source address is transferred to the destination address in turn.

In this embodiment, after executing S122 or S123, the method further includes: S130, determining a state of the verification mode; if the verification mode is to open file processing, executing S131; if the verification mode is closing file processing, executing S132.

S131. Store the index verification file in a preset path of the network end to which the target address belongs.

S132. Delete the index verification file or store the index verification file in a preset path of the source address network.

It should be noted that the status of the verification mode in this embodiment includes a verification status and a file processing status, and the verification status is used to control whether to perform a verification function, that is, whether to use an index verification file to search for a file to be transmitted, and a file processing status. After the file transfer is performed, it is used to control the subsequent processing operations on the index check file. The specific operation manner is as shown in the above S130 to S132.

It should be noted that, in the embodiment shown in FIG. 2, the file to be transmitted is searched by the file format and the file to be transmitted is searched by the index check file in the embodiment shown in FIG. 3, which is two ways of selectively searching for the file of the present invention. In another possible implementation manner of this embodiment, if there is no file format in the transmission rule, and the verification mode is determined to be the shutdown check in S121, the manner of transferring the file may also be : Transfer all files in the source address of the file transfer path to the destination address.

Optionally, FIG. 4 is a flowchart of still another file transmission method according to an embodiment of the present invention. On the basis of the foregoing embodiments of the present invention, S120 in the method provided by this embodiment includes:

S121. Search for a file that meets the transmission rule in the source address of the file transmission path according to the configuration file. The configuration file in this embodiment further includes a compression mode.

S122, determining the state of the compressed mode; if the compressed mode is on, executing S123; if the compressed mode is off, executing S124.

S123, compressing the found file, for example, performing LZO (Lempel-Ziv-Oberhumer) algorithm compression on the file. Therefore, it is also necessary to execute S124 to transfer the file to the target address, and the file transferred to the target address is the compressed file. It should be noted that the present embodiment is further described by taking the example shown in FIG. 1 as an example.

Optionally, on the basis of the foregoing embodiments of the present invention, the configuration file may further include a backup mode and a backup address, which are illustrated on the basis of the embodiment shown in FIG. 4, and after performing S124 in this embodiment. ,Also includes:

S130: Determine the state of the backup mode; if the backup mode is off, execute S131; if the backup mode is on, execute S132.

S131. Delete the file that has been transferred to the destination address in the source address of the file transfer path.

S132. Back up the file that has been transferred to the destination address in the source address of the file transmission path to the backup address of the network end to which the source address belongs.

In this embodiment, if the network end to which the source address belongs is the local terminal device, the file is stored in the backup address of the terminal device, and if the network end to which the source address belongs is the network end where the relational database is located, the file is stored in the file. In the backup address of the relational database, the network to which the source address belongs can also be a Hadoop system, and the file is stored in the backup address of the HDFS. It should be noted that the backup address can be the same as the source address or the source. The address is different.

Optionally, on the basis of the foregoing embodiments of the present invention, if the file to be transmitted is a dynamic data stream, for example, the content of the voice call or the video conference is transmitted to the HDFS, the file transfer operation needs to be repeatedly performed, and The current configuration file can be changed by the setting of the user. The specific implementation manner is as follows: the configuration file further includes a scan time interval. In the method provided in this embodiment, after S120, the method further includes: performing a timing operation, and reaching a scan time at the time of the time. At interval, S120 is re-executed, at this time, according to the current configuration file, the source of the file transmission path of the current configuration file. The file in the address is found to match the transmission rule of the current current configuration file, and the found file is transferred to the destination address of the file transmission path of the current configuration file. It should be noted that the method provided by each embodiment of the present invention performs timing after S120, and may perform loop S120.

It should be noted that the terminal device that performs the file transmission method provided by the embodiments of the present invention can be configured as a multi-thread configuration in a multi-process mode, can perform multi-thread startup, and has large parallel execution capability when performing file transmission. Further, the transmission rate is further improved; in addition, the startup Loader program can perform the dual-machine mode of the host and the standby machine, thereby improving the reliability of the file transmission method.

The specific content of the configuration file in the foregoing embodiments of the present invention is specifically described below. The name of the configuration file generated by the user's setting is, for example, transferConfig.xml, which can be changed by the user's setting when the terminal device is restarted. The contents of the configuration file transferConfig.xml include, for example:

1. TransferPath: indicates the file transfer path and file backup path. The file backup path is specifically the path from the source address to the backup address in the embodiment shown in Figure 4. The transfer path can be filled with an absolute path or a relative path. The relative path refers to the hadooploader. The installation path is the installation path of the Loader program on the terminal device, and the index indicates the path index value.

2, transferRule represents the transport rule, where name represents the rule name, path represents the index of the file transfer path, corresponding to the path index value in the transferpath, file represents the file name, type represents the type of file transfer to Hadoop, specifically Hbase or HDFS, Hbase corresponding fill-in value is db, HDFS or corresponding fill-in value is hdfs, and dest fills in the specific content of file transfer; it should be noted that Hbase is a database in Hadoop system, which can support unstructured column mode. For the data of this mode, unstructured data in HDFS can be queried through Hbase.

3. For db type: name indicates the name of the table, field indicates the name of the field, key indicates the index, where field indicates the field contained in the index, ctm indicates whether the index contains time, and Y indicates that the content contains N. Indicates not included, sqc indicates whether the index contains a serial number, Y indicates inclusion, and N indicates no inclusion.

4. For the hdfs type: the path in the dest configuration item indicates the address in the HDFS, and the region indicates the directory partition mode. If the RegionType in the Config.properties is 0, the file to be transmitted supports the day (day), week (week), and month ( Month), the year (year) storage form (/YYYYMMWWDD), the storage form is a layer storage path, or the storage file supports the storage form of days (/YYYYMMWW/DD), the storage form is a two-tier storage path Form; if the RegionType in Config.properties is 1, it means that the transferred file supports the day (day), month (month), year (year) storage form (/YYYYMMDD), or the file that supports the transfer supports days (/YYYYMM/DD). ) The form of storage.

5. The configuration file supports multiple file transmission paths, and can perform transmission operations on files in multiple file transmission paths in parallel or in turn; and supports dynamic implementation of configuration files.

6. The index verification file is, for example, an OK file. The plug-in verification of the OK file is based on a plug-in verification function extended by the Loader program, and is used to provide a verification interface to complete the user's custom verification of the OK file. If the user does not configure a custom verification plugin, the verification function can be completed using the default verification provided by the terminal device.

FIG. 5 is a schematic structural diagram of a file transmission apparatus according to an embodiment of the present invention. In the case where the file transmission apparatus provided in this embodiment is suitable for file transmission with the HDFS of the Hadoop system, the file transmission apparatus is implemented in a hardware and software manner, and the apparatus may be integrated in a processor of the terminal apparatus for the processor. Called for use. As shown in FIG. 5, the file transmission apparatus of this embodiment includes: a configuration module 11 and a transmission module 12.

The configuration module 11 is configured to generate a configuration file according to the input of the user, where the configuration file includes a file transmission path and a transmission rule corresponding to the file transmission path, where the file transmission path includes a source address and a target address of the file to be transmitted.

The transmission module 12 is configured to be in the file transmission path according to the configuration file generated by the configuration module 11. Find the file that meets the transmission rule in the source address of the path, and transfer the found file to the destination address. The transferred file includes the formatted data file and/or the unformatted data file.

The file transmission device provided by the embodiment of the present invention is used to perform the file transmission method provided by the embodiment shown in FIG. 1 of the present invention, and has a corresponding function module, and the implementation principle and the technical effect thereof are similar, and details are not described herein again.

Optionally, FIG. 6 is a schematic structural diagram of another file transmission apparatus according to an embodiment of the present invention. In this embodiment, the transmission rule corresponding to the file transmission path is, for example, a file format. On the basis of the embodiment shown in FIG. 5, the transmission module 12 in this embodiment includes: a searching unit 13 configured to be in a file transmission path. Searching for a file matching the file format in the source address, and generating a file transfer list; the transfer unit 14 is configured to sequentially transfer the corresponding file in the source address to the target address according to the file transfer list generated by the lookup unit 13.

In a specific implementation, the file transmission path in the embodiment may be one or more, and the configuration file includes multiple file transmission paths, and may simultaneously query the files to be transmitted in each file transmission path, or may be divided into several times. Executing, optionally, the file transmission path includes a first path and a second path, and the first path and the second path both include at least one file transmission path, and the searching unit 13 is further configured to be sequentially Searching for a file matching the file format corresponding to the current file transmission path in the source address of the path and the second path, generating a current file transfer list; and the transmitting unit 14 is further configured to sequentially perform the current file transfer list according to the search unit 13 The corresponding file in the source address of the file transfer path is transferred to the destination address.

The file transmission device provided by the embodiment of the present invention is used to perform the file transmission method provided by the embodiment shown in FIG. 2 of the present invention, and has a corresponding function module, and the implementation principle and technical effects thereof are similar, and details are not described herein again.

In another possible implementation manner of the embodiment of the present invention, the configuration file further includes a verification mode, and the transmission rule corresponding to the file transmission path may also be an index verification file, where the file transmission device shown in FIG. 6 is used. In addition, the transmission module 12 further includes: a determining unit 15 configured to determine a state of the verification mode; correspondingly, the searching unit 13 is further configured to be in the file transmission path when the determining unit 15 determines that the verification mode is the opening check Searching for a file corresponding to the index entry of the index check file in the source address; the transfer unit 14 is further configured to transfer the file found by the search unit 13 to the target address; in addition, if the transfer rule in the embodiment further includes the file Format, search unit 13, It is further configured to: when the determining unit 15 determines that the verification mode is the off-check, search for a file matching the file format in the source address of the file transmission path to generate a file transfer list; and the transmission unit 14 is further configured to be based on the search unit 13 The generated file transfer list in turn transfers the corresponding file in the source address to the destination address. The state of the verification mode in this embodiment further includes the opening or closing of the file processing for further processing after the file transmission is performed by the transmission unit 14. Optionally, the determining unit 15 in the embodiment is further configured to transmit After the unit 14 transfers the file to the target address, the status of the verification mode is determined. Accordingly, the file transmission apparatus further includes: a processing unit 16 configured to set the index when the determination unit 15 determines that the verification mode is the open file processing. The verification file is stored in the preset path of the network end to which the target address belongs; the processing unit 16 is further configured to delete the index verification file or store the index verification file to the source address when the determination unit 15 determines that the verification mode is the closed file processing. In the default path of the network.

It should be noted that, in this embodiment, the file to be transmitted is searched by the index check file, and the file to be transmitted is searched for by the file format in the foregoing embodiment, which is a method for separately searching for a file provided by the embodiment of the present invention. Alternatively, in another possible implementation manner of this embodiment, if there is no file format in the transmission rule, and the determining unit 15 determines that the verification mode is the shutdown check, the method for transmitting the file may also be: The unit 14 is further arranged to transmit all the files in the source address of the file transfer path to the target address when the judging unit 15 judges that the check mode is the close check.

The file transmission device provided by the embodiment of the present invention is used to execute the file transmission method provided by the embodiment shown in FIG. 3 of the present invention, and has a corresponding function module, and the implementation principle and the technical effect thereof are similar, and details are not described herein again.

Optionally, on the basis of the foregoing embodiments of the present invention, the configuration file in this embodiment may further include a compression mode, which is illustrated by using the structure of the file transmission apparatus shown in FIG. The transmission module 12 in this embodiment is further configured to: a compression unit 17 configured to compress the found file when the determination unit 15 determines that the compression mode is on. The transmission module 12 is further configured to transmit the compressed file of the compression unit 17 to the target address; in addition, the transmission module 12 is further configured to: when the determination unit 15 determines that the compression mode is off, the file to be found Transfer to the destination address.

Optionally, the configuration file in this embodiment may further include a backup mode and a backup address, and then The judging unit 15 in the file transfer device shown in FIG. 6 is further configured to determine the state of the backup mode after the transfer module 12 transfers the found file to the target address; accordingly, the processing unit 16 is further configured to be in the judging unit. When it is determined that the backup mode is off, the file that has been transferred to the target address in the source address of the file transfer path is deleted; and the processing unit 16 is further configured to set the file transfer path when the determining unit 15 determines that the backup mode is on. The file whose source address has been transferred to the destination address is backed up to the backup address of the network side to which the source address belongs.

The file transmission device provided by the embodiment of the present invention is used to execute the file transmission method provided by the embodiment shown in FIG. 4 of the present invention, and has a corresponding function module, and the implementation principle and the technical effect thereof are similar, and details are not described herein again.

In a specific implementation, on the basis of the foregoing embodiments of the present invention, if the file to be transmitted is a dynamic data stream, for example, the content of the voice call or the video conference is transmitted to the HDFS, the file transfer operation needs to be repeatedly performed, and The current configuration file can be changed by the setting of the user. The configuration file in this embodiment further includes a scanning time interval. Based on the structure of the file transmission device shown in FIG. 6 , the device provided in this embodiment further includes: The module 18 is configured to perform a timing operation after the transmission module 12 transmits the found file to the target address; correspondingly, the transmission module 12 is further configured to reach the scanning time interval timing at the timing of the timing module 18, according to the current The configuration file searches for a file conforming to the transmission rule of the current configuration file in the source address of the file transfer path of the current configuration file, and transfers the found file to the destination address of the file transfer path of the current configuration file.

One of ordinary skill in the art will appreciate that all or a portion of the above steps may be performed by a program to instruct related hardware, such as a processor, which may be stored in a computer readable storage medium, such as a read only memory, disk or optical disk. Wait. Alternatively, all or part of the steps of the above embodiments may also be implemented using one or more integrated circuits. Correspondingly, each module/unit in the above embodiment may be implemented in the form of hardware, for example, by implementing an integrated circuit to implement its corresponding function, or may be implemented in the form of a software function module, for example, executing a program stored in the memory by a processor. / instruction to achieve its corresponding function. The invention is not limited to any specific form of combination of hardware and software.

While the embodiments of the present invention have been described above, the described embodiments are merely for the purpose of understanding the invention and are not intended to limit the invention. Any modification and variation in the form and details of the embodiments may be made by those skilled in the art without departing from the spirit and scope of the invention. The scope defined by the appended claims shall prevail.

Industrial applicability

The above technical solution realizes mass transfer of massive files with the Hadoop system only by the configuration file set by the user, and the transferred multiple files can be formatted data files and/or unformatted data files; solved in Hadoop with Hadoop In the way that the system performs file transfer, the file transfer operation mode is complicated and the efficiency is low.

Claims

A file transfer method includes:

And generating, by the user input, a configuration file, where the configuration file includes a file transmission path and a transmission rule corresponding to the file transmission path, where the file transmission path includes a source address and a target address of the file to be transmitted;

Determining, according to the configuration file, a file that meets the transmission rule in a source address of the file transmission path, and transmitting the found file to the target address, where the transmitted file includes a formatted data file and / or unformatted data files.
The file transfer method according to claim 1, wherein the transfer rule includes a file format, the file in the source address of the file transfer path is searched for a file conforming to the transfer rule, and the found file is transferred To the target address, including:

Finding a file matching the file format in a source address of the file transmission path, and generating a file transfer list;

And correspondingly transferring the corresponding file in the source address to the target address according to the file transfer list.
The file transfer method according to claim 2, wherein

The file transmission path includes a first path and a second path, and the first path and the second path both include at least one file transmission path.

Searching, in a source address of the file transmission path, a file matching the file format, and generating a file transfer list; and sequentially transmitting a corresponding file in the source address to the target according to the file transfer list In the address, including:

And searching for a file matching the file format corresponding to the current file transmission path in the source addresses of the first path and the second path, generating a current file transfer list, and sequentially performing the according to the current file transfer list. A corresponding file in the source address of the current file transfer path is transferred to the target address.
The file transfer method according to claim 1, wherein the configuration file further comprises a check mode, the transfer rule including an index check file,

And searching for the file that meets the transmission rule in the source address of the file transmission path, and transmitting the found file to the target address, including:

When the verification mode is an open check, searching for a file corresponding to an index entry of the index check file in a source address of the file transfer path, and transmitting the found file to the target address in;

The method further includes: after the file is transferred to the target address, when the verification mode is open file processing, storing the index verification file to a pre-network to which the target address belongs In the path, when the verification mode is to close the file processing, the index verification file is deleted or the index verification file is stored in the preset path of the network end of the source address.
The file transfer method according to claim 4, wherein the transfer rule further comprises a file format,

The searching for the file that meets the transmission rule in the source address of the file transmission path, and transmitting the found file to the target address, further includes:

When the verification mode is a shutdown check, searching for a file matching the file format in a source address of the file transmission path, generating a file transfer list, and sequentially, according to the file transfer list, the source address The corresponding file in the file is transferred to the target address.
The file transfer method according to any one of claims 1 to 5, wherein the configuration file further includes a compressed mode,

Transmitting the found file to the target address, including:

When the compression mode is on, compressing the searched file, and transmitting the compressed file to the target address;

The searched file is transferred to the target address when the compressed mode is off.
The file transfer method according to any one of claims 1 to 5, wherein the configuration file further includes a backup mode and a backup address.

The method further includes:

After transmitting the found file to the target address, deleting the file that has been transmitted to the target address in the source address of the file transmission path when the backup mode is off;

When the backup mode is enabled, the file that has been transferred to the target address in the source address of the file transmission path is backed up to the backup address of the network end to which the source address belongs.
The file transfer method according to any one of claims 1 to 5, wherein the configuration file further includes a scan time interval,

The method further includes:

After searching for a file that meets the transmission rule in the source address of the file transmission path, and transmitting the found file to the target address, when the timing time reaches the scanning time interval, according to the current configuration The file searches for a file conforming to the transmission rule of the current configuration file in a source address of a file transmission path of the current configuration file, and transmits the found file to a destination address of a file transmission path of the current configuration file. .
A file transfer device comprising:

a configuration module, configured to generate a configuration file according to a user input, where the configuration file includes a file transmission path and a transmission rule corresponding to the file transmission path, where the file transmission path includes a source address and a destination address of the file to be transmitted ;

a transmission module, configured to search for a file that meets the transmission rule in a source address of the file transmission path according to a configuration file generated by the configuration module, and transmit the found file to the target address, where The transferred files include formatted data files and/or unformatted data files.
The file transfer device according to claim 9, wherein said transfer rule includes a file format,

The transmission module includes:

a searching unit configured to search for a file matching the file format in a source address of the file transmission path, and generate a file transfer list;

And a transmitting unit, configured to sequentially transmit the corresponding file in the source address to the target address according to the file transfer list generated by the searching unit.
The file transfer device according to claim 10, wherein said file transfer path comprises a first path and a second path, said first path and said second path each comprising at least one of said file transfer paths,

The searching unit is configured to sequentially search for a file matching the file format corresponding to the current file transmission path in the source addresses of the first path and the second path, and generate a current file transmission list;

And the transmitting unit is configured to sequentially transmit the corresponding file in the source address of the current file transmission path to the target address according to the current file transmission list generated by the searching unit.
The file transfer apparatus according to claim 9, wherein said configuration file further comprises a check mode, said transfer rule including an index check file,

The transmission module includes:

a determining unit, configured to determine a state of the verification mode;

a searching unit, configured to: when the determining unit determines that the verification mode is an open check, searching for a file corresponding to an index entry of the index check file in a source address of the file transfer path;

a transmission unit, configured to transmit the file found by the searching unit to the target address;

The file transfer module further includes:

a processing unit, configured to: after the determining unit determines that the verification mode is open file processing, after the determining unit transmits the file to the target address, storing the index verification file to the network to which the target address belongs In the preset path of the end; when the determining unit determines that the verification mode is closed file processing, deleting the index check file or storing the index check file to the network terminal Set the path.
The file transfer device according to claim 12, wherein said transfer rule further comprises a file format,

The searching unit is further configured to: when the determining unit determines that the verification mode is a shutdown check, search for a file matching the file format in a source address of the file transmission path, and generate a file transfer list. .
The file transfer device of claim 9, the configuration file further comprising a compressed mode,

The transmission module includes: a determining unit, a compression unit, and a transmission unit;

The determining unit is configured to determine a state of the compressed mode;

The compression unit is configured to perform compression processing on the found file when the determining unit determines that the compression mode is on;

The transmission unit is configured to transmit the compressed file of the compression unit to the target address; and when the determining unit determines that the compression mode is off, transmit the found file to In the target address.
The file transfer device according to claim 10, wherein said configuration file further includes a backup mode and a backup address.

The transmission module further includes: a determining unit and a processing unit;

The determining unit is configured to determine a state of the backup mode after the transmitting unit transmits the found file to the target address;

The processing unit is configured to, when the determining unit determines that the backup mode is off, delete a file that has been transmitted to the target address in a source address of the file transmission path; and determine, by the determining unit When the backup mode is enabled, the file that has been transferred to the target address in the source address of the file transmission path is backed up to the backup address of the network end to which the source address belongs.
The file transfer device according to any one of claims 9 to 13, wherein the configuration file further includes a scan time interval,

The file transfer device further includes:

a timing module, configured to perform a timing operation after the transmission module transmits the found file to the target address;

The transmission module is further configured to: when the timing time of the timing module reaches the scanning time interval, searching for the current configuration file according to the current configuration file in the source address of the file transmission path of the current configuration file. Transfer the file of the rule and transfer the found file to the destination address of the file transfer path of the current profile.
A computer storage medium having stored therein computer executable instructions for performing the method of any one of claims 1-8.