CN110881062A - File transmission method, device, equipment and storage medium based on big data - Google Patents
File transmission method, device, equipment and storage medium based on big data Download PDFInfo
- Publication number
- CN110881062A CN110881062A CN201910992524.6A CN201910992524A CN110881062A CN 110881062 A CN110881062 A CN 110881062A CN 201910992524 A CN201910992524 A CN 201910992524A CN 110881062 A CN110881062 A CN 110881062A
- Authority
- CN
- China
- Prior art keywords
- file
- task
- target
- server
- configuration parameters
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
- G06F16/1824—Distributed file systems implemented using Network-attached Storage [NAS] architecture
- G06F16/183—Provision of network file services by network file servers, e.g. by using NFS, CIFS
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention relates to the technical field of big data, in particular to a file transmission method, a file transmission device, file transmission equipment and a storage medium based on big data. The method comprises the following steps: receiving a task request, acquiring configuration parameters in the task request, and determining a task type through the configuration parameters; acquiring a source file, distributing the source file to a preset temporary directory for storage to obtain a target file, and verifying the target file; if the verification fails, terminating the task and returning a failure prompt; if the verification is successful, a task request corresponding to the task type is initiated to the target server, and the target file is distributed to the target server. By the method, the data of the source file is synchronized to the target server from the client, and is synchronized back to the client from the target server, so that the safe and reliable data uploading and downloading functions are realized.
Description
Technical Field
The invention relates to the technical field of big data, in particular to a file transmission method, a file transmission device, file transmission equipment and a storage medium based on big data.
Background
With the increasing application of a distributed System infrastructure (hadoop), the requirements of data transmission on a distributed File System (HDFS) and a linux server are more and more urgent, and there are various ways to download data on the HDFS to the local and upload data on the local to the HDFS, for example, the requirements can be realized by encapsulating HDFS shell commands such as HDFS dfs-put and HDFS dfs-get into a script.
However, the existing various modes have the following disadvantages: configuration according to the requirement cannot be supported friendly; not meeting the security specifications, not allowing all users to use HDFS shell commands to upload and download, etc.
Disclosure of Invention
In view of the above, it is necessary to provide a file transmission method based on big data and related apparatus for solving the problem of uploading and downloading data between the HDFS and the linux server.
A file transmission method based on big data comprises the following steps:
receiving a task request started by a user, acquiring configuration parameters in the task request, and determining a task type according to the configuration parameters, wherein the task type comprises an uploading task and a downloading task;
acquiring a source file, distributing the source file to a preset temporary directory for storage to obtain a target file, and verifying the target file stored in the temporary directory;
if the verification fails, terminating the task and returning a failure prompt;
if the verification is successful, initiating a task request corresponding to the task type to a target server, and distributing the target file to the target server.
In one possible design, the receiving a user-initiated task request includes:
receiving a task request started by a user through a preset task scheduling tool, searching whether an existing task identical to the task request exists, if so, starting a sub-process corresponding to the task request after finishing the sub-process corresponding to the existing task, and if not, directly starting the sub-process corresponding to the task request.
In one possible design, the obtaining the configuration parameter in the task request and determining the task type according to the configuration parameter include:
acquiring configuration parameters in a task request, and carrying out format verification on the configuration parameters;
if the verification fails, a failure prompt is returned, and if the verification passes, the configuration parameters are analyzed to obtain parameters including the task type, the IP list of the server, the file owner and the directory where the file is located.
In one possible design, the obtaining a source file, distributing the source file to a preset temporary directory for storage, obtaining a target file, and verifying the target file stored in the temporary directory includes:
if the task type is an uploading task, reading the source file through the configuration parameters, and if the task type is a downloading task, directly reading the source file from a distributed file system;
storing the read source file in a preset temporary directory to obtain a target file;
and obtaining the file size and the file name of the target file, comparing the file size and the file name with the file size and the file name of the source file respectively, if the file size of the target file is consistent with the file size of the source file, and the file name of the target file is consistent with the file name of the source file, the verification is successful, otherwise, the verification fails.
In one possible design, if the task type is an upload task, reading the source file according to the configuration parameters includes:
after determining the position of a source file through an IP list of a server, a file owner and a directory where the file is located in the configuration parameters, reading the source file;
and if the reading of the source file fails, acquiring the source file from a backup server.
In a possible design, if the verification is successful, initiating a task request corresponding to the task type to a target server, and distributing the target file to the target server includes:
judging whether the target file needs to be compressed and decompressed or not according to the configuration parameters, if not, directly initiating a task request corresponding to the task type to a target server, and distributing the target file to the target server;
if not, if the task type is an uploading task, decompressing the target file, then initiating a task request corresponding to the task type to a target server, and distributing the decompressed target file to the target server;
if the task type is a downloading task, after the target file is compressed, a task request corresponding to the task type is initiated to a target server, and the compressed target file is distributed to the target server.
In a possible design, if the verification is successful, initiating a task request corresponding to the task type to a target server, and distributing the target file to the target server includes:
if the task type is an uploading task, the target server is a distributed file system, and the target file is directly distributed to the distributed file system;
if the task type is a downloading task, the target file is distributed to the target server after the position of the target server is determined through an IP list of the server, a file owner and a directory where the file is located in the configuration parameters, and if the target file cannot be distributed to the target server, the target file is synchronized to a backup server.
A big data based file transfer apparatus, comprising:
the system comprises a receiving request module, a processing module and a processing module, wherein the receiving request module is used for receiving a task request started by a user, acquiring configuration parameters in the task request and determining a task type according to the configuration parameters, and the task type comprises an uploading task and a downloading task;
the temporary storage module is used for acquiring a source file, distributing the source file to a preset temporary directory for storage to obtain a target file, and verifying the target file stored in the temporary directory;
the verification failure module is used for terminating the task and returning a failure prompt if the verification fails;
and the file transmission module is used for initiating a task request corresponding to the task type to a target server and distributing the target file to the target server if the verification is successful.
A computer device comprising a memory and a processor, the memory having stored therein computer-readable instructions which, when executed by the processor, cause the processor to perform the steps of the above-described big data based file transfer method.
A storage medium storing computer-readable instructions which, when executed by one or more processors, cause the one or more processors to perform the steps of the above-described big data based file transfer method.
The file transmission method, the device, the equipment and the storage medium based on the big data comprise the steps of receiving a task request started by a user, obtaining configuration parameters in the task request, and determining the task type according to the configuration parameters, wherein the task type comprises an uploading task and a downloading task; acquiring a source file, distributing the source file to a preset temporary directory for storage to obtain a target file, and verifying the target file stored in the temporary directory; if the verification fails, terminating the task and returning a failure prompt; if the verification is successful, initiating a task request corresponding to the task type to a target server, and distributing the target file to the target server. By the method, a better file transmission channel is established between the user side and the target server, the data of the source file is synchronized to the target server from the user side, and the data is synchronously recycled to the user side from the target server, so that the safe and reliable data uploading and downloading functions are realized, the data can be automatically decompressed before uploading, and the data can be automatically compressed after downloading, thereby not only meeting the safety specification, but also supporting the parameter configuration of the file server.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention.
FIG. 1 is a flow chart of a big data based file transfer method in one embodiment of the invention;
FIG. 2 is a flowchart of step S1 according to an embodiment of the present invention;
FIG. 3 is a flowchart of step S2 according to an embodiment of the present invention;
FIG. 4 is a flowchart of step S4 according to an embodiment of the present invention;
FIG. 5 is a diagram illustrating a file transfer according to an embodiment of the present invention;
fig. 6 is a block diagram of a big data based file transfer apparatus according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
Fig. 1 is a flowchart of a big data based file transmission method according to an embodiment of the present invention, and as shown in fig. 1, a big data based file transmission method includes the following steps:
step S1, determining the task type: receiving a task request started by a user, acquiring configuration parameters in the task request, and determining the task type through the configuration parameters, wherein the task type comprises an uploading task and a downloading task.
In the step, a file server (Fileserver) is additionally arranged between a source file server and a distributed file system (HDFS), data are synchronized into the distributed file system from the source file server through the file server, and the data are synchronized back to the source file server from the distributed file system. The file server is used as a bridge between the source file server and the distributed file system to realize the tasks of uploading and downloading data.
A user starts a task request command through a client of a source file server, and the client can be deployed outside a cluster or in the cluster, so that a source node during uploading and a target node during downloading can be inside the cluster or outside the cluster. When a user starts a task request command through a client, the user can schedule the command through a task scheduling tool (oozie), after the user starts the command, the task scheduling tool starts a task, and the task is sent to any node of a cluster through a resource manager (yann) to directly execute a shell command starting script. The script executes ssh commands provided by linux, connects to the file server to initiate a task request, and the file server receives the task request.
In one embodiment, in step S1, receiving a user-initiated task request includes:
receiving a task request started by a user through a preset task scheduling tool, searching whether an existing task identical to the task request exists, if so, starting a sub-process corresponding to the task request after finishing the sub-process corresponding to the existing task, and if not, directly starting the sub-process corresponding to the task request.
Before uploading or downloading tasks, whether the same tasks exist is judged in a task searching mode, during searching, a ps-ef searching command of a liunx server is adopted for searching, if the same tasks exist, the pulled-up sub-process corresponding to the tasks is forcibly killed through a kill-9 command provided by the liunx, and then the sub-process corresponding to the tasks is pulled up.
In the embodiment, data abnormity caused by problems of data collision and the like in the process of repeated uploading or downloading of the same task is prevented by a mode of searching the task and forcibly stopping the same task, and the uniqueness of the task is ensured.
In one embodiment, in step S1, obtaining the configuration parameters in the task request, and determining the task type according to the configuration parameters, as shown in fig. 2, includes:
step S101, verifying configuration parameters: and acquiring the configuration parameters in the task request, and performing format verification on the configuration parameters.
Configuration parameters are contained in a task request submitted by a user, real parameters are assigned to the form parameters through mapping, the parameters adopt a preset shell command script to carry out non-null and format verification, and whether the logic judgment of the preset specification is met or not is verified.
Step S102, analyzing configuration parameters: if the verification fails, a failure prompt is returned, and if the verification passes, the configuration parameters are analyzed to obtain parameters including the task type, the IP list of the server, the file owner and the directory where the file is located.
And if the data in the configuration parameters do not meet the preset specification or the configuration parameters are null, the task request is considered to be failed or have risks, and a failure prompt is returned to a task requesting party, such as a client of a user. If the verification is passed, the configuration parameters need to be analyzed, and in the step, the parameters are analyzed through a preset shell command script to obtain an IP list (srcips), a file owner (srrcuer), a directory (filepath) where the file is located, a task type (processtype), a maximum bandwidth (bandwidth) and a file synchronization time (trigger) of the server. The task type is a downloading request, the task type is a downloading task, the uploading request is obtained when the processtype parameter obtained through analysis is upload, and the task type is an uploading task.
In this embodiment, before parsing the configuration parameters in the task request and determining the task type, non-empty format verification is also performed on the configuration parameters, so as to ensure the security of the request. On the premise of passing the verification, all the parameters are analyzed through the preset command script, so that data can be uploaded or downloaded in a personalized mode according to the user-defined parameters, and personalized requirements of users are met.
Step S2, temporarily storing and verifying: the method comprises the steps of obtaining a source file, distributing the source file to a preset temporary directory for storage, obtaining a target file, and verifying the target file stored in the temporary directory.
When the source file is obtained, the file server obtains the source file in different modes according to the configuration parameters and different task types. In order to meet the safety specification, after the source file is acquired, the file server specially opens a preset appointed temporary directory for each user, the source file is temporarily stored in the temporary directory, and after the integrity of the file is confirmed, an uploading or downloading request is sent.
In the step, before the source file is obtained, the maximum bandwidth and the file synchronization time in the configuration parameters are read, and at the beginning of the file synchronization time, the step is adopted to carry out bandwidth limitation by using the maximum bandwidth parameter to obtain the source file, further realize user personalized configuration and realize the purpose of file transmission.
In one embodiment, step S2, as shown in fig. 3, includes:
step S201, acquiring a source file: and if the task type is an uploading task, reading the source file through the configuration parameters, and if the task type is a downloading task, directly reading the source file from the distributed file system.
If the task type is an uploading task, reading a source file through configuration parameters, and the method comprises the following steps: and reading the source file after determining the position of the source file through the IP list of the server, the file owner and the directory where the file is located in the configuration parameters. And if the source file reading fails, obtaining the source file from the backup server.
When the request acquired by the file server is an uploading request and the task type is an uploading task, the source file is acquired by analyzing three parameters of an IP list of the server, a file owner and a directory where the file is located, wherein the three parameters are acquired after the configuration parameters are analyzed.
And when uploading the request, if the source server corresponding to the IP list of the server is unavailable, acquiring the source file from the backup server. A mapping relation exists between a source server and a backup server, an IP list of the server is obtained through a parameter srcips, the parameters in the IP list of the server are separated by commas, the first IP of the parameters is the source server of a source file, and the parameters are separated by commas to be the backup server. The method for judging whether the source server is available is as follows: whether a file owner has permission or not, whether a source server can be used or not and whether a directory where the file is located exists or not are sensed by executing a preset shell command script.
When the request acquired by the file server is a downloading request and the task type is a downloading task, the file triggered by the distributed file system is directly read as the source file when the source file is acquired.
Step S202, storing the target file: and storing the read source file in a preset temporary directory to obtain a target file.
The file server presets an independent file synchronization user (filesyncUser) for each user, develops a corresponding designated temporary directory for temporarily storing the source file, and once the source file is stored in the temporary directory, the source file becomes a target file.
Step S203, verifying the target file: and obtaining the file size and the file name of the target file, comparing the file size and the file name with the file size and the file name of the source file respectively, if the file size of the target file is consistent with the file size of the source file, and the file name of the target file is consistent with the file name of the source file, the verification is successful, otherwise, the verification fails.
The integrity of the file is checked by the consistency of the file size and the file name: respectively obtaining the file size and the file name of a target file stored in the temporary directory, the file size and the file name of a source file, comparing the file size and the file name respectively, and if the file size and the file name are consistent, passing the verification.
In the embodiment, the source file is acquired in different modes according to the task type determined by the configuration file, different temporary directories are developed according to different users to ensure that the whole file transmission meets the safety specification, and when the source server of the source file is unavailable, a backup server is also configured to ensure that the source file is acquired. Before the request is made to the target server, the target file in the temporary directory is also checked, and the security of the file needing to be transmitted is further ensured.
Step S3, check failure prompt: if the verification fails, the task is terminated, and a failure prompt is returned.
When any one or both of the file size and the file name of the target file and the file size and the file name of the source file are inconsistent, the verification is considered to be failed, and storage risk or data abnormality exists, at the moment, the task can be terminated through the task scheduling tool, and a failure prompt is returned to the user.
Step S4, transfer destination file: if the verification is successful, a task request corresponding to the task type is initiated to the target server, and the target file is distributed to the target server.
The file server initiates a task request corresponding to the task type to the target server only under the condition that the file size and the file name of the target file are completely consistent with those of the source file, if the task type is an uploading task, the task request is an uploading request, and if the task type is a downloading task, the task request is a downloading request.
When the task type is an uploading task, the target server is a distributed file system, and the target file is directly distributed to the distributed file system. I.e. the target file needs to be uploaded into the directory triggered from the distributed file system.
When the task type is a downloading task, after the position of the target server is determined through an IP list of the server, a file owner and a directory where the file is located in the configuration parameters, the target file is distributed to the target server, and if the target file cannot be distributed to the target server, the target file is synchronized to the backup server.
Similar to the uploading request to obtain the source file, in the downloading task process, if the target server corresponding to the IP list of the server is unavailable, the target file is synchronized to the backup server. The mapping relation exists between the target server and the backup server, an IP list of the server is obtained through a parameter srcips, the parameters in the IP list of the server are separated by commas, the first IP of the parameters is the target server of the target file, and the later IP separated by the commas is the backup server. The method for judging whether the target server is available is as follows: whether a file owner has permission or not, whether a target server can be used or not and whether a directory where the file is located exists or not are sensed by executing a preset shell command script.
In one embodiment, step S4, as shown in fig. 4, further includes:
step S401, determining compression parameters: and judging whether the target file needs to be compressed and decompressed or not through the configuration parameters, if not, directly initiating a task request corresponding to the task type to the target server, and distributing the target file to the target server.
Before initiating a task request to a target server, the file server further determines whether the target file needs to be compressed or decompressed, and in step 1), the configuration parameters further include a compression suffix (compressosfix) parameter, and if the parameter exists, for example, the compression suffix: and gz, considering that compression or decompression is needed, if the parameter is empty, the task does not need to be compressed or decompressed, and the file server does not do other work and directly makes a task request to the target server.
Step S402, decompression when uploading a task: otherwise, if the task type is the uploading task, decompressing the target file, then initiating a task request corresponding to the task type to the target server, and distributing the decompressed target file to the target server.
And if compression and decompression are needed, decompressing the target file and storing the target file in the distributed file system when the task is uploaded. For example, when the compressostfix in the configuration parameters: and gz, namely decompressing the gz compressed target file and storing the decompressed target file into the trigger directory of the distributed file system.
Step S403, compression when downloading the task: if the task type is a downloading task, after the target file is compressed, a task request corresponding to the task type is initiated to the target server, and the compressed target file is distributed to the target server.
And if compression and decompression are needed, compressing and storing the target file when the task is downloaded. For example, when the compressostfix in the configuration parameters: and gz, which is a target file downloaded from the distributed file system, is compressed into a compressed file in a gz format and then is stored in a target server determined by three configuration parameters, namely an IP list of the server, a file owner and a directory where the file is located.
The compressed and decompressed format can be a format of tar, gz, zip, tgz, etc., and the purpose of automatic decompression before uploading and automatic compression after downloading can be realized. In the embodiment, whether the target file is compressed or decompressed is determined by reading the analyzed configuration parameters, so that more personalized configurations are provided for the user.
In one embodiment, as shown in FIG. 5, the file server performs scheduling and opening of tasks via a task scheduling tool. When uploading a task, the file server Safely Copies (SCP) a source file from a directory specified by the source file server to a specified temporary directory to obtain a target file, and uploads the target file to the distributed file system after verifying the target file. And when the task is downloaded, the file server downloads the source file from the distributed file system to the appointed temporary directory to obtain the target file, and the target file is distributed to the target server through safe copying.
In the file transmission method based on big data, the file server is additionally arranged between the source file server and the distributed file system, the file server realizes synchronization of the source file server and the distributed file system, a user can perform personalized configuration in the synchronization process, and the file server performs personalized operations such as compression or decompression, file transmission bandwidth limitation, file transmission time determination and the like according to different configuration parameters. The file server also realizes the uploading and downloading work of the shell command by opening up a special user temporary directory, so that the invention conforms to the safety standard. When the source file server or the target file server cannot be used, the stability of synchronization is ensured through the backup server.
In one embodiment, a big data based file transfer apparatus is provided, as shown in fig. 6, including:
the receiving request module is used for receiving a task request started by a user, acquiring configuration parameters in the task request and determining the task type through the configuration parameters, wherein the task type comprises an uploading task and a downloading task;
the temporary storage module is used for acquiring a source file, distributing the source file to a preset temporary directory for storage to obtain a target file, and verifying the target file stored in the temporary directory;
the verification failure module is used for terminating the task and returning a failure prompt if the verification fails;
and the file transmission module is used for initiating a task request corresponding to the task type to the target server and distributing the target file to the target server if the verification is successful.
In one embodiment, a computer device is provided, which includes a memory and a processor, the memory stores computer readable instructions, and when the computer readable instructions are executed by the processor, the processor implements the steps in the big data based file transmission method of the foregoing embodiments when executing the computer readable instructions.
In one embodiment, a storage medium storing computer-readable instructions is provided, and the computer-readable instructions, when executed by one or more processors, cause the one or more processors to perform the steps of the big data based file transmission method of the above embodiments. The storage medium may be a nonvolatile storage medium.
Those skilled in the art will appreciate that all or part of the steps in the methods of the above embodiments may be implemented by associated hardware instructed by a program, which may be stored in a computer-readable storage medium, and the storage medium may include: a Read Only Memory (ROM), a Random Access Memory (RAM), a magnetic or optical disk, or the like.
The technical features of the embodiments described above may be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the embodiments described above are not described, but should be considered as being within the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express some exemplary embodiments of the present invention, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the inventive concept, which falls within the scope of the present invention. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. A file transmission method based on big data is characterized by comprising the following steps:
receiving a task request started by a user, acquiring configuration parameters in the task request, and determining a task type according to the configuration parameters, wherein the task type comprises an uploading task and a downloading task;
acquiring a source file, distributing the source file to a preset temporary directory for storage to obtain a target file, and verifying the target file stored in the temporary directory;
if the verification fails, terminating the task and returning a failure prompt;
if the verification is successful, initiating a task request corresponding to the task type to a target server, and distributing the target file to the target server.
2. The big data based file transmission method according to claim 1, wherein the receiving of the user-initiated task request comprises:
receiving a task request started by a user through a preset task scheduling tool, searching whether an existing task identical to the task request exists, if so, starting a sub-process corresponding to the task request after finishing the sub-process corresponding to the existing task, and if not, directly starting the sub-process corresponding to the task request.
3. The big data-based file transmission method according to claim 1, wherein the obtaining of the configuration parameter in the task request and the determining of the task type through the configuration parameter include:
acquiring configuration parameters in a task request, and carrying out format verification on the configuration parameters;
if the verification fails, a failure prompt is returned, and if the verification passes, the configuration parameters are analyzed to obtain parameters including the task type, the IP list of the server, the file owner and the directory where the file is located.
4. The big-data-based file transmission method according to claim 3, wherein the obtaining a source file, distributing the source file to a preset temporary directory for storage, obtaining a target file, and verifying the target file stored in the temporary directory comprises:
if the task type is an uploading task, reading the source file through the configuration parameters, and if the task type is a downloading task, directly reading the source file from a distributed file system;
storing the read source file in a preset temporary directory to obtain a target file;
and obtaining the file size and the file name of the target file, comparing the file size and the file name with the file size and the file name of the source file respectively, if the file size of the target file is consistent with the file size of the source file, and the file name of the target file is consistent with the file name of the source file, the verification is successful, otherwise, the verification fails.
5. The big data-based file transmission method according to claim 4, wherein reading the source file according to the configuration parameters if the task type is an upload task comprises:
after determining the position of a source file through an IP list of a server, a file owner and a directory where the file is located in the configuration parameters, reading the source file;
and if the reading of the source file fails, acquiring the source file from a backup server.
6. The big data-based file transmission method according to claim 1, wherein if the verification is successful, initiating a task request corresponding to the task type to a target server, and distributing the target file to the target server includes:
judging whether the target file needs to be compressed and decompressed or not according to the configuration parameters, if not, directly initiating a task request corresponding to the task type to a target server, and distributing the target file to the target server;
if not, if the task type is an uploading task, decompressing the target file, then initiating a task request corresponding to the task type to a target server, and distributing the decompressed target file to the target server;
if the task type is a downloading task, after the target file is compressed, a task request corresponding to the task type is initiated to a target server, and the compressed target file is distributed to the target server.
7. The big data-based file transmission method according to claim 3, wherein if the verification is successful, initiating a task request corresponding to the task type to a target server, and distributing the target file to the target server includes:
if the task type is an uploading task, the target server is a distributed file system, and the target file is directly distributed to the distributed file system;
if the task type is a downloading task, the target file is distributed to the target server after the position of the target server is determined through an IP list of the server, a file owner and a directory where the file is located in the configuration parameters, and if the target file cannot be distributed to the target server, the target file is synchronized to a backup server.
8. A big data-based file transmission device is characterized by comprising:
the system comprises a receiving request module, a processing module and a processing module, wherein the receiving request module is used for receiving a task request started by a user, acquiring configuration parameters in the task request and determining a task type according to the configuration parameters, and the task type comprises an uploading task and a downloading task;
the temporary storage module is used for acquiring a source file, distributing the source file to a preset temporary directory for storage to obtain a target file, and verifying the target file stored in the temporary directory;
the verification failure module is used for terminating the task and returning a failure prompt if the verification fails;
and the file transmission module is used for initiating a task request corresponding to the task type to a target server and distributing the target file to the target server if the verification is successful.
9. A computer device comprising a memory and a processor, the memory having stored therein computer-readable instructions which, when executed by the processor, cause the processor to perform the steps of the big data based file transfer method according to any of claims 1 to 7.
10. A storage medium storing computer readable instructions, which when executed by one or more processors, cause the one or more processors to perform the steps of the big data based file transfer method according to any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910992524.6A CN110881062A (en) | 2019-10-18 | 2019-10-18 | File transmission method, device, equipment and storage medium based on big data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910992524.6A CN110881062A (en) | 2019-10-18 | 2019-10-18 | File transmission method, device, equipment and storage medium based on big data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110881062A true CN110881062A (en) | 2020-03-13 |
Family
ID=69728027
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910992524.6A Pending CN110881062A (en) | 2019-10-18 | 2019-10-18 | File transmission method, device, equipment and storage medium based on big data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110881062A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111538637A (en) * | 2020-04-27 | 2020-08-14 | 苏州浪潮智能科技有限公司 | Method and system for automatically monitoring file synchronization result |
| CN111783407A (en) * | 2020-07-17 | 2020-10-16 | 西思艾(北京)软件有限公司 | Electronic form creating system |
| CN111880740A (en) * | 2020-07-29 | 2020-11-03 | 平安科技（深圳）有限公司 | Data processing method, device, computer system and readable storage medium |
| CN111917659A (en) * | 2020-07-23 | 2020-11-10 | 上海聚音信息科技有限公司 | Concurrent data transmission method, system, device and computer readable storage medium |
| CN112380292A (en) * | 2020-12-07 | 2021-02-19 | 厦门熵基科技有限公司 | Biological template data synchronization method, device, equipment and storage medium |
| CN111880740B (en) * | 2020-07-29 | 2024-05-24 | 平安科技（深圳）有限公司 | Data processing method, device, computer system and readable storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109474606A (en) * | 2018-12-04 | 2019-03-15 | 平安科技（深圳）有限公司 | Document transmission method, device, computer equipment and storage medium |
| CN110069451A (en) * | 2019-03-28 | 2019-07-30 | 浪潮卓数大数据产业发展有限公司 | A kind of method and device of HDFS storage small documents |
| CN110247951A (en) * | 2019-05-08 | 2019-09-17 | 建信融通有限责任公司 | A kind of file transfer control method, device, equipment and storage medium |
-
2019
- 2019-10-18 CN CN201910992524.6A patent/CN110881062A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109474606A (en) * | 2018-12-04 | 2019-03-15 | 平安科技（深圳）有限公司 | Document transmission method, device, computer equipment and storage medium |
| CN110069451A (en) * | 2019-03-28 | 2019-07-30 | 浪潮卓数大数据产业发展有限公司 | A kind of method and device of HDFS storage small documents |
| CN110247951A (en) * | 2019-05-08 | 2019-09-17 | 建信融通有限责任公司 | A kind of file transfer control method, device, equipment and storage medium |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111538637A (en) * | 2020-04-27 | 2020-08-14 | 苏州浪潮智能科技有限公司 | Method and system for automatically monitoring file synchronization result |
| CN111538637B (en) * | 2020-04-27 | 2022-08-12 | 苏州浪潮智能科技有限公司 | Method and system for automatically monitoring file synchronization result |
| CN111783407A (en) * | 2020-07-17 | 2020-10-16 | 西思艾(北京)软件有限公司 | Electronic form creating system |
| CN111917659A (en) * | 2020-07-23 | 2020-11-10 | 上海聚音信息科技有限公司 | Concurrent data transmission method, system, device and computer readable storage medium |
| CN111880740A (en) * | 2020-07-29 | 2020-11-03 | 平安科技（深圳）有限公司 | Data processing method, device, computer system and readable storage medium |
| WO2021174828A1 (en) * | 2020-07-29 | 2021-09-10 | 平安科技（深圳）有限公司 | Data processing method, apparatus, computer system, and readable storage medium |
| CN111880740B (en) * | 2020-07-29 | 2024-05-24 | 平安科技（深圳）有限公司 | Data processing method, device, computer system and readable storage medium |
| CN112380292A (en) * | 2020-12-07 | 2021-02-19 | 厦门熵基科技有限公司 | Biological template data synchronization method, device, equipment and storage medium |
| CN112380292B (en) * | 2020-12-07 | 2023-11-17 | 厦门熵基科技有限公司 | Biological template data synchronization method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110881062A (en) | File transmission method, device, equipment and storage medium based on big data | |
| CN109040252B (en) | File transmission method, system, computer device and storage medium | |
| WO2021217868A1 (en) | Git-based project version release method and apparatus, device, and medium | |
| US9256414B2 (en) | Method, device and system for running application | |
| CN109634619B (en) | Trusted execution environment implementation method and device, terminal device and readable storage medium | |
| CN108235763B (en) | Detection system, method and related device | |
| WO2018170741A1 (en) | Operating system updating method and device | |
| CN107623735B (en) | Accurate updating and upgrading system and method based on openssl in credit investigation machine system | |
| US11307905B2 (en) | Method and a device comprising an edge cloud agent for providing a service | |
| CN112860282B (en) | Cluster plug-in upgrading method, device and server | |
| CN113300854B (en) | Edge node capability expansion method, system and expansion box | |
| CN112416391A (en) | System upgrading method and device, computer equipment and storage medium | |
| CN113448601B (en) | Software package installation method, device, equipment and system based on content distribution network | |
| CN113590169A (en) | Application deployment method, application deployment system, and computer-readable storage medium | |
| CN106326777A (en) | System mirror image document signature method and system, client and server | |
| CN113835642A (en) | Distributed storage network construction method based on IPFS and distributed storage network | |
| CN106161517B (en) | The method and apparatus for realizing cloud storage access by cloud file system | |
| CN115878138B (en) | Application pre-download method, device, computer and storage medium | |
| US20130227162A1 (en) | Management of Data Upload Speed | |
| CN110769016A (en) | File uploading method and computer storage medium | |
| CN113806815A (en) | File signing method and system | |
| CN113315795A (en) | Synchronization method and device of cloud host mirror image and storage medium | |
| CN110673869B (en) | Library file loading method, device and system | |
| CN109617707A (en) | Without configuration method, device, computer equipment and the storage medium for burying a parameter | |
| CN112685102B (en) | Gateway plug-in hot loading method, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |