CN110650164A - File uploading method and device, terminal and computer storage medium - Google Patents
File uploading method and device, terminal and computer storage medium Download PDFInfo
- Publication number
- CN110650164A CN110650164A CN201810672831.1A CN201810672831A CN110650164A CN 110650164 A CN110650164 A CN 110650164A CN 201810672831 A CN201810672831 A CN 201810672831A CN 110650164 A CN110650164 A CN 110650164A
- Authority
- CN
- China
- Prior art keywords
- file
- uploading
- state
- positioning information
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/12—Applying verification of the received information
- H04L63/123—Applying verification of the received information received data contents, e.g. message integrity
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Abstract
The application discloses a method, a device, a terminal and a computer storage medium for uploading frequency files, comprising the following steps: acquiring index information of a file; acquiring the positioning information of the file from the index information, and storing the positioning information into a queue storage system; and acquiring the positioning information from the queue storage system, and further acquiring the file by utilizing the positioning information and uploading the file to a distributed file system for calling. By the method, the file is uploaded to the distributed system conveniently and quickly.
Description
Technical Field
The present application relates to the field of file uploading technologies, and in particular, to a file uploading method, an uploading apparatus, an uploading terminal, and a computer storage medium.
Background
In daily life, either downloaded files or locally stored files are generally stored in a local server or memory.
However, as the data storage requirement from the network or from the enterprise itself is getting larger and larger, the file system in the traditional meaning cannot meet the existing requirements of large capacity, high reliability, easy expansion and the like due to insufficient space, limited processing capability and the like. Therefore, in recent years, distributed file systems have been developed rapidly due to their advantages of high availability, high performance, redundant backup, and the like.
However, although the distributed file system solves the problem of file storage with high performance and easy expansion, when uploading a file, the existing ordinary uploading mode cannot meet the requirement of uploading a large amount of files because the files which need to be uploaded are large.
Disclosure of Invention
The technical problem mainly solved by the application is to provide a file uploading method, an uploading device, an uploading terminal and a computer storage medium, which can conveniently and quickly upload files to a distributed system.
In order to solve the above technical problem, the first technical solution adopted by the present application is: provided is a file uploading method, comprising the following steps: acquiring index information of a file; acquiring the positioning information of the file from the index information, and storing the positioning information into a queue storage system; and acquiring the positioning information from the queue storage system, and further acquiring a file by using the positioning information and uploading the file to the distributed file system for calling.
Wherein, after obtaining the index information of the file, the method comprises the following steps: storing the index information into a database; setting the state of the index information in the database as an initial state; after obtaining the positioning information from the queue storage system, the method comprises the following steps: and if the positioning information is acquired, changing the state of the index information in the database from the initial state to the first state.
Wherein the positioning information comprises a storage path of the file;
obtaining and uploading files to a distributed file system using the location information includes: if the uploading is successful, changing the state of the index information in the database from the first state to the second state, and storing the address of the index information stored in the distributed file system in the database; if the uploading fails because the storage path is invalid, changing the state of the index information in the database from the first state to a third state; if uploading fails because the content of the file is empty, changing the state of the index information in the database from the first state to a fourth state; if uploading fails due to network reasons, the state of the index information in the database is changed from the first state to the initial state.
The step of obtaining the index information of the file comprises the following steps: downloading a file meeting the set conditions from a third-party server; establishing connection with a third-party server, and downloading files meeting set conditions in batches from the third-party server according to set frequency; detecting the downloaded file and judging whether the file is complete or not; if the file is not complete, the file is downloaded again; and if the file is complete, executing the step of acquiring the index information of the file.
The steps of detecting the downloaded file and judging whether the file is complete specifically comprise: obtaining the MD5 value of the file, comparing the MD5 value with the original MD5 value of the file, and judging whether the MD5 value is equal to the original MD5 value; if the MD5 value is equal to the original MD5 value, the file is determined to be complete; otherwise, the audio is determined to be incomplete.
The queue storage system is a Redis storage system; the steps of obtaining the positioning information from the queue storage system, obtaining the file by using the positioning information and uploading the file to the distributed file system specifically comprise: determining a file to be uploaded; and acquiring positioning information from the redis storage system in a single-thread serial mode, and acquiring a file by using the positioning information and uploading the file to the distributed file system.
The steps of obtaining the positioning information from the queue storage system, obtaining the file by using the positioning information and uploading the file to the distributed file system further comprise: and deleting the positioning information of the uploaded file in the queue storage system.
In order to solve the above technical problem, the second technical solution adopted by the present application is: provided is a file uploading apparatus including: the index information acquisition module is used for acquiring the index information of the file; the positioning information acquisition module is used for acquiring the positioning information of the file from the index information and storing the positioning information into a queue storage system; and the file uploading module is used for acquiring the positioning information from the queue storage system, acquiring the file by utilizing the positioning information and uploading the file to the distributed file system.
The index information acquisition module is also used for storing the index information into a database;
setting the state of the index information in the database to be an initial state; the positioning information obtaining module is further configured to change the state of the index information in the database from the initial state to a first state if the positioning information is obtained.
Wherein the positioning information comprises a storage path of the file;
the file uploading module is further used for changing the state of the index information in the database from the first state to a second state if the uploading is successful, and storing the address of the index information stored in the distributed file system in the database; if the uploading fails because the storage path is invalid, changing the state of the index information in the database from the first state to a third state; if uploading fails because the content of the file is empty, changing the state of the index information in the database from the first state to the fourth state; and if the uploading fails for network reasons, changing the state of the index information in the database from the first state to the initial state.
The file uploading device further comprises a downloading module, wherein the downloading module is used for establishing connection with a third-party server and downloading files meeting set conditions in batches from the third-party server according to set frequency; detecting the downloaded file and judging whether the file is complete or not; if the file is not complete, the file is downloaded again, and if the file is complete, the index information acquisition module acquires the index information of the file.
The downloading module is specifically configured to obtain an MD5 value of the file, compare the MD5 value with an original MD5 value of the file, and determine whether the MD5 value is equal to the original MD5 value; determining that the file is complete if the MD5 value is equal to the original MD5 value; otherwise, the file is determined to be incomplete.
Wherein the queue storage system is a redis storage system; the positioning information acquisition module is specifically used for determining a file to be uploaded; and acquiring the positioning information from the redis storage system in a single-thread serial mode, acquiring the file by using the positioning information and uploading the file to the distributed file system.
The file uploading module is further configured to delete the location information of the uploaded file in the queue storage system after the location information is obtained from the queue storage system, the file is obtained by using the location information and uploaded to the distributed file system.
In order to solve the above technical problem, the third technical solution adopted by the present application is: the file uploading terminal comprises a processor and a communication circuit which are coupled with each other, wherein the communication circuit is used for being matched with the communication circuit to realize the file uploading method of any one of the above embodiments when working with a third-party server processor.
In order to solve the above technical problem, a fourth technical solution adopted by the present application is: there is provided a computer storage medium having stored thereon program data which when executed by a processor implements the file upload method of any of the above.
Compared with the prior art, the beneficial effects of this application are: the file uploading method and the file uploading system have the advantages that the index information of the file is obtained, the positioning information of the file is obtained from the index information, and compared with the prior art, the positioning information of the file can be extracted more quickly, the file is identified at a higher speed in the follow-up file uploading process, the server congestion in the file uploading process is avoided, and the uploading speed is increased. The positioning information is stored in the queue storage system, the positioning information is obtained from the queue storage system, and the file acquisition speed is further increased by the mode of obtaining from the queue storage system, and simultaneously, the disorder and omission of the file uploading sequence can not be caused, so that the files can be uploaded to the distributed file system quickly and completely.
Drawings
FIG. 1 is a schematic flow chart diagram illustrating an embodiment of an uploading method of the present document;
FIG. 2 is a schematic flow chart diagram illustrating another embodiment of an uploading method of the present document;
FIG. 3 is a schematic flow chart diagram illustrating a further embodiment of an uploading method of the present document;
FIG. 4 is a schematic structural diagram of an embodiment of an uploading apparatus of the present document;
FIG. 5 is a schematic structural diagram of another embodiment of an uploading device;
FIG. 6 is a schematic structural diagram of an embodiment of an upload terminal according to the present disclosure;
FIG. 7 is a schematic structural diagram of an embodiment of a computer storage medium according to the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The distributed file system is an open-source lightweight distributed file system, and can effectively balance the problems of large-capacity storage and load balance due to good file storage, file synchronization and file access functions.
Specifically, as shown in fig. 1, fig. 1 is a schematic flow chart of an embodiment of an uploading method of the present application. The file uploading method of the embodiment comprises the following steps:
step 101: index information of the file is acquired.
The file comprises at least one of an audio file, a picture file, a video file and a document file.
In order to speed up the identification of the file to be uploaded, in the embodiment, first, index information of the file is acquired from a database.
The index information includes at least one of a storage address, a serial number, a file acquisition time, and a file size of the file in the database, and for example, the file is an audio file such as a recording file, the index information includes a start time and an end time of the recording, identification information of both parties of the call such as a telephone number, and a serial number of the audio file, and the serial number is a unique ID number. After the storage of the index information of the file is completed, in order to facilitate the distinction of different processing states or validity of the file, the state words of the index information corresponding to the file are all set to be in an initial state in the database, for example, the initial state is set to be 0.
Step 102: and acquiring the positioning information of the file from the index information, and storing the positioning information into a queue storage system.
The positioning information at least comprises a storage path and a number of the file.
The queue memory may be any queue memory, such as a kafaka queue. In the embodiment, in order to improve the rate of uploading the file to the distributed file system by the terminal, the positioning information of the file is stored in the redis storage system, so that the terminal or the server is more efficient when calling the storage path of the file.
The Redis storage system is a high-performance key-value storage system, and provides 5 data types: the data types all support push/pop, add/remove, intersection union, difference and richer operations, and the operations are atomic and have high processing speed. In the embodiment, capturing and storing of the positioning information is realized by calling the operation instruction in the Redis storage system.
In the present embodiment, a linked list data structure is used as a message queue for the file location information. Specifically, after acquiring the storage path and the number of the file from the index information stored in the database, the storage path and the number of the file in the database are acquired as information of a specific field of the index information is acquired, and the storage path and the number are pushed to the redis storage system through a push operation. The number of the stored positioning information at a time may be single or multiple, and only the setting is required in the configuration file, which is not limited herein.
In order to distinguish index information of which the storage of the positioning information is finished from unfinished index information, in a preferred embodiment, after the storage of the positioning information of each piece of index information is finished, whether the positioning information is acquired is judged, if the positioning information is acquired, the state of the index information in the database is changed from an initial state to a first state, for example, a state word of the index information in the database is changed from 0 to 1, and if the positioning information is not acquired, for example, prompt information of acquisition failure is returned, the original state of the index information in the database is maintained to be acquired again.
By the mode of changing the state, whether the positioning information of the index information is successfully stored can be effectively and quickly determined, repeated processing or missing processing is avoided, and the working efficiency is improved.
Step 103: and acquiring the positioning information from the queue storage system, acquiring the file by using the positioning information and uploading the file to the distributed file system.
In order to facilitate subsequent query, storage and use of the file, the file is further uploaded to the distributed file system in the embodiment.
Preferably, the distributed system adopted by the embodiment is a Fast distributed file system.
Specifically, in the embodiment, a redis storage system is adopted, after a file to be uploaded is determined, if the file to be uploaded is determined according to a preset condition date, a storage path in the positioning information of the file to be uploaded is acquired from the redis storage system in a single-thread serial mode, the storage path is called to extract the file from the terminal server, and the file is uploaded to the distributed file system. In the calling process, whether the file is correct or not can be checked through the serial number, and the problem of file uploading errors is avoided.
In the embodiment, the storage address of the file to be uploaded is acquired from the redis storage system in a single-thread serial mode, system or network disorder caused by simultaneous multi-thread processing can be avoided, excessive task waiting can not be caused even if the redis storage system executes in a single thread due to the millisecond-level processing speed of the redis storage system, and the uploading requirement of the file can be effectively met.
In a specific embodiment, a pop operation is performed to obtain a storage path of a file from a redis storage system, and the storage path is used as a call parameter to be transmitted to a corresponding template method, such as an upLoad () method, where the template method obtains the file by calling the storage address and uploads the file to a distributed file system, so as to facilitate subsequent calls.
In an alternative embodiment, in order to save queue resources, such as redis resources, and increase processing speed, each set of location information is called and then deleted from the redis storage system.
Further, in order to save resources of the terminal server, after each file is successfully uploaded, the file is deleted from the terminal or the server.
In any of the above embodiments, in order to know whether the file is successfully uploaded, the uploading status of the file is further detected, for example, the uploading status of the file is determined by feedback information of the distributed file system.
Specifically, if the file is uploaded successfully, the distributed file system returns the storage address url of the file in the distributed file system, and after receiving the feedback, the terminal or the server stores the address of the file stored in the distributed file system in the database, and changes the state of the index information in the database to a second state, for example, changes the state word of the index information from 1 to 3.
If the file uploading is unsuccessful, the reason for the uploading failure may be many, so that the reason for the uploading failure needs to be analyzed. Specifically, an abnormal event which fails to upload currently is captured, and the state of the index information in the database is changed according to the reason of the abnormal event.
Firstly, judging whether the positioning information of the file, such as a storage path, is valid, if so, judging whether the format of the storage path is correct or whether the storage path is empty, if not, obtaining that the storage path is empty, then determining that the storage path is invalid, and changing the state of the index information in the database to a third state, if so, changing the state word of the index information from 1 to 2.
If the storage path is valid, judging whether the content of the file to be uploaded is empty, if the information returned by the distributed file system is FileNotFoundException, namely the prompt information of the file cannot be found, determining that the content of the file to be uploaded is empty, changing the state of the index information of the file in the database to a fourth state which is normal for uploading, and if the state word of the index information is changed from 1 to 4.
In the file uploading process, an uncontrollable abnormal condition such as network jitter may inevitably occur to cause file uploading failure, and the state of the index information in the database is updated to an initial state, for example, the state word is changed from 1 to 0.
When the terminal or the processor searches for the file to be uploaded again, the file to be uploaded is in the initial state, the positioning information of the index information is obtained again, the positioning information is stored in a queue such as a redis storage system, a storage address in the redis storage system is called to extract the file from the database, and the file is uploaded to the distributed file system. The file is automatically uploaded from the triggering system again by updating the state of the index information in the database, so that the file loss can be effectively avoided, and manual operation can be reduced.
It should be noted that, the above process of determining the cause of the abnormal event is not fixed, and the cause of the upload failure may be flexibly adjusted according to needs, for example, whether the file is empty is determined first, or whether the network is jittered is determined first, and the technical scheme that the state of the file in the database is changed according to the upload cause belongs to the protection scope of the present application.
In another embodiment, when a file to be uploaded is stored in a third-party server, in order to upload a file that belongs to the local area of the implementation, the file needs to be downloaded from the third-party server first, specifically, as shown in fig. 2, fig. 2 is a flowchart of another embodiment of the file uploading method of the present application. The uploading method of the embodiment comprises the following steps:
step 201: and downloading the file meeting the set conditions from the third-party server.
Specifically, a connection is established with the third-party server through the terminal or a local server where the terminal is located, and then a file meeting the set conditions is downloaded from the third-party server.
The terminal comprises intelligent equipment such as a PC (personal computer), a tablet personal computer and a smart phone.
The setting condition is range limiting information of the downloaded file, such as identity information or attribution name.
When the file is generated continuously, the frequency of audio download is limited, such as by setting the frequency to once per day or once per week.
In one embodiment, to make the data downloading process smoother, a jsch open source framework under JAVA architecture is used to operate a third party server, such as an SFTP server.
Optionally, when there are multiple file downloading tasks in the terminal or the server, in order to avoid crosstalk, repeated downloading, or missed downloading between the multiple downloading tasks under the same configuration, in a preferred embodiment, a comment operation is performed on each downloading task, and an action range of the current task is specified, for example, by adding a spring comment, which is not limited herein.
Further, the terminal of the embodiment supports breakpoint download, that is, when the download is not completed, the current breakpoint record can be retained, and when the same terminal is connected again next time, the breakpoint can be downloaded without re-downloading, which saves time and processor resources.
Step 202: and acquiring the index information of the file.
After the download is completed, the file is stored in a database or a server of the terminal, and the downloaded file is generally a compressed file, so the compressed file is decompressed first.
The process of obtaining the index information is the same as that in step 101, please refer to the related text description in step 101, and is not repeated again.
Step 203: and acquiring the positioning information of the file from the index information, and storing the positioning information into a queue storage system.
The step is the same as the step 102, and please refer to the description of the relevant text in the step 102, which is not described herein again.
Step 204: and acquiring the positioning information from the queue storage system, acquiring the file by using the positioning information and uploading the file to the distributed file system.
The step is the same as step 103, and please refer to the related text description in step 103, which is not described herein again.
Compared with the prior art, the file uploading method and the file uploading device have the advantages that the file uploading method and the file uploading device can quickly extract the positioning information of the file, the subsequent file identification speed during file uploading is increased, the server congestion during file uploading is avoided, and the uploading speed is increased. The positioning information is stored in the queue storage system, the positioning information is obtained from the queue storage system, and the file acquisition speed is further increased by the mode of obtaining from the queue storage system, and simultaneously, the disorder and omission of the file uploading sequence can not be caused, so that the files can be uploaded to the distributed file system quickly and completely.
In another embodiment, the storage path and the number of the file are saved in the redis storage system, so that the terminal or the processor is more efficient in calling the storage path of the file due to the millisecond-level processing speed of the redis storage system.
In another embodiment, when downloading a file, although the download is shown to be completed, the download may not be complete due to network jitter, crosstalk, or other reasons
Step 301: after downloading the file meeting the set conditions from the third-party server, the method also comprises the following steps:
step 302: and detecting the downloaded file and judging whether the file is complete or not.
If the detected file is complete, step 303 is performed. If the file is detected to be incomplete, the method returns to the step 301.
The MD5 check checks the correctness of the data by performing a hash operation on the received file. By means of a hash operation, a hash value can be calculated and the calculated hash value is compared with the hash value transmitted with the data. If the two values are the same, the transmitted data is complete and has not been tampered. In this embodiment, the MD5 value is detected to determine whether the received file is complete.
Specifically, the terminal or the processor acquires the MD5 value of the downloaded file, compares the MD5 value with the original MD5 value of the file provided by the third-party server, and determines whether the two values are the same. If the MD5 value is the same as the original MD5 value of the file provided by the third party server, the file is determined to be complete, and if the MD5 value is not equal to the original MD5 value, the file is determined to be incomplete.
303: index information of the file is acquired.
304: and acquiring the positioning information of the file from the index information, and storing the positioning information into a queue storage system.
305: and acquiring the positioning information from the queue storage system, acquiring the file by using the positioning information and uploading the file to the distributed file system for calling.
Wherein, the steps 303-305 are the same as the steps 101-104, and please refer to fig. 1 and the related text description for details, which are not repeated herein.
Different from the above embodiments, in the present embodiment, after the file is downloaded, the integrity of the downloaded file is detected, and when the file is detected to be incomplete, the file is directly downloaded again, so that each subsequent file to be uploaded is ensured to be complete, and unsuccessful uploading caused by incomplete files is solved from the source. In addition, in the embodiment, the detection can be performed after the downloading, so that the incomplete file is prevented from being further processed, and the memory resource of the processor is saved.
Referring to fig. 4, fig. 4 is a schematic structural diagram of an embodiment of an uploading apparatus of the present application. As shown in fig. 4, the uploading device of the present embodiment includes an index information obtaining module 401, a positioning information obtaining module 402, and a file uploading module 403.
The index information obtaining module 401 is configured to obtain index information of a file.
The file comprises at least one of an audio file, a picture file, a video file and a document file. The index information includes at least one of a storage address, a serial number, a file acquisition time, and a file size of the file in the database, and for example, the file is an audio file such as a recording file, the index information includes a start time and an end time of the recording, identification information of both parties of the call such as a telephone number, and a serial number of the audio file, and the serial number is a unique ID number.
The index information obtaining module 401 is configured to implement step 101, and please refer to the description of the relevant file in step 101.
The positioning information obtaining module 402 is configured to obtain positioning information of the file from the index information, and store the positioning information in a queue storage system.
The positioning information at least comprises a storage path and a number of the file.
The queue memory may be any queue memory, such as a kafaka queue.
In the embodiment, in order to improve the rate of uploading the file to the distributed file system by the terminal, the positioning information of the file is stored in the redis storage system, so that the terminal or the server is more efficient when calling the storage path of the file.
In this embodiment, the positioning information obtaining module 402 is used to implement step 202, and please refer to the related text description in step 101.
The file uploading module 403 is configured to obtain the positioning information from the queue storage system, obtain the file by using the positioning information, and upload the file to the distributed file system.
The distributed system adopted by the embodiment is a Fast distributed file system.
In this embodiment, the file uploading module 403 obtains the storage address of the file to be uploaded from the redis storage system in a single-thread serial manner, which can avoid system or network disorder caused by simultaneous multithreading, and because of the millisecond-level processing speed of the redis storage system, even if the single-thread execution is performed, excessive task waiting is not caused, and the uploading requirement of the file can be effectively met.
Further, in order to save resources of the terminal server, the file uploading module 403 deletes each file from the terminal or the server after the file is successfully uploaded.
In any of the above embodiments, in order to know whether the file is uploaded successfully, the file uploading module 403 further detects the uploading status of the file, for example, determines the uploading status of the file through feedback information of the distributed file system.
For the detection process and the state change, please refer to the related text description in step 103, which is not described herein again.
In another embodiment, when a file to be uploaded is stored in a third-party server, the file needs to be downloaded from the third-party server in order to upload a file that is local to the third-party server, so as to distinguish the uploading terminal of the above embodiments, the file uploading apparatus of the present embodiment further includes a downloading module 504, where the downloading module 504 is configured to download a file that meets a set condition from the third-party server.
The terminal comprises intelligent equipment such as a PC (personal computer), a tablet personal computer and a smart phone.
The setting condition is range limiting information of the downloaded file, such as identity information or attribution name.
When the file is generated continuously, the frequency of audio download is limited, such as by setting the frequency to once per day or once per week.
The downloading module 504 downloads files in the same way and processing way as the step 201, and please refer to the step 201 again. In another embodiment, in order to ensure the integrity of the file, after the downloading of the file meeting the set conditions from the third-party server is completed, the downloading module 504 detects the downloaded file, determines whether the file is complete, and when the index information obtaining module 501 detects that the file is complete, obtains the index information of the file. And when the downloaded file is not complete, the file is downloaded again.
Specifically, the downloading module 504 obtains the MD5 value of the downloaded file, compares the MD5 value with the original MD5 value of the file provided by the third-party server, and determines whether the MD5 value and the original MD5 value are the same. If the MD5 value is the same as the original MD5 value of the file provided by the third party server, the file is determined to be complete, and if the MD5 value is not equal to the original MD5 value, the file is determined to be incomplete.
The downloading module 504 can ensure that each subsequently uploaded file is complete through the above judgment on the integrity of the downloaded file, and the uploading failure caused by incomplete files is solved from the source. In addition, in the embodiment, the detection can be performed after the downloading, so that the incomplete file is prevented from being further processed, and the memory resource of the processor is saved.
Compared with the prior art, the file uploading method and the file uploading device have the advantages that the file uploading method and the file uploading device can quickly extract the positioning information of the file, the subsequent file identification speed during file uploading is increased, the server congestion during file uploading is avoided, and the uploading speed is increased. The file uploading module stores the positioning information into the queue storage system, acquires the positioning information from the queue storage system, further accelerates the file acquisition speed in a mode of acquiring from the queue storage system, and simultaneously does not cause disorder and omission of the file uploading sequence, so that the files can be uploaded into the distributed file system quickly and completely.
In another embodiment, the storage path and the number of the file are saved in the redis storage system, so that the terminal or the processor is more efficient in calling the storage path of the file due to the millisecond-level processing speed of the redis storage system.
Referring to fig. 6, fig. 6 is a schematic structural diagram of an embodiment of an upload terminal of the present disclosure. The upload terminal 60 of the present embodiment includes a processor 601 and a communication circuit 602 coupled to each other. The communication circuit 602 is used to establish a connection with an external device.
The processor 601 is further configured to obtain index information of the file.
The file comprises at least one of an audio file, a picture file, a video file and a document file. The index information includes at least one of a storage address, a serial number, a file acquisition time, and a file size of the file in the database, and for example, the file is an audio file such as a recording file, the index information includes a start time and an end time of the recording, identification information of both parties of the call such as a telephone number, and a serial number of the audio file, and the serial number is a unique ID number.
The processor 601 is further configured to obtain location information of the file from the index information, and store the location information in a queue storage system. The positioning information at least comprises a storage path and a number of the file.
The queue memory may be any queue memory, such as a kafaka queue.
In the embodiment, in order to improve the rate of uploading the file to the distributed file system by the terminal, the positioning information of the file is stored in the redis storage system, so that the terminal or the server is more efficient when calling the storage path of the file.
The processor 601 is further configured to obtain the location information from the queue storage system, obtain the file using the location information, and upload the file to the distributed file system.
The distributed system adopted by the embodiment is a Fast distributed file system.
In this embodiment, the processor 601 obtains the storage address of the file to be uploaded from the redis storage system in a single-thread serial manner, which can avoid system or network disorder caused by simultaneous multi-thread processing, and because of the millisecond-level processing speed of the redis storage system, even if the single-thread execution is performed, excessive task waiting is not caused, and the uploading requirement of the file can be effectively met.
Further, in order to save resources of the terminal server, the processor 601 deletes each file from the terminal or the server after the file is successfully uploaded.
In any of the above embodiments, in order to know whether the file is successfully uploaded, the processor 601 further detects the uploading status of the file, for example, determines the uploading status of the file through feedback information of the distributed file system.
In another embodiment, when a file to be uploaded is stored in a third-party server, the file needs to be downloaded from the third-party server in order to upload a file belonging to the local implementation, so that the uploading terminal of the above embodiment is distinguished, and the processor 601 downloads a file meeting the set conditions from the third-party server.
The setting condition is range limiting information of the downloaded file, such as identity information or attribution name.
When the file is generated continuously, the frequency of audio download is limited, such as by setting the frequency to once per day or once per week.
In another embodiment, in order to ensure that the integrity processor 601 of the file detects the downloaded file after the downloading of the file meeting the set conditions from the third-party server is completed, whether the downloaded file is complete is determined, for example, by comparing the MD5 value of the downloaded file with the original MD5 value of the file, whether the MD5 value of the downloaded file is the same to determine whether the file is complete. And when the integrity of the file is detected, acquiring the index information of the file. And when the downloaded file is not complete, the file is downloaded again.
By the aid of the judgment of the integrity of the downloaded files, each subsequently uploaded file can be guaranteed to be complete, and uploading failure caused by incomplete files is solved from the source. In addition, in the embodiment, the detection can be performed after the downloading, so that the incomplete file is prevented from being further processed, and the memory resource of the processor is saved.
Compared with the prior art, the method and the device have the advantages that the processor acquires the index information of the file to be uploaded and acquires the positioning information of the file from the index information, and compared with the prior art, the positioning information of the file can be extracted more quickly, the subsequent file identification speed during uploading is increased, the server congestion during uploading is avoided, and the uploading speed is increased. The positioning information is stored in the queue storage system, the positioning information is obtained from the queue storage system, and the file acquisition speed is further increased by the mode of obtaining from the queue storage system, and simultaneously, the disorder and omission of the file uploading sequence can not be caused, so that the files can be uploaded to the distributed file system quickly and completely.
In another embodiment, the processor saves the storage path and the number of the file to the redis storage system, which makes the terminal or the processor more efficient in invoking the storage path of the file due to the millisecond level processing speed of the redis storage system.
Please refer to fig. 7, a schematic structural diagram of an embodiment of a memory device is further provided in the present application. In this embodiment, the storage device 70 stores processor-executable computer instructions 71, and the computer instructions 71 are used for executing the method in the above-described embodiment.
The storage device 70 may be a medium that can store computer instructions, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, or may be a server that stores the computer instructions, and the server may send the stored computer instructions to other devices for operation or may self-operate the stored computer instructions.
In the several embodiments provided in the present application, it should be understood that the disclosed method and apparatus may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, a division of a module or a unit is merely a logical division, and an actual implementation may have another division, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
Units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, a network device, or the like) or a processor (processor) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The above description is only for the purpose of illustrating embodiments of the present application and is not intended to limit the scope of the present application, and all modifications of equivalent structures and equivalent processes, which are made by the contents of the specification and the drawings of the present application or are directly or indirectly applied to other related technical fields, are also included in the scope of the present application.
Claims (10)
1. A file uploading method is characterized by comprising the following steps:
acquiring index information of the file;
acquiring positioning information of the file from the index information, and storing the positioning information into a queue storage system;
and acquiring the positioning information from the queue storage system, acquiring the file by using the positioning information and uploading the file to a distributed file system.
2. The upload method according to claim 1,
the obtaining of the index information of the file comprises:
storing the index information into a database;
setting the state of the index information in the database to be an initial state;
the obtaining the positioning information from the queue storage system comprises:
and if the positioning information is acquired, changing the state of the index information in the database from the initial state to a first state.
3. The upload method according to claim 2,
the positioning information comprises a storage path and a number of the file;
the obtaining the file and uploading the file to a distributed file system by using the positioning information comprises:
if the uploading is successful, changing the state of the index information in the database from the first state to a second state, and storing the address of the file stored in the distributed file system in the database;
if the uploading fails because the storage path is invalid, changing the state of the index information in the database from the first state to a third state;
if uploading fails because the content of the file is empty, changing the state of the index information in the database from the first state to the fourth state;
and if the uploading fails for network reasons, changing the state of the index information in the database from the first state to the initial state.
4. The uploading method according to claim 1, wherein the step of obtaining the index information of the file comprises:
establishing connection with a third-party server, and downloading the files meeting the set conditions in batches from the third-party server according to the set frequency;
detecting the downloaded file and judging whether the file is complete or not;
if the file is not complete, the file is downloaded again; and if the audio file is complete, executing the step of acquiring the index information of the file.
5. The uploading method according to claim 4, wherein the step of detecting the downloaded file and determining whether the file is complete specifically comprises:
obtaining an MD5 value of the file, comparing the MD5 value with an original MD5 value of the file, and judging whether the MD5 value is equal to the original MD5 value;
determining that the file is complete if the MD5 value is equal to the original MD5 value; otherwise, the file is determined to be incomplete.
6. A piece uploading method according to claim 1,
the queue storage system is a redis storage system;
the step of obtaining the location information from the queue storage system, obtaining the file by using the location information, and uploading the file to a distributed file system specifically includes:
determining a file to be uploaded;
and acquiring the positioning information from the redis storage system in a single-thread serial mode, acquiring the file by using the positioning information and uploading the file to the distributed file system.
7. The uploading method according to claim 1, wherein the steps of obtaining the location information from the queue storage system, obtaining the file using the location information, and uploading the file to a distributed file system further comprise:
and deleting the positioning information of the uploaded file in the queue storage system.
8. An uploading apparatus of a document, the uploading apparatus comprising:
the index information acquisition module is used for acquiring the index information of the file;
the positioning information acquisition module is used for acquiring the positioning information of the file from the index information and storing the positioning information into a queue storage system;
and the file uploading module is used for acquiring the positioning information from the queue storage system, acquiring the file by utilizing the positioning information and uploading the file to the distributed file system.
9. An upload terminal of a file, characterized in that the upload terminal comprises:
a processor and a communication circuit coupled to each other, the communication circuit being configured to establish a connection with a third party server, the processor being configured to implement the method for uploading a file according to any of claims 1-7.
10. A computer storage medium, characterized in that the computer storage medium has stored thereon program data which, when executed by a processor, implements the uploading method of a file according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810672831.1A CN110650164B (en) | 2018-06-26 | 2018-06-26 | File uploading method and device, terminal and computer storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810672831.1A CN110650164B (en) | 2018-06-26 | 2018-06-26 | File uploading method and device, terminal and computer storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110650164A true CN110650164A (en) | 2020-01-03 |
| CN110650164B CN110650164B (en) | 2021-03-05 |
Family
ID=69008871
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810672831.1A Active CN110650164B (en) | 2018-06-26 | 2018-06-26 | File uploading method and device, terminal and computer storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110650164B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112769906A (en) * | 2020-12-25 | 2021-05-07 | 三盟科技股份有限公司 | File breakpoint uploading method and system, computer equipment and storage medium |
| CN113726842A (en) * | 2021-07-09 | 2021-11-30 | 北京房江湖科技有限公司 | File uploading method and device, electronic equipment and storage medium |
| CN113986835A (en) * | 2021-10-27 | 2022-01-28 | 平安国际智慧城市科技股份有限公司 | Management method, device, equipment and storage medium for FastDFS distributed files |
| CN114301627A (en) * | 2021-11-29 | 2022-04-08 | 北京天融信网络安全技术有限公司 | Uploaded file security scanning method and device and computer readable storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101145933A (en) * | 2007-10-26 | 2008-03-19 | 北大方正集团有限公司 | Transmission method, receiving method and corresponding device for distribution system and media data |
| CN101567893A (en) * | 2009-05-26 | 2009-10-28 | 中兴通讯股份有限公司 | Method and system for uploading files in WEB application |
| CN104090913A (en) * | 2014-06-10 | 2014-10-08 | 深圳市深信服电子科技有限公司 | File operation method and device based on thin client |
| US20170185615A1 (en) * | 2015-04-02 | 2017-06-29 | Tencent Technology (Shenzhen) Company Limited | Data processing method and device |
| CN107508897A (en) * | 2017-08-31 | 2017-12-22 | 深圳市丰巢科技有限公司 | A kind of file uploading method and system based on distributed file storage system |
-
2018
- 2018-06-26 CN CN201810672831.1A patent/CN110650164B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101145933A (en) * | 2007-10-26 | 2008-03-19 | 北大方正集团有限公司 | Transmission method, receiving method and corresponding device for distribution system and media data |
| CN101567893A (en) * | 2009-05-26 | 2009-10-28 | 中兴通讯股份有限公司 | Method and system for uploading files in WEB application |
| CN104090913A (en) * | 2014-06-10 | 2014-10-08 | 深圳市深信服电子科技有限公司 | File operation method and device based on thin client |
| US20170185615A1 (en) * | 2015-04-02 | 2017-06-29 | Tencent Technology (Shenzhen) Company Limited | Data processing method and device |
| CN107508897A (en) * | 2017-08-31 | 2017-12-22 | 深圳市丰巢科技有限公司 | A kind of file uploading method and system based on distributed file storage system |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112769906A (en) * | 2020-12-25 | 2021-05-07 | 三盟科技股份有限公司 | File breakpoint uploading method and system, computer equipment and storage medium |
| CN113726842A (en) * | 2021-07-09 | 2021-11-30 | 北京房江湖科技有限公司 | File uploading method and device, electronic equipment and storage medium |
| CN113986835A (en) * | 2021-10-27 | 2022-01-28 | 平安国际智慧城市科技股份有限公司 | Management method, device, equipment and storage medium for FastDFS distributed files |
| CN114301627A (en) * | 2021-11-29 | 2022-04-08 | 北京天融信网络安全技术有限公司 | Uploaded file security scanning method and device and computer readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110650164B (en) | 2021-03-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110650164B (en) | File uploading method and device, terminal and computer storage medium | |
| US10838838B2 (en) | Method and apparatus for dealing with abnormality of application program and storage medium | |
| CN107315825B (en) | Index updating system, method and device | |
| WO2015183527A1 (en) | Synchronization system for multiple client devices | |
| CN106940699B (en) | Synchronous processing method, device, server and system for memory data | |
| CN110225078B (en) | Application service updating method, system and terminal equipment | |
| CN111367925A (en) | Data dynamic real-time updating method, device and storage medium | |
| CN110019873B (en) | Face data processing method, device and equipment | |
| CN111277639A (en) | Method and device for maintaining data consistency | |
| US10938773B2 (en) | Method and apparatus for synchronizing contact information and medium | |
| US10318385B2 (en) | Service recovery using snapshots and interservice messages | |
| CN115329170A (en) | Webpage crawling method, device, equipment and storage medium | |
| US8510426B2 (en) | Communication and coordination between web services in a cloud-based computing environment | |
| CN111274325B (en) | Platform automatic test method and system | |
| CN110620798A (en) | Control method, system, equipment and storage medium for FTP connection | |
| CN111831954A (en) | Content data updating method and device, computer equipment and storage medium | |
| CN112711466B (en) | Hanging affair inspection method and device, electronic equipment and storage medium | |
| CN112241332B (en) | Interface compensation method and device | |
| CN112732728A (en) | Data synchronization method and system | |
| CN112153148A (en) | Message accumulation processing method and device based on message receiving end | |
| US11953995B1 (en) | Centralized data backup platform supporting multiple data environments | |
| CN115114361B (en) | Resource retrieval method and system based on unified interface of container cloud platform | |
| CN115037595B (en) | Network recovery method, device, equipment and storage medium | |
| CN114185625A (en) | Method, apparatus, device and medium for confirming interface automation delay execution result | |
| CN115982285A (en) | Data processing method, device and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |