CN114401260A - Method, system, equipment and storage medium for uploading micro-service cluster file fragments - Google Patents
Method, system, equipment and storage medium for uploading micro-service cluster file fragments Download PDFInfo
- Publication number
- CN114401260A CN114401260A CN202111557192.2A CN202111557192A CN114401260A CN 114401260 A CN114401260 A CN 114401260A CN 202111557192 A CN202111557192 A CN 202111557192A CN 114401260 A CN114401260 A CN 114401260A
- Authority
- CN
- China
- Prior art keywords
- file
- uploaded
- uploading
- fragment
- storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 239000012634 fragment Substances 0.000 title claims abstract description 147
- 238000000034 method Methods 0.000 title claims abstract description 45
- 238000012545 processing Methods 0.000 claims abstract description 48
- 238000013467 fragmentation Methods 0.000 claims description 20
- 238000006062 fragmentation reaction Methods 0.000 claims description 20
- 238000004590 computer program Methods 0.000 claims description 14
- 230000001960 triggered effect Effects 0.000 claims description 5
- 230000008030 elimination Effects 0.000 claims description 2
- 238000003379 elimination reaction Methods 0.000 claims description 2
- 230000000694 effects Effects 0.000 abstract description 5
- 230000005540 biological transmission Effects 0.000 description 9
- 238000010586 diagram Methods 0.000 description 8
- 230000001360 synchronised effect Effects 0.000 description 5
- 238000013024 troubleshooting Methods 0.000 description 4
- 230000011218 segmentation Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/178—Techniques for file synchronisation in file systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The application relates to a method, a system, equipment and a storage medium for uploading files of a micro-service cluster in a fragment mode, wherein the method comprises the following steps: uploading the pre-fragmented fragment files to corresponding storage terminals through a service terminal, and returning the file information of the uploaded fragment files to a processing program; sending a corresponding processing command according to the file information, and re-initiating an uploading request for the fragmented file which fails to be uploaded; after all the fragment files are successfully uploaded, downloading all the fragment files from each storage terminal, combining the fragment files to obtain a complete file to be uploaded, uploading the file to be uploaded to the storage terminals, and simultaneously synchronizing the complete file to be uploaded between the storage terminals. By the method and the device, the problem that file fragments in the micro-service cluster are unsuccessfully uploaded is solved, and the effect that the file fragments can be successfully combined is achieved by ensuring that the file fragments can be successfully uploaded.
Description
Technical Field
The present application relates to the field of computer file transmission, and in particular, to a method, a system, a device, and a storage medium for fragmented uploading of a microservice cluster file.
Background
Because the microservice has the characteristics of high speed, stability, reliability, dynamic expansion of service capability and the like, the microservice is widely applied to some businesses of enterprises.
When a large file exceeding the memory is transmitted in the micro service cluster, the large file is generally fragmented, the fragmented files are transmitted through different nodes in the micro service cluster to improve the transmission efficiency, then the fragmented files are merged, and if a certain node does not successfully upload the fragmented files, all the fragmented files cannot be merged normally finally, so that the file upload fails, and therefore the problem of file fragment upload failure in the micro service cluster cannot be solved.
Aiming at the problem that file fragment uploading failure in a micro-service cluster cannot be solved in the related technology, no effective solution is provided at present.
Disclosure of Invention
The embodiment provides a method, a system, a device and a storage medium for uploading file fragments of a micro service cluster, so as to solve the problem that file fragment uploading failure in the micro service cluster cannot be solved in the related art.
In a first aspect, in this embodiment, a micro service cluster file fragment uploading method is provided, and includes:
uploading the pre-fragmented fragment files to corresponding storage terminals through a service terminal, and returning the file information of the uploaded fragment files to a processing program;
sending a corresponding processing command according to the file information, and re-initiating an uploading request for the fragmented file which fails to be uploaded;
after all the fragment files are successfully uploaded, downloading all the fragment files from each storage terminal, combining the fragment files to obtain a complete file to be uploaded, uploading the file to be uploaded to the storage terminals, and simultaneously synchronizing the complete file to be uploaded between the storage terminals.
In some embodiments, before the uploading, by the service terminal, the pre-fragmented file to a corresponding storage terminal and returning file information of the uploaded fragmented file to the processing program, the method further includes:
and in the processing program, based on a preset rule, carrying out fragmentation processing on the acquired file to be uploaded to obtain the fragmented file.
In some embodiments, the performing, based on a preset rule, fragmentation processing on the acquired file to be uploaded to obtain the fragmented file includes:
marking the fragment file to obtain a fragment mark;
and uploading the fragment file and the fragment mark to the service terminal.
In some embodiments, the sending a corresponding processing command according to the file information and reinitiating an upload request for the fragmented file that fails to be uploaded includes:
when the file information is that the fragmented file is not successfully uploaded to the storage terminal, sending the corresponding processing command to the service terminal according to the error type obtained from the file information;
and executing corresponding command content according to the processing command, and re-triggering the uploading request of the fragmented file with uploading failure.
In some embodiments, the executing, according to the processing command, corresponding command content, and re-triggering an upload request of the fragmented file for which upload is failed further includes:
in the service terminal, corresponding fault elimination processing is carried out by executing the command content;
and after the uploading request is re-triggered, updating the file information of the fragmented file according to a triggering result, and returning the updated file information to the processing program.
In some embodiments, the downloading all the fragmented files from each storage terminal, combining the fragmented files to obtain a complete file to be uploaded, and uploading the file to be uploaded to the storage terminal includes:
when the processing program receives the file information of the last fragment file, downloading all the uploaded fragment files from the storage terminal through the service terminal;
and combining the downloaded fragment files to obtain a complete file to be uploaded, and uploading the file to be uploaded to the storage terminal for storage.
In a second aspect, in this embodiment, a microservice cluster file fragment uploading system is provided, including: the system comprises a service cluster, a processing program module and a storage cluster;
the service cluster comprises a plurality of service terminals and is used for uploading the fragmented files subjected to pre-fragmentation to corresponding storage terminals and returning the file information of the uploaded fragmented files to the processing program module;
after all the fragment files are successfully uploaded, downloading all the fragment files from the storage cluster, combining the fragment files to obtain a complete file to be uploaded, and uploading the file to be uploaded to the storage cluster;
the processing program module sends a corresponding processing command according to the file information through the processing program and re-initiates an uploading request to the fragmented file with uploading failure;
the storage cluster comprises a plurality of storage terminals and is used for storing the fragment files and the files to be uploaded by the service cluster and synchronizing the complete files to be uploaded.
In some embodiments, the service cluster is further configured to:
when the processing program receives the file information of the last fragment file, downloading all the uploaded fragment files from the storage cluster;
and combining the downloaded fragment files to obtain the complete file to be uploaded, and uploading the file to be uploaded to the storage cluster for storage.
In a third aspect, in this embodiment, a computer device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the micro service cluster file fragment uploading method according to the first aspect is implemented.
In a fourth aspect, in this embodiment, a storage medium is provided, on which a computer program is stored, and when executed by a processor, the storage medium implements the micro service cluster file fragment uploading method described in the first aspect.
Compared with the related art, the method, the system, the device and the storage medium for uploading the micro-service cluster file fragments provided in the embodiment upload the fragmented file after pre-fragmentation to the corresponding storage terminal through the service terminal, and return the file information of the uploaded fragmented file to the processing program; sending a corresponding processing command according to the file information, and re-initiating an uploading request for the fragmented file which fails to be uploaded; after all the fragment files are successfully uploaded, all the fragment files are downloaded from each storage terminal, the complete files to be uploaded are obtained through combination, the files to be uploaded are uploaded to the storage terminals, and meanwhile the files to be uploaded are synchronized and complete among the storage terminals, so that the problem that file fragment uploading in a micro-service cluster fails is solved, and the effect of ensuring that file fragments can be successfully uploaded in the micro-service cluster is achieved.
The details of one or more embodiments of the application are set forth in the accompanying drawings and the description below to provide a more thorough understanding of the application.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:
FIG. 1 is a diagram illustrating a hardware configuration of a method for fragmented upload of microservice cluster files in an embodiment;
FIG. 2 is a block diagram that illustrates the architecture of a microservice cluster file shard upload system, according to an embodiment;
FIG. 3 is a flow diagram that illustrates a method for fragmented upload of microservice cluster files, according to an embodiment;
FIG. 4 is a flowchart of a method for fragmented upload of microservice cluster files in a preferred embodiment.
In the figure: 200. a microservice cluster file fragment uploading system; 210. service clustering; 220. a processing program module; 230. the cluster is stored.
Detailed Description
For a clearer understanding of the objects, aspects and advantages of the present application, reference is made to the following description and accompanying drawings.
Unless defined otherwise, technical or scientific terms used herein shall have the same general meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The use of the terms "a" and "an" and "the" and similar referents in the context of this application do not denote a limitation of quantity, either in the singular or the plural. The terms "comprises," "comprising," "has," "having," and any variations thereof, as referred to in this application, are intended to cover non-exclusive inclusions; for example, a process, method, and system, article, or apparatus that comprises a list of steps or modules (elements) is not limited to the listed steps or modules, but may include other steps or modules (elements) not listed or inherent to such process, method, article, or apparatus. Reference throughout this application to "connected," "coupled," and the like is not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. Reference to "a plurality" in this application means two or more. "and/or" describes an association relationship of associated objects, meaning that three relationships may exist, for example, "A and/or B" may mean: a exists alone, A and B exist simultaneously, and B exists alone. In general, the character "/" indicates a relationship in which the objects associated before and after are an "or". The terms "first," "second," "third," and the like in this application are used for distinguishing between similar items and not necessarily for describing a particular sequential or chronological order.
The method embodiments provided in the present embodiment may be executed in a terminal, a computer, or a similar computing device. For example, the method is executed on a terminal, and fig. 1 is a block diagram of a hardware structure of the terminal of the micro service cluster file fragment uploading method according to the embodiment. As shown in fig. 1, the terminal may include one or more processors 102 (only one shown in fig. 1) and a memory 104 for storing data, wherein the processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA. The terminal may also include a transmission device 106 for communication functions and an input-output device 108. It will be understood by those of ordinary skill in the art that the structure shown in fig. 1 is merely an illustration and is not intended to limit the structure of the terminal described above. For example, the terminal may also include more or fewer components than shown in FIG. 1, or have a different configuration than shown in FIG. 1.
The memory 104 may be used to store computer programs, for example, software programs and modules of application software, such as a computer program corresponding to the microservice file fragmentation uploading method in the present embodiment, and the processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, so as to implement the method described above. The memory 104 may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory located remotely from the processor 102, which may be connected to the terminal over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The transmission device 106 is used to receive or transmit data via a network. The network described above includes a wireless network provided by a communication provider of the terminal. In one example, the transmission device 106 includes a Network adapter (NIC) that can be connected to other Network devices through a base station to communicate with the internet. In one example, the transmission device 106 may be a Radio Frequency (RF) module, which is used to communicate with the internet in a wireless manner.
The micro-service cluster is a cluster structure consisting of a plurality of micro-service terminals, and all the micro-service terminals in each cluster provide the same service at the same time. Each service provided by the micro-service cluster is independent and independent, does not depend on each other, and only interacts in an interface or queue mode. The file fragment uploading system of the micro-service cluster is provided in the following embodiments, wherein the provided file fragment uploading service and the storage service are processed by separate micro-service clusters, and each service terminal and storage terminal is a separate individual container, such as a docker micro-server.
In this embodiment, a micro service cluster file fragment uploading system is provided, fig. 2 is a corresponding structural block diagram of the system in this embodiment, and as shown in fig. 2, the micro service cluster file fragment uploading system 200 includes: a service cluster 210, a handler module 220, and a storage cluster 230;
the service cluster 210 comprises a plurality of service terminals, and is used for uploading the pre-fragmented fragment files to corresponding storage terminals and returning the file information of the uploaded fragment files to the processing program module;
and after all the fragment files are successfully uploaded, downloading all the fragment files from the storage cluster, combining the fragment files to obtain a complete file to be uploaded, and uploading the file to be uploaded to the storage cluster.
Specifically, each service terminal uploads the received fragmented file subjected to pre-fragmentation to a corresponding storage terminal, and returns file information of the uploaded fragmented file. When the fragment file is uploaded successfully, the file information comprises some record information which is uploaded successfully; and when the uploading of the fragment file fails, returning the error type and the error information recording the uploading failure to the processing program.
Further, after all the fragment files are successfully uploaded, since the service cluster only provides service services, the stored fragment files need to be downloaded from the storage cluster, and are combined in any service terminal, and the original file to be uploaded is recovered and then uploaded to any storage terminal for storage. It should be noted that, in the storage cluster, the storage terminals may be synchronized, so that the final file to be uploaded is uploaded to any one of the storage terminals and then synchronized in the whole storage cluster.
The processing program module 220 sends a corresponding processing command according to the file information through a processing program, and re-initiates an upload request for the fragmented file with the upload failure.
Specifically, a processing program in the processing program module sends a corresponding processing command to the service terminal with failed uploading according to the error type in the file information returned by the service terminal when the uploading of the fragmented file fails, and the service terminal re-initiates the uploading request of the fragmented file.
The storage cluster 230 includes a plurality of storage terminals, and is configured to store the fragment file uploaded by the service cluster and the file to be uploaded, and synchronize the complete file to be uploaded.
Specifically, the fragment files uploaded by each service terminal are stored in each storage terminal in the storage cluster, and further, the files to be uploaded after the completion of the combination are synchronously stored among the storage terminals.
The above modules may be functional modules or program modules, and may be implemented by software or hardware. For a module implemented by hardware, the modules may be located in the same processor; or the modules can be respectively positioned in different processors in any combination.
By the micro-service cluster file fragment uploading system and the service cluster, the processing program module and the storage cluster thereof provided by the embodiment, the fragment files are uploaded to the storage cluster through the service terminal in the service cluster, and the uploading request is re-initiated according to the file information of the fragment files which are unsuccessfully uploaded, so that the problem of file fragment uploading failure in the micro-service cluster is solved, and the effect that the file fragments can be successfully uploaded is achieved by ensuring that the file fragments can be successfully uploaded.
In this embodiment, a method for uploading a file fragment of a micro service cluster is provided, and fig. 3 is a flowchart of the method in this embodiment, and as shown in fig. 3, the flowchart includes the following steps:
step S310, the pre-fragmented fragment file is uploaded to a corresponding storage terminal through the service terminal, and file information of the uploaded fragment file is returned to the processing program.
Specifically, each service terminal uploads the received fragmented file subjected to pre-fragmentation to a corresponding storage terminal, and returns file information of the uploaded fragmented file. When the fragment file is uploaded successfully, the file information comprises some record information which is uploaded successfully; and when the uploading of the fragment file fails, returning the error type and the error information recording the uploading failure to the processing program.
Step S320, sending a corresponding processing command according to the file information, and re-initiating an upload request for the fragmented file that fails to be uploaded.
Specifically, the error type and the corresponding processing command are defined in the processing program in advance, and the error type includes but is not limited to: when the error type is network fault, the corresponding processing command can be router reconnection; when the error type is a fault of insufficient hard disk memory, the corresponding processing command may be to clear a cache (delete cache) or to add a mount (mount) of the hard disk. And sending a corresponding processing command to the service terminal with failed uploading according to the error type in the file information returned by the service terminal when the uploading of the fragmented file fails, and restarting the uploading request of the fragmented file through the service terminal after the service terminal correspondingly performs troubleshooting.
And step S330, after all the fragment files are successfully uploaded, downloading all the fragment files from each storage terminal, combining the fragment files to obtain a complete file to be uploaded, uploading the file to be uploaded to the storage terminals, and synchronizing the complete file to be uploaded among the storage terminals.
Specifically, after steps S310 and S320, all the fragment files are successfully uploaded to the corresponding storage terminals, and since the service cluster only provides service, the stored fragment files need to be downloaded from the storage cluster, and merged in any service terminal, so as to recover the original file to be uploaded, and then uploaded to any storage terminal for storage. It should be noted that, in the storage cluster, the storage terminals may be synchronized, so that the final file to be uploaded is uploaded to any one of the storage terminals and then synchronized in the whole storage cluster.
In the file fragment uploading method in the prior art, fragment files after large file fragmentation are respectively transmitted through different nodes in a distributed system or a micro-service cluster, and then the fragment files are merged, so that once a certain node does not successfully upload a corresponding fragment file, the final fragment files cannot be merged normally, and the large file transmission fails. The data processing method provided by the embodiment provides effective supplement on the basis of the prior art, and through the steps, relevant file information is returned after the fragmented files are uploaded, if uploading failure occurs, a corresponding error type can be obtained from the file information, and an uploading request is reinitiated after a corresponding processing command is executed, so that the problem of file fragment uploading failure in a micro-service cluster is solved, and the effect that the file fragments can be successfully uploaded is realized by ensuring that the file fragments can be successfully uploaded.
In some embodiments, before uploading the pre-fragmented file to a corresponding storage terminal through a service terminal and returning file information of the uploaded fragmented file to a processing program, the method further includes the following steps:
in a processing program, based on a preset rule, carrying out fragmentation processing on an acquired file to be uploaded to obtain a fragmented file;
marking the fragment file to obtain fragment marks; and uploading the fragment file and the fragment mark to a service terminal.
Specifically, firstly, binary conversion is performed on the acquired file to be uploaded, and then based on a preset rule, the specific rule includes that segmentation is performed according to the file length or the number of files, and equal-length segmentation or number segmentation is performed on the file to be uploaded to obtain a fragment file.
Furthermore, when file fragmentation is performed, fragmentation files need to be marked, and the position of each fragmentation file in the whole fragmentation file queue can be known according to the context of the fragmentation files. The mark of the fragment file may be specifically marked in the name of the fragment file or in the header information of http, for example, a field isLast ═ true is included in the header information of http, which indicates that this is the last fragment file. And finally, uploading the fragment files with the fragment marks to each service terminal.
The file to be uploaded is pre-fragmented in the embodiment to obtain fragmented files, and the fragmented files are marked, so that when a large file is uploaded, the large file can be transmitted in the form of the fragmented file through a fragmentation method, and the fragmented file is uploaded in the form of a micro-service cluster in the embodiment to improve the uploading efficiency.
In some embodiments, the sending a corresponding processing command according to the file information and reinitiating the upload request for the fragmented file with the upload failure includes the following steps:
when the file information is that the fragment file is not successfully uploaded to the storage terminal, sending a corresponding processing command to the service terminal according to the error type obtained in the file information;
and executing corresponding command content according to the processing command, and re-triggering the uploading request of the fragmented file with uploading failure.
Specifically, when the file information of the fragmented file returned from the service terminal is unsuccessfully uploaded, the file information further includes a specific error type and error information of the fragmented file. The processing program includes predefined error types and corresponding processing commands, and the error types include but are not limited to: when the error type is network fault, the corresponding processing command can be router reconnection; when the error type is a fault of insufficient hard disk memory, the corresponding processing command may be to clear a cache (delete cache) or to add a mount (mount) of the hard disk. And sending a corresponding processing command to the service terminal according to the specific error type in the file information, executing corresponding command content by the service terminal with the uploading failure of the fragmented file, and re-sending an uploading request of the fragmented file.
Further, the sending of the corresponding processing command according to the file information and the re-initiating of the upload request to the fragmented file that failed to upload further includes the following steps:
in the service terminal, corresponding troubleshooting is carried out by executing the command content;
and after the uploading request is triggered again, updating the file information of the fragmented files according to the triggering result, and returning the updated file information to the processing program.
Specifically, the service terminal performs corresponding troubleshooting by executing the command content corresponding to the processing command. For example, when the processing command sent by the processing program received by the service terminal is to clear the cache, the fault is eliminated by executing the content of the command to clear the cache, and then the upload request is triggered again.
Updating the file information of the fragmented file according to the result of triggering the upload request again, which may specifically include performing successful recording, or performing failure, recording the reason of failure and tracking subsequent processing, and may also include optimizing processing of the upload failure condition.
By returning the information of the fragmented files which are failed to be uploaded to the processing program from the service terminal in the embodiment, and then sending corresponding processing commands to the service terminal by the processing program according to the error information in the information, the situation of the failure to be uploaded can be specifically cleared in the service terminal, and then the fragmented file uploading request is initiated again, so that all the fragmented files can be successfully uploaded to the storage terminal, and the effect of normally combining the fragmented files can be finally realized.
In some embodiments, the downloading all the fragment files from each storage terminal, combining the fragment files to obtain a complete file to be uploaded, and uploading the file to be uploaded to the storage terminal includes the following steps:
when the processing program receives the file information of the last fragment file, downloading all uploaded fragment files from the storage terminal through the service terminal;
and combining the downloaded fragment files to obtain a complete file to be uploaded, and uploading the file to be uploaded to a storage terminal for storage.
Specifically, the processing program distinguishes through the fragment mark on the fragment file, and when the header information in the fragment file has the field isLast ═ true or the name of the fragment file has Last, the processing program can be determined as the Last fragment file. When the file information of the last fragment file is received, it is indicated that all fragment files are successfully uploaded to the storage terminal. Because the service terminals in the service cluster only provide service, all uploaded fragment files need to be downloaded from the storage terminal, and the prime fragment files are recombined in any one service terminal according to information such as fragment marks on the fragment files to obtain a complete file to be uploaded, and the complete file to be uploaded is uploaded to any one storage terminal for storage. It should be noted that the storage terminals in the storage cluster synchronize the complete file to be uploaded.
Further, after the file is uploaded to the storage terminal, the name of the storage terminal and other file information are returned to the processing program for recording, such as the name, size, and number of fragments of the file to be uploaded.
After all the fragment files are uploaded, the service terminal downloads all the fragment files from the storage terminal for combination, so that all the fragment users can combine normally, and the complete file to be uploaded is recovered and uploaded to the storage terminal for storage.
The present embodiment is described and illustrated below by means of preferred embodiments.
Fig. 4 is a flowchart of a microservice cluster file fragment uploading method in the preferred embodiment, as shown in fig. 4, the method includes the following steps:
and step S410, based on a preset rule, carrying out fragmentation processing and marking processing on the acquired file to be uploaded to obtain a fragmented file with a mark.
Step S420, uploading the fragmented file to a corresponding storage terminal through the service terminal, and returning file information of the uploaded fragmented file to the processing program.
In step S430, in the processing procedure, the error type and the corresponding processing command are predefined.
Step S440, when the file information is that the fragmented file is not successfully uploaded to the storage terminal, the corresponding processing command is sent to the service terminal according to the error type obtained in the file information.
Step S450, corresponding troubleshooting is carried out by executing corresponding command content, and then the uploading request of the fragmented file is triggered again.
Step S460, updating the file information of the fragmented file according to the triggering result, and returning the updated file information to the processing program.
In step S470, when the processing program receives the file information of the last fragmented file, all the uploaded fragmented files are downloaded from the storage terminal through the service terminal.
And step S480, merging the downloaded fragment files to obtain a complete file to be uploaded, and uploading the file to be uploaded to a storage terminal for storage.
It should be noted that the steps illustrated in the above-described flow diagrams or in the flow diagrams of the figures may be performed in a computer system, such as a set of computer-executable instructions, and that, although a logical order is illustrated in the flow diagrams, in some cases, the steps illustrated or described may be performed in an order different than here. For example, it is contemplated that step S430 may be performed at any step prior to step S440, between step S410 and step S420, or prior to step S410.
There is also provided in this embodiment a computer device comprising a memory having a computer program stored therein and a processor arranged to run the computer program to perform the steps of any of the method embodiments described above.
Optionally, the computer device may further include a transmission device and an input/output device, wherein the transmission device is connected to the processor, and the input/output device is connected to the processor.
It should be noted that, for specific examples in this embodiment, reference may be made to the examples described in the foregoing embodiments and optional implementations, and details are not described again in this embodiment.
In addition, in combination with the micro service cluster file fragment uploading method provided in the foregoing embodiment, a storage medium may also be provided to implement the method in this embodiment. The storage medium having stored thereon a computer program; when executed by a processor, the computer program implements any one of the above-described micro service cluster file fragment uploading methods.
It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to be limiting. All other embodiments, which can be derived by a person skilled in the art from the examples provided herein without any inventive step, shall fall within the scope of protection of the present application.
It is obvious that the drawings are only examples or embodiments of the present application, and it is obvious to those skilled in the art that the present application can be applied to other similar cases according to the drawings without creative efforts. Moreover, it should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another.
The term "embodiment" is used herein to mean that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is to be expressly or implicitly understood by one of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the patent protection. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.
Claims (10)
1. A method for uploading micro-service cluster file fragments is characterized by comprising the following steps:
uploading the pre-fragmented fragment files to corresponding storage terminals through a service terminal, and returning the file information of the uploaded fragment files to a processing program;
sending a corresponding processing command according to the file information, and re-initiating an uploading request for the fragmented file which fails to be uploaded;
after all the fragment files are successfully uploaded, downloading all the fragment files from each storage terminal, combining the fragment files to obtain a complete file to be uploaded, uploading the file to be uploaded to the storage terminals, and simultaneously synchronizing the complete file to be uploaded between the storage terminals.
2. The method according to claim 1, wherein before the uploading the pre-fragmented file to the corresponding storage terminal through the service terminal and returning the file information of the uploaded fragmented file to the processing program, the method further comprises:
and in the processing program, based on a preset rule, carrying out fragmentation processing on the acquired file to be uploaded to obtain the fragmented file.
3. The method for uploading the micro-service cluster file fragments according to claim 2, wherein the obtaining the file to be uploaded by fragmenting the file based on a preset rule to obtain the fragmented file comprises:
marking the fragment file to obtain a fragment mark;
and uploading the fragment file and the fragment mark to the service terminal.
4. The method for uploading the fragments of the microservice cluster file according to claim 1, wherein the step of sending the corresponding processing command according to the file information and re-initiating the uploading request for the fragment file which fails to be uploaded comprises:
when the file information is that the fragmented file is not successfully uploaded to the storage terminal, sending the corresponding processing command to the service terminal according to the error type obtained from the file information;
and executing corresponding command content according to the processing command, and re-triggering the uploading request of the fragmented file with uploading failure.
5. The method for uploading fragmented files of micro-service cluster according to claim 4, wherein the executing the corresponding command content according to the processing command re-triggers the uploading request of the fragmented file with failed uploading, further comprising:
in the service terminal, corresponding fault elimination processing is carried out by executing the command content;
and after the uploading request is re-triggered, updating the file information of the fragmented file according to a triggering result, and returning the updated file information to the processing program.
6. The method for uploading the fragments of the microservice cluster file according to claim 1, wherein the downloading all the fragment files from each storage terminal, combining the fragment files to obtain a complete file to be uploaded, and uploading the file to be uploaded to the storage terminal comprises:
when the processing program receives the file information of the last fragment file, downloading all the uploaded fragment files from the storage terminal through the service terminal;
and combining the downloaded fragment files to obtain a complete file to be uploaded, and uploading the file to be uploaded to the storage terminal for storage.
7. A micro-service cluster file fragmentation uploading system is characterized by comprising: the system comprises a service cluster, a processing program module and a storage cluster;
the service cluster comprises a plurality of service terminals and is used for uploading the fragmented files subjected to pre-fragmentation to corresponding storage terminals and returning the file information of the uploaded fragmented files to the processing program module;
after all the fragment files are successfully uploaded, downloading all the fragment files from the storage cluster, combining the fragment files to obtain a complete file to be uploaded, and uploading the file to be uploaded to the storage cluster;
the processing program module sends a corresponding processing command according to the file information through the processing program and re-initiates an uploading request to the fragmented file with uploading failure;
the storage cluster comprises a plurality of storage terminals and is used for storing the fragment files and the files to be uploaded by the service cluster and synchronizing the complete files to be uploaded.
8. The micro-service cluster file fragmentation upload system of claim 7, wherein the service cluster is further configured to:
when the processing program receives the file information of the last fragment file, downloading all the uploaded fragment files from the storage cluster;
and combining the downloaded fragment files to obtain the complete file to be uploaded, and uploading the file to be uploaded to the storage cluster for storage.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the computer program to perform the micro-service cluster file fragment uploading method according to any one of claims 1 to 6.
10. A computer-readable storage medium, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of the micro-service cluster file fragment uploading method according to any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111557192.2A CN114401260B (en) | 2021-12-18 | 2021-12-18 | Micro-service cluster file fragment uploading method, system, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111557192.2A CN114401260B (en) | 2021-12-18 | 2021-12-18 | Micro-service cluster file fragment uploading method, system, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114401260A true CN114401260A (en) | 2022-04-26 |
| CN114401260B CN114401260B (en) | 2024-05-28 |
Family
ID=81226598
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111557192.2A Active CN114401260B (en) | 2021-12-18 | 2021-12-18 | Micro-service cluster file fragment uploading method, system, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114401260B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101227460A (en) * | 2007-01-19 | 2008-07-23 | 秦晨 | Method for uploading and downloading distributed document and apparatus and system thereof |
| CN102710630A (en) * | 2012-05-29 | 2012-10-03 | 新华网股份有限公司 | Cloud upload and download method and system of multi-threaded slice |
| CN106993054A (en) * | 2017-05-05 | 2017-07-28 | 腾讯科技（深圳）有限公司 | Document distribution method, node and system |
| CN109714330A (en) * | 2018-12-24 | 2019-05-03 | 武汉烽火众智数字技术有限责任公司 | A kind of breaker point continuous transmission method and system of across a network |
| CN111404990A (en) * | 2020-02-14 | 2020-07-10 | Oppo（重庆）智能科技有限公司 | File transmission method, device, client and storage medium |
| WO2021184992A1 (en) * | 2020-03-16 | 2021-09-23 | 华为技术有限公司 | Mirror image file uploading method, related device and computer storage medium |
-
2021
- 2021-12-18 CN CN202111557192.2A patent/CN114401260B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101227460A (en) * | 2007-01-19 | 2008-07-23 | 秦晨 | Method for uploading and downloading distributed document and apparatus and system thereof |
| CN102710630A (en) * | 2012-05-29 | 2012-10-03 | 新华网股份有限公司 | Cloud upload and download method and system of multi-threaded slice |
| CN106993054A (en) * | 2017-05-05 | 2017-07-28 | 腾讯科技（深圳）有限公司 | Document distribution method, node and system |
| CN109714330A (en) * | 2018-12-24 | 2019-05-03 | 武汉烽火众智数字技术有限责任公司 | A kind of breaker point continuous transmission method and system of across a network |
| CN111404990A (en) * | 2020-02-14 | 2020-07-10 | Oppo（重庆）智能科技有限公司 | File transmission method, device, client and storage medium |
| WO2021184992A1 (en) * | 2020-03-16 | 2021-09-23 | 华为技术有限公司 | Mirror image file uploading method, related device and computer storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114401260B (en) | 2024-05-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9906630B2 (en) | Processing data packets in performance enhancing proxy (PEP) environment | |
| CN107864228B (en) | Connection establishment method and system in content distribution network | |
| CN103647869B (en) | A kind of matching method of terminal, terminal and system | |
| EP3588914A1 (en) | Data storage method, encoding device and decoding device | |
| CN106612284B (en) | Streaming data transmission method and device | |
| CN106576108B (en) | Communication method, equipment and system in communication system | |
| CN111385268B (en) | Data packet header compression confirmation method and communication equipment | |
| CN111111214B (en) | Game archiving processing method, device and system | |
| TWI530147B (en) | Apparatus and method for level-based self-adjusting peer-to-peer media streaming | |
| CN104954101B (en) | Multiple terminals method of data synchronization based on synchronous ACK | |
| CN112612769A (en) | File processing method, device and storage medium | |
| CN107204924B (en) | Link discovery method and device | |
| WO2017220021A1 (en) | Short message processing method and apparatus | |
| CN108810936A (en) | A kind of configuration method and device of LoRa terminals | |
| CN103873443A (en) | Information processing method, local proxy server and network proxy server | |
| WO2023125056A1 (en) | Network data control method and apparatus, and storage medium and electronic device | |
| US11973824B2 (en) | Method for data transmission of audio and video in end-to-end system | |
| CN114401260A (en) | Method, system, equipment and storage medium for uploading micro-service cluster file fragments | |
| CN109996349B (en) | Session recovery method and device | |
| CN115865895A (en) | File uploading method and device and storage medium | |
| CN102082726B (en) | Data processing method, device and transmission system | |
| CN113612811B (en) | Method, system, equipment and medium for client mounting in multiple channels | |
| CN104754760A (en) | Packet service reconstruction method and terminal | |
| CN106714294B (en) | Multi-channel TCP/IP header compression synchronization method applied to smart power grid | |
| CN106209666B (en) | Link multiplexing method and system based on load balancer |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |