CN111343220A - Transponder, distributed file transmission method, system, medium and electronic equipment - Google Patents
Transponder, distributed file transmission method, system, medium and electronic equipment Download PDFInfo
- Publication number
- CN111343220A CN111343220A CN201811554177.0A CN201811554177A CN111343220A CN 111343220 A CN111343220 A CN 111343220A CN 201811554177 A CN201811554177 A CN 201811554177A CN 111343220 A CN111343220 A CN 111343220A
- Authority
- CN
- China
- Prior art keywords
- data center
- downloading
- download
- file
- value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 41
- 230000005540 biological transmission Effects 0.000 title claims abstract description 32
- 238000012546 transfer Methods 0.000 claims description 11
- 238000004590 computer program Methods 0.000 claims description 9
- 230000004931 aggregating effect Effects 0.000 abstract description 7
- 238000004891 communication Methods 0.000 abstract description 6
- 238000010586 diagram Methods 0.000 description 12
- 230000006870 function Effects 0.000 description 8
- 230000008569 process Effects 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 230000009471 action Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000010348 incorporation Methods 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1004—Server selection for load balancing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The embodiment of the invention provides a repeater, a distributed file transmission method, a distributed file transmission system, a computer readable medium and electronic equipment, and relates to the technical field of wireless communication. The distributed file transmission method comprises the following steps: receiving a downloading request sent by a business machine of a region where the second data center is located; judging whether a download file corresponding to the download request exists in the second data center; and when the second data center does not have the download file corresponding to the download request, downloading the file from the first data center according to the aggregated download request and storing the file in the second data center. According to the technical scheme of the embodiment of the invention, the cross-machine-room transmission of the file is reduced by aggregating the downloading requests, the rapid synchronization of the data is realized, and the service deployment speed is increased.
Description
Technical Field
The invention relates to the technical field of wireless communication, in particular to a repeater, a distributed file transmission method, a distributed file transmission system, a computer readable medium and electronic equipment.
Background
With the continuous enlargement of the scale of internet companies, some internet companies start to implement a remote multi-live internet data center machine room deployment scheme, that is, the internet companies build the internet data center machine rooms in multiple places.
For example, a service system of an internet company is deployed in a machine room a of a city a, and starts to deploy a machine room B in the city B in order to increase the access speed of telecommunication users in an area where the city B is located and reduce the machine room cost. In the deployment process of a machine room B, a network A and a network B establish an AB special line to realize connection, and a large number of service machines B download data from a data center A through the AB special line at the same time, so that the dependence on special line bandwidth is particularly large, the cost is high, the downloading speed is very slow, the service deployment and updating are slow, and a large amount of working hours are consumed in operation and maintenance online. Under the severe conditions of high cost, low bandwidth and large network delay of the AB special line, the synchronization of data from the A machine room is necessary because a service system is huge and has strong dependence on various data.
In the big data era, it is not practical to completely synchronize data in the B room by one copy or double-write data in both places. Therefore, how to reduce the transmission of data across the computer room is a technical problem to be solved at present.
It is to be noted that the information disclosed in the above background section is only for enhancement of understanding of the background of the present invention and therefore may include information that does not constitute prior art known to a person of ordinary skill in the art.
Disclosure of Invention
The embodiment of the invention aims to provide a repeater, a distributed file transmission method, a distributed file transmission system, a distributed file transmission medium and electronic equipment, so that cross-computer-room transmission of data is reduced to at least a certain extent.
Additional features and advantages of the invention will be set forth in the detailed description which follows, or may be learned by practice of the invention.
According to a first aspect of the embodiments of the present invention, a distributed file transmission method is provided, in which a first data center and a second data center are provided, and the second data center downloads a file from the first data center, and the method includes: receiving a downloading request sent by a business machine of a region where the second data center is located; judging whether a download file corresponding to the download request exists in the second data center; and when the second data center does not have the download file corresponding to the download request, downloading the file from the first data center according to the aggregated download request and storing the file in the second data center.
In the above scheme, the method further comprises: and when the second data center has the download file corresponding to the download request, downloading the download file from the second data center.
In the foregoing solution, when the second data center does not have the download file corresponding to the download request, the method further includes: and sending notification information for notifying the service machine of later downloading to the service machine.
In the foregoing scheme, the downloading the file from the first data center and storing the file in the second data center according to the aggregated download request includes: generating a key value and a value according to the downloading request, wherein the key value comprises a url abstract of the downloading request, and the value comprises url data of the downloading request; storing the key value and value to a key-value database; and downloading the file from the first data center at set first time intervals according to value values corresponding to different key values in the key-value database.
In the foregoing scheme, the downloading the file from the first data center and storing the file in the second data center according to the aggregated download request includes: generating a key value and a value according to the downloading request, wherein the key value comprises a url abstract of the downloading request, and the value comprises url data of the downloading request; and downloading the file from the first data center according to the key value and the value, and saving the downloaded file to the second data center.
According to a second aspect of the embodiments of the present invention, there is provided a repeater that is provided with a first data center and a second data center, the repeater being located in the second data center, the repeater including: the receiving unit is used for receiving a downloading request sent by a business machine of a region where the second data center is located; the judging unit is used for judging whether the second data center has a download file corresponding to the download request; and the downloading unit is used for downloading the file from the first data center according to the aggregated downloading request and storing the file to the second data center when the second data center does not have the downloading file corresponding to the downloading request.
In the foregoing scheme, the downloading unit is further configured to download the download file from the second data center when the download file corresponding to the download request exists in the second data center.
In the foregoing solution, the repeater further includes a sending unit, configured to send, to the service machine, notification information notifying that the service machine downloads later when the second data center does not have a download file corresponding to the download request.
In the foregoing solution, the downloading unit includes: the generation subunit is used for generating a key value and a value according to the download request, wherein the key value comprises the url abstract of the download request, and the value comprises the url data of the download request; the storage subunit is used for storing the key value and the value into a key-value database; and the downloading subunit is used for downloading the file from the first data center at set first time intervals according to the value values corresponding to different key values in the key-value database.
According to a third aspect of the embodiments of the present invention, a distributed file transmission system is provided, where the system includes a first data center and a second data center, the system includes a service machine located in a region where the second data center is located, a load balancer, and a plurality of repeaters according to the second aspect of the embodiments, the plurality of repeaters are located in the second data center, and the service machine is configured to receive a user instruction and generate a download request according to the user instruction; and the load balancer is used for distributing the corresponding repeater to the user according to the link condition and the load condition of the repeater.
According to a fourth aspect of embodiments of the present invention, there is provided a computer readable medium, on which a computer program is stored, which when executed by a processor, implements the distributed file transfer method as described in the first aspect of the embodiments above.
According to a fifth aspect of embodiments of the present invention, there is provided an electronic apparatus, including: one or more processors; storage means for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement the distributed file transfer method as described in the first aspect of the embodiments above.
The technical scheme provided by the embodiment of the invention has the following beneficial effects:
in the technical solutions provided by some embodiments of the present invention, cross-machine-room transmission of files is reduced by aggregating download requests, so that fast data synchronization is realized, and the service deployment speed is increased.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention. It is obvious that the drawings in the following description are only some embodiments of the invention, and that for a person skilled in the art, other drawings can be derived from them without inventive effort. In the drawings:
FIG. 1 schematically illustrates a flow diagram of a method of distributed file transfer according to an embodiment of the invention;
FIG. 2 is a schematic diagram illustrating a file download process in the prior art;
FIG. 3 schematically shows a schematic diagram of a file download process according to an embodiment of the invention;
FIG. 4 schematically shows a block diagram of a repeater according to an embodiment of the present invention;
FIG. 5 schematically illustrates a block diagram of another repeater in accordance with an embodiment of the present invention;
FIG. 6 illustrates a schematic structural diagram of a computer system suitable for use with the electronic device to implement an embodiment of the invention.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art.
Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to provide a thorough understanding of embodiments of the invention. One skilled in the relevant art will recognize, however, that the invention may be practiced without one or more of the specific details, or with other methods, components, devices, steps, and so forth. In other instances, well-known methods, devices, implementations or operations have not been shown or described in detail to avoid obscuring aspects of the invention.
The block diagrams shown in the figures are functional entities only and do not necessarily correspond to physically separate entities. I.e. these functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor means and/or microcontroller means.
The flow charts shown in the drawings are merely illustrative and do not necessarily include all of the contents and operations/steps, nor do they necessarily have to be performed in the order described. For example, some operations/steps may be decomposed, and some operations/steps may be combined or partially combined, so that the actual execution sequence may be changed according to the actual situation.
Fig. 1 schematically illustrates a distributed file transfer method according to an exemplary embodiment of the present disclosure. According to an exemplary embodiment of the present disclosure, a first data center and a second data center are provided, the second data center downloading a file from the first data center. Referring to fig. 1, the distributed file transfer method may include the steps of:
step S102, receiving a downloading request sent by a business machine of a region where the second data center is located.
And step S104, judging whether the second data center has a download file corresponding to the download request.
And step S106, when the second data center does not have the download file corresponding to the download request, downloading the file from the first data center according to the aggregated download request and storing the file in the second data center.
As shown in fig. 2, in the related art, the B service system 240 synchronizes data directly from the a data center 210, the a network and the B network establish an AB dedicated line to implement connection, and the AB dedicated line has a small bandwidth and a large network delay, so that the downloading speed is also very slow, which results in slow service deployment and update. Node 211 in FIG. 2 refers to a node of a distributed file system cluster; IP 231 and IP241 refer to business physical machines or containers, simply business machines.
In an exemplary embodiment of the disclosure, the B-room is provided with a B-data center built based on technologies such as MFS or Hdfs distributed file system. A data center built based on the distributed file system cluster can be simultaneously used as a data buffer pool between two places, so that cross-machine room transmission is reduced.
The first data center and the second data center may be located in different regions. As shown in fig. 3, the first data center and the second data center which are set up in different regions are the a data center 210 and the B data center 340. The service machines such as IP in the service system B360 access the data center of the machine room B340 through the load balancer 350 by using a load balancing technology. In addition, a repeater 341 is deployed on each data node of the B data center 340. Here, the repeater 341 has a function of processing, rewriting, and forwarding the download request.
In step S102, after the repeater 341 receives the download request, the file is downloaded from the first data center according to the aggregated download request by the method of aggregating the download requests, so as to avoid multiple times of repeated downloads of the same download file from the first data center according to repeated download requests.
As shown in fig. 3, in the exemplary embodiment of the present disclosure, based on the intelligent domain name system resolution technology, the B service system 360 forwards the download request through the load balancer 350 located in B, and the a service system 230 forwards the download request through the load balancer 220 located in a, so as to implement physical isolation of download request traffic between B and a. Here, IP 361 is a service machine.
After the repeater 340 of the node of the B data center 340 receives the download request link, analyzes the download request link, and then executes step S104 to determine whether the download file corresponding to the download request exists in the B data center. After step S104, if there is a download file corresponding to the download request in the second data center, the download file is downloaded from the second data center. That is, if the data center B has the download file corresponding to the download request, the data center B returns the download file to start downloading. And if the download file corresponding to the download request does not exist in the data center B, executing the step S106. The scheme can reduce the transmission of the files across the computer room.
After step S106 is executed, notification information notifying the service machine of later downloading may also be sent to the service machine. After receiving the return value, i.e., notification message, from the repeater 341 to retry downloading later, the server 361 of the B service system attempts to download again at an interval.
In step S106, aggregation of the download requests is performed using the redis database. Specifically, a key value and a value are generated according to the download request and stored to a key-value database.
Here, the key value includes a url digest of the download request, and the value includes url data of the download request.
After the key value and the value are stored in a key-value database, url data corresponding to the value are acquired according to the uniqueness of the key value, and a file is downloaded from a first number center. Here, the file may be downloaded from the first data center according to value values corresponding to different key values in the key-value database at every set first time. After the key value and the value are stored in the key-value database, the file can be downloaded from the first data center according to the key value and the value, and the downloaded file can be saved in the second data center.
When the key value and the value are generated according to the download request, the url abstract of the download request, namely the key value, is generated according to the download request by using the MD5 Hash. MD5Hash is a document MD5 verification tool. The value of the download request includes a string of a request link of the download request.
Storing the key value and the value into the redis database requires the incorporation of a set expiration time. The redis database is a memory type key-value database, the key value is a unique value in the redis database, and the same key value cannot exist, so that the downloading uniform resource locators are aggregated, a large number of repeated requests are aggregated, and the cross-machine-room transmission of files is reduced.
Thereafter, in step S106, the repeater reads the download request link in the redis database, and transmits a download request to the a data center.
In the distributed file transmission method provided by some embodiments of the present invention, the cross-machine-room transmission of files is reduced by aggregating download requests, so as to implement fast data synchronization and improve the service deployment speed.
The following describes an embodiment of the apparatus of the present invention, which can be used to execute the above-mentioned distributed file transmission method of the present invention. According to an exemplary embodiment of the present disclosure, a first data center and a second data center are provided, and a repeater is located in the second data center, and specifically, referring to fig. 4, a repeater 400 provided by an embodiment of the present invention includes:
a receiving unit 402, configured to receive a download request sent by a service machine in a region where the second data center is located.
The determining unit 404 is configured to determine whether the second data center has a download file corresponding to the download request.
And the downloading unit 406 is configured to, when there is no download file corresponding to the download request in the second data center, download the file from the first data center according to the aggregated download request and store the file in the second data center.
The first data center and the second data center may be located in different regions. By the method for aggregating the download requests, the files are downloaded from the first data center according to the aggregated download requests, repeated downloading of the same download files from the first data center according to repeated download requests is avoided, and compared with the prior art, cross-machine room transmission of the files can be reduced, so that data which strongly depends on services in remote double-activity projects can be quickly synchronized, the bandwidth used by a special line is reduced, the cost is reduced, and the service deployment and updating speed is increased.
In the exemplary embodiment of the present disclosure, after the receiving unit 402 receives the download request link, the determining unit 404 determines whether the download file corresponding to the download request exists in the B data center according to the download request link, and the downloading unit 406 downloads the download file from the second data center when the download file corresponding to the download request exists in the second data center.
When the data center B does not have the download file corresponding to the download request, in order to reduce transmission of the file across the computer room, the download unit 406 aggregates the download request and downloads the download file from the second data center.
The download unit 406 includes a generation subunit, a storage subunit, and a download subunit. And the generating subunit is used for generating a key value and a value according to the downloading request. And the storage subunit is used for storing the key value and the value into a key-value database.
Here, the key value includes a url digest of the download request, and the value includes url data of the download request
After the key value and the value are stored in the key-value database, the downloading subunit acquires url data corresponding to the value according to the uniqueness of the key value and downloads the file from the first data center. Here, the download subunit may download the file from the first data center at every set first time according to value values corresponding to different key values in the key-value database. The downloading subunit may also download the file from the first data center according to the key value and the value after the key value and the value are stored in the key-value database, and store the downloaded file in the second data center.
When the generation subunit generates the key value and the value according to the download request, the url digest of the download request, namely the key value, is generated according to the download request by using the MD5 Hash. MD5Hash is a document MD5 verification tool. The value of the download request includes a string of a request link of the download request.
When the storage subunit stores the key value and the value into the redis database, the set expiration time needs to be combined. The key value and the value are stored in the database, so that the downloaded uniform resource locators are aggregated, a large number of repeated requests are aggregated, and the cross-computer-room transmission of files is reduced.
According to an exemplary embodiment of the present disclosure, referring to fig. 5, compared to the repeater 400, the repeater 500 includes not only the receiving unit 402, the judging unit 404, and the downloading unit 406, but also the transmitting unit 508.
The sending unit 508 sends notification information notifying that the server downloads later to the server when the second data center does not have the download file corresponding to the download request. After receiving the return value, i.e., notification message, from the repeater 341 to retry downloading later, the server 361 of the B service system attempts to download again at an interval.
Other functions of the repeater 500 are the same as those of the repeater 400 and will not be described again.
For details that are not disclosed in the embodiments of the apparatus of the present invention, reference is made to the above-described embodiments of the distributed file transmission method of the present invention for the functional modules of the repeater of the exemplary embodiment of the present invention correspond to the steps of the above-described exemplary embodiment of the distributed file transmission method.
In the repeater provided by the embodiment of the invention, the cross-machine-room transmission of files is reduced by aggregating the downloading requests, the rapid synchronization of data is realized, and the service deployment speed is increased.
An embodiment of the present invention further provides a distributed file transfer system, and in this exemplary embodiment, a first data center and a second data center are provided. The distributed file transmission system comprises a service machine, a load balancer and a plurality of repeaters in the embodiment of the device, wherein the service machine is positioned in a region where a second data center is positioned, and the repeaters are positioned in the second data center; and the load balancer is used for distributing the corresponding repeater to the user according to the link condition and the load condition of the repeater.
Specifically, as shown in fig. 3, in an exemplary embodiment of the present disclosure, a distributed file transfer system may include a service machine in a B-service system 360, a load balancer 350, and a plurality of repeaters 341.
Due to the existence of the load balancer, the pressure of the second data center can be greatly reduced, so that the downloading request sent by the service machine is processed in time, and the resource availability is improved.
The distributed file transmission system provided in the embodiment of the present invention includes the repeater 341 in the above technical solution, and has the same technical features as the repeater 341, so that the cross-machine-room transmission of files can be reduced by aggregating download requests, fast data synchronization is realized, and the service deployment speed is increased.
Referring now to FIG. 6, shown is a block diagram of a computer system 600 suitable for use with the electronic device implementing an embodiment of the present invention. The computer system 600 of the electronic device shown in fig. 6 is only an example, and should not bring any limitation to the function and the scope of the use of the embodiments of the present invention.
As shown in fig. 6, the computer system 600 includes a Central Processing Unit (CPU)601 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)602 or a program loaded from a storage section 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data necessary for system operation are also stored. The CPU601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
The following components are connected to the I/O interface 605: an input portion 606 including a keyboard, a mouse, and the like; an output portion 607 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 608 including a hard disk and the like; and a communication section 609 including a network interface card such as a LAN card, a modem, or the like. The communication section 609 performs communication processing via a network such as the internet. The driver 610 is also connected to the I/O interface 605 as needed. A removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 610 as necessary, so that a computer program read out therefrom is mounted in the storage section 608 as necessary.
In particular, according to an embodiment of the present invention, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the invention include a computer program product comprising a computer program embodied on a computer-readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 609, and/or installed from the removable medium 611. The above-described functions defined in the system of the present application are executed when the computer program is executed by the Central Processing Unit (CPU) 601.
It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in the embodiments of the present invention may be implemented by software, or may be implemented by hardware, and the described units may also be disposed in a processor. Wherein the names of the elements do not in some way constitute a limitation on the elements themselves.
As another aspect, the present application also provides a computer-readable medium, which may be contained in the electronic device described in the above embodiments; or may exist separately without being assembled into the electronic device. The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to implement the distributed file transfer method as described in the above embodiments.
For example, the electronic device may implement the following as shown in fig. 1: step S102, receiving a downloading request sent by a business machine of a region where the second data center is located; step S104, judging whether a download file corresponding to the download request exists in the second data center; and step S106, when the second data center does not have the download file corresponding to the download request, downloading the file from the first data center according to the aggregated download request and storing the file in the second data center.
It should be noted that although in the above detailed description several modules or units of the device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit, according to embodiments of the invention. Conversely, the features and functions of one module or unit described above may be further divided into embodiments by a plurality of modules or units.
Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the embodiment of the present invention.
Other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the invention and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the invention being indicated by the following claims.
It will be understood that the invention is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the invention is limited only by the appended claims.
Claims (12)
1. A distributed file transmission method is characterized in that a first data center and a second data center are arranged, and the second data center downloads files from the first data center, and the method comprises the following steps:
receiving a downloading request sent by a business machine of a region where the second data center is located;
judging whether a download file corresponding to the download request exists in the second data center;
and when the second data center does not have the download file corresponding to the download request, downloading the file from the first data center according to the aggregated download request and storing the file in the second data center.
2. The method of claim 1, further comprising:
and when the second data center has the download file corresponding to the download request, downloading the download file from the second data center.
3. The method according to claim 2, wherein when the second data center does not have a download file corresponding to the download request, the method further comprises:
and sending notification information for notifying the service machine of later downloading to the service machine.
4. The method of claim 1, wherein the downloading and storing the file from the first data center to the second data center according to the aggregated download request comprises:
generating a key value and a value according to the downloading request, wherein the key value comprises a url abstract of the downloading request, and the value comprises url data of the downloading request;
storing the key value and value to a key-value database;
and downloading the file from the first data center at set first time intervals according to value values corresponding to different key values in the key-value database.
5. The method of claim 1, wherein the downloading and storing the file from the first data center to the second data center according to the aggregated download request comprises:
generating a key value and a value according to the downloading request, wherein the key value comprises a url abstract of the downloading request, and the value comprises url data of the downloading request;
and downloading the file from the first data center according to the key value and the value, and saving the downloaded file to the second data center.
6. A repeater, wherein a first data center and a second data center are provided, and wherein the repeater is located at the second data center, the repeater comprising:
the receiving unit is used for receiving a downloading request sent by a business machine of a region where the second data center is located;
the judging unit is used for judging whether the second data center has a download file corresponding to the download request;
and the downloading unit is used for downloading the file from the first data center according to the aggregated downloading request and storing the file to the second data center when the second data center does not have the downloading file corresponding to the downloading request.
7. The repeater according to claim 6, wherein the downloading unit is further configured to download the download file from the second data center when the download file corresponding to the download request exists in the second data center.
8. The repeater according to claim 7, further comprising a sending unit, configured to send notification information notifying the service machine of later downloading when the second data center does not have the download file corresponding to the download request.
9. The repeater according to claim 6, wherein the download unit comprises:
the generation subunit is used for generating a key value and a value according to the download request, wherein the key value comprises the url abstract of the download request, and the value comprises the url data of the download request;
the storage subunit is used for storing the key value and the value into a key-value database;
and the downloading subunit is used for downloading the file from the first data center at set first time intervals according to the value values corresponding to different key values in the key-value database.
10. A distributed file transfer system, characterized in that a first data center and a second data center are provided, the system comprising a service machine located in the area of the second data center, a load balancer and a plurality of repeaters according to any one of claims 6 to 9, the plurality of repeaters being located in the second data center, wherein,
the service machine is used for receiving a user instruction and generating a downloading request according to the user instruction;
and the load balancer is used for distributing the corresponding repeater to the user according to the link condition and the load condition of the repeater.
11. A computer-readable medium, on which a computer program is stored, which program, when being executed by a processor, is adapted to carry out the distributed file transfer method according to any one of claims 1 to 5.
12. An electronic device, comprising:
one or more processors;
storage means for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement the distributed file transfer method of any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811554177.0A CN111343220B (en) | 2018-12-18 | 2018-12-18 | Repeater, distributed file transmission method, distributed file transmission system, medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811554177.0A CN111343220B (en) | 2018-12-18 | 2018-12-18 | Repeater, distributed file transmission method, distributed file transmission system, medium and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111343220A true CN111343220A (en) | 2020-06-26 |
| CN111343220B CN111343220B (en) | 2024-02-09 |
Family
ID=71185110
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811554177.0A Active CN111343220B (en) | 2018-12-18 | 2018-12-18 | Repeater, distributed file transmission method, distributed file transmission system, medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111343220B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114567646A (en) * | 2022-03-08 | 2022-05-31 | 京东科技信息技术有限公司 | Data processing method, data processing system, electronic device, and storage medium |
| CN115412546A (en) * | 2022-08-26 | 2022-11-29 | 特赞（上海）信息科技有限公司 | File batch downloading method and device |
| CN117061503A (en) * | 2023-10-11 | 2023-11-14 | 腾讯科技（深圳）有限公司 | Image file management method, device, computer readable medium and electronic equipment |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101662503A (en) * | 2009-09-14 | 2010-03-03 | 金蝶软件（中国）有限公司 | Information transmission method, proxy server and service system in network |
| CN103218455A (en) * | 2013-05-07 | 2013-07-24 | 中国人民解放军国防科学技术大学 | Method of high-speed concurrent processing of user requests of Key-Value database |
| CN103747097A (en) * | 2014-01-22 | 2014-04-23 | 电子科技大学 | Mobile terminal HTTP (Hyper Text Transport Protocol) request aggregation compression system and method |
| CN105743730A (en) * | 2014-12-12 | 2016-07-06 | 阿里巴巴集团控股有限公司 | Method and system used for providing real-time monitoring for webpage service of mobile terminal |
| CN107026897A (en) * | 2017-02-20 | 2017-08-08 | 阿里巴巴集团控股有限公司 | A kind of methods, devices and systems of data processing |
| WO2018038741A1 (en) * | 2016-08-26 | 2018-03-01 | Nokia Technologies Oy | Data download via group collaboration |
| CN108804242A (en) * | 2018-05-23 | 2018-11-13 | 武汉斗鱼网络科技有限公司 | A kind of data counts De-weight method, system, server and storage medium |
-
2018
- 2018-12-18 CN CN201811554177.0A patent/CN111343220B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101662503A (en) * | 2009-09-14 | 2010-03-03 | 金蝶软件（中国）有限公司 | Information transmission method, proxy server and service system in network |
| CN103218455A (en) * | 2013-05-07 | 2013-07-24 | 中国人民解放军国防科学技术大学 | Method of high-speed concurrent processing of user requests of Key-Value database |
| CN103747097A (en) * | 2014-01-22 | 2014-04-23 | 电子科技大学 | Mobile terminal HTTP (Hyper Text Transport Protocol) request aggregation compression system and method |
| CN105743730A (en) * | 2014-12-12 | 2016-07-06 | 阿里巴巴集团控股有限公司 | Method and system used for providing real-time monitoring for webpage service of mobile terminal |
| WO2018038741A1 (en) * | 2016-08-26 | 2018-03-01 | Nokia Technologies Oy | Data download via group collaboration |
| CN107026897A (en) * | 2017-02-20 | 2017-08-08 | 阿里巴巴集团控股有限公司 | A kind of methods, devices and systems of data processing |
| CN108804242A (en) * | 2018-05-23 | 2018-11-13 | 武汉斗鱼网络科技有限公司 | A kind of data counts De-weight method, system, server and storage medium |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114567646A (en) * | 2022-03-08 | 2022-05-31 | 京东科技信息技术有限公司 | Data processing method, data processing system, electronic device, and storage medium |
| CN115412546A (en) * | 2022-08-26 | 2022-11-29 | 特赞（上海）信息科技有限公司 | File batch downloading method and device |
| CN117061503A (en) * | 2023-10-11 | 2023-11-14 | 腾讯科技（深圳）有限公司 | Image file management method, device, computer readable medium and electronic equipment |
| CN117061503B (en) * | 2023-10-11 | 2024-01-26 | 腾讯科技（深圳）有限公司 | Image file management method, device, computer readable medium and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111343220B (en) | 2024-02-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10951395B2 (en) | Data fetching in data exchange networks | |
| CN113657900B (en) | Cross-chain transaction verification method and system and cross-chain transaction system | |
| CN109447641B (en) | Method and apparatus for transmitting blockchain data to blockchain browser | |
| CN109032796B (en) | Data processing method and device | |
| CN109104368B (en) | Connection request method, device, server and computer readable storage medium | |
| CN109614439B (en) | Data synchronization method, device, electronic equipment and storage medium | |
| CN110572422B (en) | Data downloading method, device, equipment and medium | |
| CN112748962B (en) | Application loading method, device, electronic equipment and computer readable medium | |
| CN113179324B (en) | Block chain link point and service processing method executed by block chain link point | |
| CN111343220B (en) | Repeater, distributed file transmission method, distributed file transmission system, medium and electronic equipment | |
| CN110727738B (en) | Global routing system based on data fragmentation, electronic equipment and storage medium | |
| CN116303608A (en) | Data processing method and device for application service | |
| CN110083341A (en) | A kind of front end development platform, front end development approach and page display method | |
| CN114760304B (en) | Processing method, processing system and computing gateway of computing information | |
| CN110798495B (en) | Method and server for end-to-end message push in cluster architecture mode | |
| CN112084254A (en) | Data synchronization method and system | |
| CN113778499B (en) | Method, apparatus, device and computer readable medium for publishing services | |
| CN111382953A (en) | Dynamic process generation method and device | |
| CN110798358A (en) | Distributed service identification method and device, computer readable medium and electronic equipment | |
| CN109284177B (en) | Data updating method and device | |
| CN110661857B (en) | Data synchronization method and device | |
| CN112187916B (en) | Cross-system data synchronization method and device | |
| CN114528140A (en) | Method and device for service degradation | |
| CN111984616A (en) | Method, device and system for updating shared file | |
| CN109547552B (en) | API request processing method and device, storage medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |