CN111046008A - System and method for synchronously distributing files in real time - Google Patents
System and method for synchronously distributing files in real time Download PDFInfo
- Publication number
- CN111046008A CN111046008A CN201911233574.2A CN201911233574A CN111046008A CN 111046008 A CN111046008 A CN 111046008A CN 201911233574 A CN201911233574 A CN 201911233574A CN 111046008 A CN111046008 A CN 111046008A
- Authority
- CN
- China
- Prior art keywords
- data
- file
- node
- connection
- data packet
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/178—Techniques for file synchronisation in file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a system and a method for synchronously distributing files in real time, wherein the method comprises the following steps: step S1, selecting configuration source file and catalog, setting categories according to the selected file and catalog, preparing data sending channels of corresponding categories, carrying out authentication connection between superior and inferior nodes, and classifying the inferior node connection into corresponding category channels according to connection request; step S2, the superior node monitors the directory and the file in real time, and when the file is updated, the data of the file change block is obtained through the mirror image process of the superior node and the subordinate node; step S3, grouping the file change block data into blocks, and adding each data group into each connection under the corresponding category channel after high-efficiency compression processing; step S4, traversing the connection under the channel, and sending the processed compressed data packet to the correspondingly connected lower node; and step S5, after receiving the compressed data packet, the lower node decompresses the data packet, judges the data type according to the data packet identification, and performs synchronous receiving processing.
Description
Technical Field
The invention relates to the technical field of computer data backup, in particular to a system and a method for synchronously distributing files in real time.
Background
With the rapid development of information technology, the information construction requirements of various industries are gradually improved, and the real-time performance and performance requirements of the data analysis and transmission field are higher and higher.
At present, a plurality of scenes with file distribution requirements are provided for files, such as market files of a fund company of a stock company, which need to be synchronized from a headquarters to a plurality of lower-level business departments for use, and meteorological data files need to be distributed by a meteorological unit, and the files in the scenes are changed in real time and are required to be synchronized to the lower level every time the files are changed.
One of the current common backup system modes is to use a point-to-point and one-to-one backup synchronization method, which has extremely low synchronization efficiency, and the scene of one-to-many data file synchronization needs to have higher requirements on node distribution and node cascade; the other way is provided for other machines by way of file system network sharing, but there are multiple problems: first, when the access number is large, the network bandwidth requirement is high. Secondly, the network security is not easy to manage. Thirdly, file authority read-write easily conflicts. And fourthly, unified monitoring and management cannot be realized.
Disclosure of Invention
In order to overcome the defects of the prior art, the invention aims to provide a file real-time synchronous distribution system and method to realize the rapid synchronous distribution of market files to a plurality of subordinate nodes, and the system and method have the advantages of small data transmission amount, small occupied bandwidth, sufficiently low forwarding delay, efficient transmission process, real-time effectiveness, stability and reliability after data are forwarded to subordinate nodes, and the like.
In order to achieve the above object, the present invention provides a real-time synchronous file distribution system, which includes a plurality of levels, each level includes a plurality of nodes, each node includes a sending module and a receiving module, each level of nodes includes a main node and a standby node, and the main node is automatically switched to the standby node when the line connection of the main node is interrupted; the method comprises the steps that an upper node at the upper level selects a configuration source file and a directory, a plurality of categories are set according to the selected file and the directory, data sending channels of the corresponding categories are prepared, lower-level connections are classified into the channels of the corresponding categories according to connection requests, the directory and the source file are scanned in real time, file change data are obtained through the mirror image process of the upper node and the lower node, the obtained file change data are subjected to block packaging and compression processing, added to all connections under the data sending channels, the connections under the channels are traversed, and the data are sent to the lower-level nodes of the corresponding connections.
In order to achieve the above object, the present invention further provides a method for synchronously distributing files in real time, which comprises the following steps:
step S1, selecting configuration source files and catalogues, setting a plurality of categories according to the selected files and catalogues, preparing data sending channels of corresponding categories, carrying out authentication connection between a superior node and a subordinate node, and classifying the subordinate node connection into the channels of corresponding categories according to a connection request;
step S2, the superior node monitors the directory and the file in real time, and when the file is updated, the data of the file change block is obtained through the mirror image process of the superior node and the subordinate node;
step S3, grouping the file change block data into blocks, grouping each data into packets, and adding the packets to each connection under the corresponding category data sending channel after high-efficiency compression processing;
step S4, traversing the connection under the data transmission channel, and transmitting the processed compressed data packet to the correspondingly connected lower node;
and step S5, after receiving the compressed data packet, the lower node decompresses the compressed data packet, judges the data type according to the data packet identifier, and performs synchronous receiving processing according to the data type.
Preferably, the step S1 further includes:
step S100, a lower node initiates a connection request for connecting an upper node, if the connection of the lower node with the upper node fails, a next standby node is selected to continuously try to connect; if the connection with the superior node is successful, the step S102 is entered for further sending an authentication request;
step S101, the superior node monitors the port, monitors the connection of the subordinate node, and waits for the authentication request sent by the subordinate node after the connection is successful.
Step S102, a lower node sends an authentication request, wherein the authentication request at least comprises request type information;
step S103, analyzing an authentication request sent by a lower node, checking the authentication request, and if the authentication is passed, adding connection to a data sending channel of a corresponding type according to the type information of the authentication request; if the authentication fails, sending response data of authentication failure to the lower node;
and step S104, if the lower node receives the response data of the authentication failure, the lower node selects the next standby node to continuously try to connect, and if the authentication is successful, the authorized category returned by the upper node is acquired, and the channel corresponding to the category is established and initialized.
Preferably, in step S102, the verification authentication request is to verify whether the login information is allowed to pass.
Preferably, the step S2 further includes:
step S200, after the authentication connection is successful, the superior node reads all data in the monitoring directory and the file, calculates the hash value in a blocking mode and records offset information;
step S201, the hash list and the offset list of the files under the corresponding category are packaged and processed, and the final data packet is sent to each subordinate connection;
step S202, after receiving the data, the lower node reads the hash value of the corresponding block of the local corresponding file, judges whether the corresponding hash values are equal, skips if the hash values are equal, records the offset information if the hash values are not equal, and sends the offset information of the data blocks with difference after comparison calculation to the upper node;
in step S203, the upper node receives the offset information of the calculated difference data sent by the lower node, and obtains partial difference block data of the corresponding file according to the offset information.
Preferably, in step S201, the corresponding lower connection in the corresponding category data transmission channel is traversed, the file offset information and the hash value are compressed and then added to the transmission queue, then the corresponding block data content is compressed and then added to the transmission queue, and the start and end marks of the single file are recorded.
Preferably, the step S3 further includes:
step S300, grouping the file change block data into blocks, setting information such as a file start identifier, a file end identifier, an offset position, a file size, a file attribute and the like, and adding the information into a data processing queue of a corresponding category data sending channel.
Step S301, traversing and reading data blocks from the data processing queue, and compressing data by using a high-efficiency compression algorithm;
step S302, traversing the lower level connection in the data transmission channel, adding the identification information to the address of the processed compressed data packet, and adding the address into the task queue.
Preferably, the step S4 further includes:
step S400, acquiring tasks from the task queue;
step S401, traversing each connection under a data transmission channel through a current task, and transmitting a compressed data packet;
step S402, if the transmission fails, the current data is stored in the transmission buffer queue connected at present, and the next data packet is sent again when arriving, so as to avoid data packet loss; and if the transmission is finished, subtracting one from the reference count until the reference count returns to zero, and releasing the corresponding memory resource.
Preferably, in step S402, if it is detected that the sending buffer queue of the current connection is full, it indicates that the current connection is abnormal in receiving or network, the current connection of the next stage is disconnected, and an abnormal log is recorded to indicate that the line is abnormal.
Preferably, the step S5 further includes:
step S500, a lower node receives a compressed data packet sent by a higher node and decompresses the data packet;
step S501, judging the data type through the data packet identification;
step S502, if the type is a file start identifier, analyzing and reading the file name, judging whether the file in the current directory exists, if so, opening the file, and if not, creating and opening the file; if the type is the offset data identifier, writing data to a local file according to the offset; if the type is the ending mark, closing the handle of the current file, reading the attribute information in the data packet, and writing the attribute information into the current file.
Compared with the prior art, the system and the method for synchronously distributing the files in real time select the configuration source files and the directories through the superior node, setting multiple categories according to the selected files and directories, preparing data transmission channels of corresponding categories, classifying the lower level connections into the channels of corresponding categories according to the connection requests, and scans the catalog and source file in real time, obtains file change data through the mirror image process of the upper and lower nodes, the obtained file change data is added to each connection under a data sending channel after being compressed by a block packaging packet, and traversing the connection under the channel, and sending the data to the correspondingly connected subordinate nodes, so that the market file is quickly and synchronously distributed to a plurality of subordinate nodes, and the method has the advantages of small data transmission amount, small occupied bandwidth, sufficiently low forwarding delay, high efficiency in the transmission process, real-time effectiveness, stability, reliability and the like after the data are forwarded to the subordinate nodes.
Drawings
FIG. 1 is a topological structure diagram of a real-time file synchronous distribution system according to the present invention;
FIG. 2 is a flowchart illustrating the steps of a real-time synchronous file distribution method according to the present invention;
FIG. 3 is a general flowchart of upper node file distribution in an embodiment of the present invention;
fig. 4 is a flowchart of connection authentication of a lower node in the embodiment of the present invention;
fig. 5 is a flowchart of connection authentication of an upper node in the embodiment of the present invention;
FIG. 6 is a flow chart of data blocking and compression for a superordinate node in an embodiment of the present invention;
fig. 7 is a flowchart of data transmission of an upper node in the embodiment of the present invention;
fig. 8 is a flow chart of a reception process of a lower node in the embodiment of the present invention;
FIG. 9 is a flowchart illustrating a process of calculating differences at a lower node in a mirroring process according to an embodiment of the present invention;
fig. 10 is a flowchart illustrating a process of calculating a difference by a higher node in the mirroring process according to the embodiment of the present invention.
Detailed Description
Other advantages and capabilities of the present invention will be readily apparent to those skilled in the art from the present disclosure by describing the embodiments of the present invention with specific embodiments thereof in conjunction with the accompanying drawings. The invention is capable of other and different embodiments and its several details are capable of modification in various other respects, all without departing from the spirit and scope of the present invention.
Fig. 1 is a topological architecture diagram of a file real-time synchronization distribution system according to the present invention. As shown in fig. 1, the file real-time synchronous distribution system of the present invention includes multiple hierarchies, each hierarchy includes multiple nodes, each node includes a sending module and a receiving module, each layer of nodes includes a main node and a standby node (under the condition of a sufficient condition, multiple standby nodes can be set, and generally one main node and one standby node are needed), and when the line connection of the main node is interrupted, the main node can be automatically switched to the standby node, as shown in fig. 1, a solid line represents the data flow direction of the main node, and a dotted line represents the data flow direction of the standby node.
The method comprises the steps that a superior node (a node at the upper level) selects a configuration source file and a directory, a plurality of categories are set according to the selected file and the directory, a data sending channel of the corresponding category is prepared, subordinate connections are classified into the channels of the corresponding category according to connection requests, the directory and the source file are scanned in real time, file change data are obtained, after block packaging and compression processing, the file change data are added to all connections under the data sending channel, connections under the channels are traversed, and the data are sent to subordinate nodes which are correspondingly connected, so that the purposes of saving host resources and network resources and efficiently transmitting the file to the subordinate nodes and using the file are achieved.
FIG. 2 is a flowchart illustrating steps of a real-time synchronous file distribution method according to the present invention. As shown in fig. 2, the method for synchronously distributing files in real time of the present invention includes the following steps:
step S1, selecting a configuration source file and a directory, setting a plurality of categories according to the selected file and directory, preparing data transmission channels of corresponding categories, performing authentication connection between a higher node and a lower node, and classifying the lower node connection into the channels of corresponding categories according to a connection request. In the present invention, the categories can be customized, taking market data as an example, such as: and the stock is handed in to a FAST market, the option market, the certificate in the stock, the deep-hand exchange index and the deep-hand exchange comprehensive market, and each category corresponds to different files or directories.
Specifically, step S1 further includes:
step S100, a lower node initiates a connection request for connecting an upper node, if the connection of the lower node with the upper node fails, a next standby node is selected to continuously try to connect; if the connection with the superior node is successful, the step S102 is entered for further sending an authentication request;
step S101, the superior node monitors the port, monitors the connection of the subordinate node, and waits for the authentication request sent by the subordinate node after the connection is successful.
Step S102, a lower node sends an authentication request, wherein the authentication request comprises request type information;
step S103, analyzing the authentication request sent by the lower node, checking the authentication request, judging the type information in the authentication request if the authentication is passed, merging the type information in the current request into a channel of a corresponding type according to the type information in the current request, and adding the connection into the corresponding data sending channel; if the authentication fails, sending response data of authentication failure to the lower node, in the specific embodiment of the invention, checking the authentication request, namely checking whether the login information can pass, and if the authentication fails, sending the response data of authentication failure to the lower node;
and step S104, if the lower node receives the response data of the authentication failure, the lower node selects the next standby node to continuously try to connect, and if the authentication is successful, the authorized category returned by the upper node is acquired, and the channel corresponding to the category is established and initialized.
And step S2, the superior node monitors the directory and the file in real time, and when the file is updated, the data of the file change block is obtained through the mirror image process of the superior node and the subordinate node.
Specifically, step S2 further includes:
step S200, after the authentication connection is successful, the superior node reads all data in the monitoring directory and the file, calculates the hash value in a blocking mode and records offset information;
step S201, package processing the hash list and offset list of the file in the corresponding category, and sending the final data package to each subordinate connection. Specifically, traversing the corresponding subordinate connection in the corresponding category data sending channel, compressing the file offset information and the hash value, adding the compressed file offset information and the compressed hash value into a sending queue, compressing the corresponding block data content, adding the compressed block data content into the sending queue, and recording the start and end marks of a single file.
Step S202, after receiving the data, the lower node reads the hash value of the corresponding block of the local corresponding file, and judges whether the corresponding hash values are equal, if equal, the corresponding hash values are skipped, if unequal, the current block is changed, and difference synchronization is needed, at the moment, offset information is recorded, and the offset information of the data blocks with difference after comparison and calculation is sent to the upper node;
in step S203, the upper node receives the offset information of the calculated difference data sent by the lower node, and obtains partial difference block data of the corresponding file, that is, the file change block data, according to the offset information.
And step S3, grouping the file change block data into blocks, grouping each data into packets, and adding the packets to each connection under the corresponding category data sending channel after efficient compression processing.
Specifically, step S3 further includes:
step S300, grouping the file change block data into blocks, setting information such as a file start identifier, an end identifier, an offset position, a file size, a file attribute and the like, and adding the information into a data processing queue of a corresponding category data sending channel.
Step S301, traversing and reading data blocks from the data processing queue, and compressing data by using a high-efficiency compression algorithm;
step S302, traversing the lower level connection in the data transmission channel, adding the identification information to the address of the processed compressed data packet, and adding the address into the task queue.
And step S4, traversing the connection under the data transmission channel, and transmitting the processed compressed data packet to the lower node of the corresponding connection.
Specifically, step S4 further includes:
step S400, acquiring tasks from a task queue;
step S401, traversing each connection under a data transmission channel through a current task, and transmitting a compressed data packet;
step S402, if the transmission fails, the current data is stored in the transmission buffer queue connected at present, and the next data packet is sent again when arriving, so as to avoid data packet loss; and if the transmission is finished, subtracting one from the reference count until the reference count returns to zero, and releasing the corresponding memory resource. Preferably, if it is detected that the sending buffer queue of the current connection is full, it indicates that the current connection is abnormal in receiving or the network is abnormal, the current connection of the next stage is disconnected, and an abnormal log is recorded to indicate that the line is abnormal.
And step S5, after receiving the compressed data packet, the lower node decompresses the compressed data packet, judges the data type according to the data packet identifier, and performs synchronous receiving processing according to the data type.
Specifically, step S5 further includes:
step S500, a lower node receives a compressed data packet sent by a higher node through a socket and decompresses the data packet;
step S501, judging the data type through the data packet identification;
step S502, if the type is a file start identifier, analyzing and reading the file name, judging whether the file in the current directory exists, if so, opening the file, and if not, creating and opening the file; if the type is the offset data identifier, writing data to a local file according to the offset; if the type is the ending mark, closing the handle of the current file, reading the attribute information in the data packet, writing the attribute information into the current file, and completing the synchronization.
Examples
Fig. 3 is a general flowchart of upper node file distribution in the embodiment of the present invention, and the process is as follows:
first, directories and files are monitored.
And when the file is monitored to be updated, acquiring data of the file change block.
And reading the changed data in blocks, packaging the new data, and adding the data to each connection under the data sending channels of the corresponding categories after efficient compression processing.
And finally, traversing the connection under the data sending channel and sending the data to the lower level of the corresponding connection.
Fig. 4 is a flowchart illustrating connection authentication of a lower node (also referred to as a client) according to an embodiment of the present invention, where the connection authentication process is as follows:
firstly, a receiving module of a lower node initiates a connection request, and if the connection with the upper node fails, a next standby node is selected to continuously try connection;
after the connection of the superior node is successful, sending an authentication request, wherein the authentication request contains request type information, and if the authentication is failed, selecting the next standby node to continuously try to connect;
and after the authentication is successful, obtaining the authorized categories returned by the superior node, and establishing and initializing channels corresponding to the categories.
And after the data is ready, waiting for receiving the data.
Fig. 5 is a flowchart of connection authentication of an upper node (also referred to as a server) in the embodiment of the present invention, and the process is as follows:
first, a service listening port is opened, and a lower node (client) is waited for connection. After the connection is successful, waiting for an authentication request sent by a lower node client;
analyzing an authentication request sent by a lower node client, checking whether login information can pass or not, and sending response data of authentication failure to the lower node client if the login information cannot pass;
after the authentication is successful, judging the type information in the authentication request, merging the type information in the current request into a channel of a corresponding type according to the type information in the current request, and adding the connection into a corresponding data sending channel. Until the connection authentication is completed, the document data can be transmitted next.
Fig. 6 is a flowchart of data blocking and compression of an upper node in the embodiment of the present invention, and the process is as follows:
firstly, monitoring and scanning directories and files to obtain file change data;
then, grouping the changed block data into blocks, setting information such as a file start identifier, an end identifier, an offset position, a file size, a file attribute and the like, and adding the information into a data processing queue of a corresponding type data sending channel;
next, the read data block is traversed from the data processing queue, compressing the data using an efficient compression algorithm.
And finally, traversing the lower-level connection in the data sending channel, adding identification information to the address of the processed compressed data packet, and adding the address into a task queue.
Fig. 7 is a data transmission flow chart of an upper node in the embodiment of the present invention, where a data transmission process is as follows:
firstly, a sending module acquires a task from a task queue;
then, traversing each connection under the data transmission channel through the current task, starting to transmit data, if the transmission fails, storing the current data into the transmission cache queue of the current connection, and re-trying to transmit the next data packet when the next data packet arrives, so as to avoid data packet loss; and if the transmission is finished, subtracting one from the reference count until the reference count returns to zero, and releasing the corresponding memory resource. If the sending buffer queue of the current connection is detected to be full, the current connection receiving abnormity or network abnormity is shown, the current connection of the next stage is disconnected, and an abnormity log is recorded to show the line abnormity.
Fig. 8 is a flow chart of a receiving process of a lower node (client) according to an embodiment of the present invention, where the receiving process is as follows:
first, the lower node receives a compressed packet through a socket and decompresses the packet.
And then, judging the data type through the data packet identifier, if the data type is the file start identifier, analyzing and reading the file name, judging whether the file exists in the current directory, if the file exists, opening the file, and if the file does not exist, creating and opening the file. And if the type is the offset data identification, writing data to the local file according to the offset. If the type is the receiving identification, closing the handle of the current file, reading the attribute information in the data packet, writing the attribute information into the current file, and synchronizing the current state.
Fig. 9 is a flowchart of a process of calculating a difference by a lower node (client) in the mirroring process according to the embodiment of the present invention, where the process of calculating the difference is as follows:
firstly, after connection authentication is successful, a superior node server side can send out the block hash value of a file under a corresponding category, and a client side reads the hash value of a block corresponding to a local corresponding file after receiving the data;
then, the comparison task judges whether the corresponding hash values are equal, if equal, skipping, if not, indicating that the current block is changed and needs differential synchronization, and at the moment, recording offset information;
and finally, sending the data block offset information with difference after comparison and calculation to a superior node.
Fig. 10 is a processing procedure of calculating a difference by a superior node (server) in the mirroring process according to the embodiment of the present invention, where the processing procedure is as follows:
firstly, reading all data in the monitoring directory and the file, calculating a hash value in a blocking mode, and recording offset information.
And then, the hash list and the offset list of the files under the corresponding category are packaged and processed, and the final data packet is sent to each subordinate connection.
And finally, receiving offset information of the calculated difference data sent by the lower node, and sending part of difference block data of the corresponding file to the appointed lower node according to the offset information to finish the process of data difference mirroring.
To sum up, the present invention provides a system and method for synchronously distributing files in real time, which selects a configuration source file and a directory through a superior node, sets a plurality of categories according to the selected file and directory, prepares a data transmission channel of a corresponding category, classifies subordinate connections into the channels of the corresponding category according to connection requests, scans the directory and the source file in real time, obtains file change data through the mirroring process of the superior node and the subordinate node, compresses the obtained file change data through block packets, adds the file change data to each connection under the data transmission channel, traverses the connections under the channels, and transmits the data to the subordinate nodes of the corresponding connections, thereby realizing the purpose of rapidly and synchronously distributing the quotation files to the plurality of subordinate nodes, and having the advantages of small data transmission amount, small occupied bandwidth, sufficiently low forwarding delay, efficient transmission process, real-time effectiveness and stability after the data is forwarded to the subordinate nodes, etc., the invention selects a TCP data transmission protocol, and the data transmission selects an efficient compression mode.
Compared with the prior art, the invention has the following advantages:
1. the data of a compression mode is provided, the compression rate of the compressed data is high, and the method is suitable for transmission under a narrow-band environment;
2. the file mirroring process uses the differential mirroring synchronous data, so that the transmission data volume is smaller;
3. the data with changed files only synchronously changes the block data, and the changed large block data continues to be transmitted in blocks;
4. the invention supports multi-level cascade transmission, and can support forwarding to an internal network in a mode of deploying nodes on the bastion machine under the condition that different networks are isolated;
5. the invention supports simultaneous connection and forwarding to a plurality of lower nodes, and is suitable for the requirement of forwarding market data from a headquarter single node to all business departments.
6. The invention supports high availability, and when data is detected or connection is interrupted or overtime, the standby node can be switched to, so that the high availability of market data is ensured.
The foregoing embodiments are merely illustrative of the principles and utilities of the present invention and are not intended to limit the invention. Modifications and variations can be made to the above-described embodiments by those skilled in the art without departing from the spirit and scope of the present invention. Therefore, the scope of the invention should be determined from the following claims.
Claims (10)
1. A real-time synchronous file distribution system comprises a plurality of levels, each level comprises a plurality of nodes, each node comprises a sending module and a receiving module, each level of nodes comprises a main node and a standby node, and the main node is automatically switched to the standby node when the line connection of the main node is interrupted; the method comprises the steps that an upper node at the upper level selects a configuration source file and a directory, a plurality of categories are set according to the selected file and the directory, data sending channels of the corresponding categories are prepared, lower-level connections are classified into the channels of the corresponding categories according to connection requests, the directory and the source file are scanned in real time, file change data are obtained through the mirror image process of the upper node and the lower node, the obtained file change data are subjected to block packaging and compression processing, added to all connections under the data sending channels, the connections under the channels are traversed, and the data are sent to the lower-level nodes of the corresponding connections.
2. A real-time synchronous file distribution method comprises the following steps:
step S1, selecting configuration source files and catalogues, setting a plurality of categories according to the selected files and catalogues, preparing data sending channels of corresponding categories, carrying out authentication connection between a superior node and a subordinate node, and classifying the subordinate node connection into the channels of corresponding categories according to a connection request;
step S2, the superior node monitors the directory and the file in real time, and when the file is updated, the data of the file change block is obtained through the mirror image process of the superior node and the subordinate node;
step S3, grouping the file change block data into blocks, grouping each data into packets, and adding the packets to each connection under the corresponding category data sending channel after high-efficiency compression processing;
step S4, traversing the connection under the data transmission channel, and transmitting the processed compressed data packet to the correspondingly connected lower node;
and step S5, after receiving the compressed data packet, the lower node decompresses the compressed data packet, judges the data type according to the data packet identifier, and performs synchronous receiving processing according to the data type.
3. The method for real-time synchronous file distribution according to claim 2, wherein the step S1 further comprises:
step S100, a lower node initiates a connection request for connecting an upper node, if the connection of the lower node with the upper node fails, a next standby node is selected to continuously try to connect; if the connection with the superior node is successful, the step S102 is entered for further sending an authentication request;
step S101, the superior node monitors the port, monitors the connection of the subordinate node, and waits for the authentication request sent by the subordinate node after the connection is successful.
Step S102, a lower node sends an authentication request, wherein the authentication request at least comprises request type information;
step S103, analyzing an authentication request sent by a lower node, checking the authentication request, and if the authentication is passed, adding connection to a data sending channel of a corresponding type according to the type information of the authentication request; if the authentication fails, sending response data of authentication failure to the lower node;
and step S104, if the lower node receives the response data of the authentication failure, the lower node selects the next standby node to continuously try to connect, and if the authentication is successful, the authorized category returned by the upper node is acquired, and the channel corresponding to the category is established and initialized.
4. The method for real-time synchronous file distribution according to claim 3, wherein: in step S102, the verification authentication request is to verify whether the login information is allowed to pass.
5. The method for real-time synchronous file distribution according to claim 2, wherein the step S2 further comprises:
step S200, after the authentication connection is successful, the superior node reads all data in the monitoring directory and the file, calculates the hash value in a blocking mode and records offset information;
step S201, the hash list and the offset list of the files under the corresponding category are packaged and processed, and the final data packet is sent to each subordinate connection;
step S202, after receiving the data, the lower node reads the hash value of the corresponding block of the local corresponding file, judges whether the corresponding hash values are equal, skips if the hash values are equal, records the offset information if the hash values are not equal, and sends the offset information of the data blocks with difference after comparison calculation to the upper node;
in step S203, the upper node receives the offset information of the calculated difference data sent by the lower node, and obtains partial difference block data of the corresponding file according to the offset information.
6. The method for real-time synchronous file distribution according to claim 5, wherein: in step S201, the corresponding lower links in the corresponding category data transmission channels are traversed, the file offset information and the hash value are compressed and then added to the transmission queue, the corresponding block data content is compressed and then added to the transmission queue, and the start and end marks of the single file are recorded.
7. The method for real-time synchronous file distribution according to claim 2, wherein the step S3 further comprises:
step S300, grouping the file change block data into blocks, setting information such as a file start identifier, a file end identifier, an offset position, a file size, a file attribute and the like, and adding the information into a data processing queue of a corresponding category data sending channel.
Step S301, traversing and reading data blocks from the data processing queue, and compressing data by using a high-efficiency compression algorithm;
step S302, traversing the lower level connection in the data transmission channel, adding the identification information to the address of the processed compressed data packet, and adding the address into the task queue.
8. The method for real-time synchronous file distribution according to claim 7, wherein the step S4 further comprises:
step S400, acquiring tasks from the task queue;
step S401, traversing each connection under a data transmission channel through a current task, and transmitting a compressed data packet;
step S402, if the transmission fails, the current data is stored in the transmission buffer queue connected at present, and the next data packet is sent again when arriving, so as to avoid data packet loss; and if the transmission is finished, subtracting one from the reference count until the reference count returns to zero, and releasing the corresponding memory resource.
9. The method for real-time synchronous file distribution according to claim 8, wherein: in step S402, if it is detected that the sending buffer queue of the current connection is full, it indicates that the current connection is abnormal in receiving or the network is abnormal, the current connection of the next stage is disconnected, and an abnormal log is recorded to indicate that the line is abnormal.
10. The method for real-time synchronous file distribution according to claim 8, wherein the step S5 further comprises:
step S500, a lower node receives a compressed data packet sent by a higher node and decompresses the data packet;
step S501, judging the data type through the data packet identification;
step S502, if the type is a file start identifier, analyzing and reading the file name, judging whether the file in the current directory exists, if so, opening the file, and if not, creating and opening the file; if the type is the offset data identifier, writing data to a local file according to the offset; if the type is the ending mark, closing the handle of the current file, reading the attribute information in the data packet, and writing the attribute information into the current file.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911233574.2A CN111046008A (en) | 2019-12-05 | 2019-12-05 | System and method for synchronously distributing files in real time |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911233574.2A CN111046008A (en) | 2019-12-05 | 2019-12-05 | System and method for synchronously distributing files in real time |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111046008A true CN111046008A (en) | 2020-04-21 |
Family
ID=70234701
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911233574.2A Pending CN111046008A (en) | 2019-12-05 | 2019-12-05 | System and method for synchronously distributing files in real time |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111046008A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112637229A (en) * | 2020-12-29 | 2021-04-09 | 湖南文理学院 | Network intrusion cooperative detection method based on security cloud |
| CN112822256A (en) * | 2020-12-31 | 2021-05-18 | 上海英方软件股份有限公司 | Method and device for transmitting data stream of quotation file |
| CN112995217A (en) * | 2021-04-29 | 2021-06-18 | 深圳华锐金融技术股份有限公司 | Data sending method and system |
| CN113965572A (en) * | 2021-11-02 | 2022-01-21 | 上海佰贝网络工程技术有限公司 | Block distribution method and system of block chain, computer equipment and computer readable storage medium |
| CN114841648A (en) * | 2022-05-19 | 2022-08-02 | 北京百度网讯科技有限公司 | Material distribution method, device, electronic equipment and medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101026490A (en) * | 2007-01-26 | 2007-08-29 | 华为技术有限公司 | Device, system and method for realizing business distribution and synchronization |
| CN101959054A (en) * | 2009-07-14 | 2011-01-26 | 中国电信股份有限公司 | Integrated P2P (Peer-To-Peer) VOD (Video-On-Demand) system and partner node selecting method |
| CN104735087A (en) * | 2015-04-16 | 2015-06-24 | 国家电网公司 | Public key algorithm and SSL (security socket layer) protocol based method of optimizing security of multi-cluster Hadoop system |
| CN106156359A (en) * | 2016-07-28 | 2016-11-23 | 四川新环佳科技发展有限公司 | A kind of data synchronization updating method under cloud computing platform |
| CN106533967A (en) * | 2016-12-08 | 2017-03-22 | 北京中安智达科技有限公司 | Data transmission method capable of customizing load balance strategy |
| CN107491565A (en) * | 2017-10-10 | 2017-12-19 | 语联网（武汉）信息技术有限公司 | A kind of method of data synchronization |
-
2019
- 2019-12-05 CN CN201911233574.2A patent/CN111046008A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101026490A (en) * | 2007-01-26 | 2007-08-29 | 华为技术有限公司 | Device, system and method for realizing business distribution and synchronization |
| CN101959054A (en) * | 2009-07-14 | 2011-01-26 | 中国电信股份有限公司 | Integrated P2P (Peer-To-Peer) VOD (Video-On-Demand) system and partner node selecting method |
| CN104735087A (en) * | 2015-04-16 | 2015-06-24 | 国家电网公司 | Public key algorithm and SSL (security socket layer) protocol based method of optimizing security of multi-cluster Hadoop system |
| CN106156359A (en) * | 2016-07-28 | 2016-11-23 | 四川新环佳科技发展有限公司 | A kind of data synchronization updating method under cloud computing platform |
| CN106533967A (en) * | 2016-12-08 | 2017-03-22 | 北京中安智达科技有限公司 | Data transmission method capable of customizing load balance strategy |
| CN107491565A (en) * | 2017-10-10 | 2017-12-19 | 语联网（武汉）信息技术有限公司 | A kind of method of data synchronization |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112637229A (en) * | 2020-12-29 | 2021-04-09 | 湖南文理学院 | Network intrusion cooperative detection method based on security cloud |
| CN112822256A (en) * | 2020-12-31 | 2021-05-18 | 上海英方软件股份有限公司 | Method and device for transmitting data stream of quotation file |
| CN112995217A (en) * | 2021-04-29 | 2021-06-18 | 深圳华锐金融技术股份有限公司 | Data sending method and system |
| CN113965572A (en) * | 2021-11-02 | 2022-01-21 | 上海佰贝网络工程技术有限公司 | Block distribution method and system of block chain, computer equipment and computer readable storage medium |
| CN113965572B (en) * | 2021-11-02 | 2024-05-14 | 上海佰贝网络工程技术有限公司 | Block distribution method, system, computer equipment and computer readable storage medium of block chain |
| CN114841648A (en) * | 2022-05-19 | 2022-08-02 | 北京百度网讯科技有限公司 | Material distribution method, device, electronic equipment and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111046008A (en) | System and method for synchronously distributing files in real time | |
| US7587506B2 (en) | Computer system and data backup method in computer system | |
| US20160308960A1 (en) | Connection management system, and a method for linking connection management server in thin client system | |
| US7584292B2 (en) | Hierarchical system configuration method and integrated scheduling method to provide multimedia streaming service on two-level double cluster system | |
| EP2200000B1 (en) | Real-time image monitoring and recording system and method | |
| CN101707543A (en) | Enterprise media bus system supporting multi-task type and enterprise media bus method supporting multi-task type | |
| CN108173691B (en) | Cross-device aggregation method and device | |
| WO2017088705A1 (en) | Data processing method and device | |
| CN109600410B (en) | Data storage system and method | |
| JP4202026B2 (en) | Storage system and storage device | |
| CN107819891A (en) | Data processing method, device, computer equipment and storage medium | |
| US11614874B2 (en) | Method and system for data storage and management | |
| US11711810B1 (en) | System, security and network management using self-organizing communication orbits in distributed networks | |
| JP7097427B2 (en) | Data processing system and data processing method | |
| JP4550604B2 (en) | Setting information synchronization program | |
| CN110290163A (en) | A kind of data processing method and device | |
| CN111600929A (en) | Transmission line detection method, route strategy generation method and proxy server | |
| CN114500254B (en) | Historical log collection method, device and storage medium | |
| CN112925848B (en) | Data synchronization method, device, equipment and readable storage medium | |
| US20160269246A1 (en) | Method and apparatus for managing computer system | |
| US9654336B2 (en) | Information processing apparatus having a function of supporting access to managing apparatuses, information processing system, and method of processing information | |
| CN111355573A (en) | Communication apparatus and communication method for processing authentication information | |
| CN111327666A (en) | Service management method, device and system, computer equipment and storage medium | |
| JP2005100343A (en) | Server having connection management function | |
| JP3740233B2 (en) | MONITOR SYSTEM, CONTROL METHOD, AND INFORMATION PROCESSING DEVICE |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |