CN111935266A - Data distribution system - Google Patents
Data distribution system Download PDFInfo
- Publication number
- CN111935266A CN111935266A CN202010766890.2A CN202010766890A CN111935266A CN 111935266 A CN111935266 A CN 111935266A CN 202010766890 A CN202010766890 A CN 202010766890A CN 111935266 A CN111935266 A CN 111935266A
- Authority
- CN
- China
- Prior art keywords
- data
- sending end
- data receiving
- receiving ends
- nodes
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 claims abstract description 6
- 230000005540 biological transmission Effects 0.000 claims description 4
- 238000013144 data compression Methods 0.000 claims 1
- 238000001514 detection method Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000005856 abnormality Effects 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000006837 decompression Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 230000001681 protective effect Effects 0.000 description 1
- 230000004622 sleep time Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/04—Protocols for data compression, e.g. ROHC
Abstract
The invention discloses a data distribution system, which comprises a data sending end and one or more data receiving ends. The data sending end used for automatically acquiring the data respectively sends the compressed data to one or more designated data receiving ends, and the data receiving ends decompress and store the data; the data sending end is provided with a distribution unit which correspondingly distributes different data receiving ends for the data of different types or nodes, and the data sending end correspondingly sends the data of different types or nodes to different data receiving ends according to the distribution of the distribution unit. The data sending end is used for automatically acquiring data from one or more directories and then sending the data to other directories of the local computer or data receiving ends of other hosts according to distribution. Meanwhile, in the whole process, all files passing through the sending end program can be counted in real time, and classified counting is carried out according to the node numbers and the data types, so that the real-time flow of the nodes can be monitored.
Description
Technical Field
The invention relates to the technical field of network data transmission, in particular to a data distribution system.
Background
The network data reduction and analysis are common means for some enterprises and government departments to perform network anomaly detection, detect network attacks and analyze user behaviors. In the whole process, the restored data file needs to be imported into a big data platform for mining analysis. In the whole link, the restored data are of various types, the detected network nodes are also of multiple types, how to transmit the data acquired by different nodes is achieved, and meanwhile, the data from one source is divided into two or even more, which is a problem to be solved in a big data detection platform.
Disclosure of Invention
The present invention is directed to a data offloading system to solve the above problems.
In order to achieve the above object, the present invention provides a data offloading system, which includes a data transmitting end and a plurality of data receiving ends. The data sending end used for obtaining the data respectively sends the compressed data to one or more appointed data receiving ends, and the data receiving ends decompress and store the data;
the data sending end is provided with a distribution unit, the distribution unit can correspondingly distribute different data receiving ends for different types or nodes of data, and the data sending end correspondingly sends the different types or nodes of data to different data receiving ends according to the distribution of the distribution unit.
Optionally, the data sending end is provided with a statistical unit, and the statistical unit is used for counting data files sent by the data sending end to the data receiving end and recording results according to nodes and types of the data.
Optionally, the data sending end sets a configuration unit, and the configuration unit configures priority transmission for a specific node or a specific type of data file.
Optionally, the data receiving end is any one or more of the following: a standard data receiving terminal program, a standard distributed data pool data receiving program and an SFTP server.
Optionally, a threshold is set for a supporting storage disk of the data receiving end, and when the storage rate of the storage disk reaches the set threshold, the data receiving end stops receiving the data file.
The invention has the beneficial effects that:
the data distribution system provides a stable and reliable mode for transmitting and distributing data, and meanwhile, data statistics and flow control are carried out, so that the data stability of a large data platform can be greatly improved.
Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention and not to limit the invention. In the drawings:
fig. 1 is a block diagram of a data distribution system according to the present invention.
Detailed Description
The following detailed description of embodiments of the invention refers to the accompanying drawings. It should be understood that the detailed description and specific examples, while indicating the present invention, are given by way of illustration and explanation only, not limitation.
As shown in fig. 1, the data distribution system according to the present invention includes a data sending end and a plurality of data receiving ends, where the data sending end for obtaining data compresses the data and then sends the compressed data to the plurality of designated data receiving ends, and the data receiving ends decompress and store the data. If the sender sends uncompressed files, the receiver can directly receive the uncompressed files, and if the sender sends compressed files, the receiver can automatically recognize the uncompressed files and decompress and store the compressed files.
The data sending end can automatically acquire the data files, distribute the files to different data receiving ends, and ensure that the files received by the different data receiving ends are consistent. Meanwhile, the data sending end also supports sending the specified file.
Specifically, there are three data receiving terminals, which are respectively a standard data receiving terminal program, a standard distributed data pool data receiving program, and an SFTP server terminal provided by the data distribution system.
The data sending end is provided with a distribution unit which correspondingly distributes different data receiving ends for the data of different types or nodes, and the data sending end correspondingly sends the data of different types or nodes to the data receiving ends according to the distribution of the distribution unit. The allocation unit only configures data of a specific node or type, and the data sending end only sends the configured data of the node and type.
Optionally, the data sending end is provided with a statistical unit, and the statistical unit is used for counting data files sent by the data sending end to the data receiving end and recording results according to nodes and types of the data.
The statistical unit generates data statistical results of different nodes and different types in each hour, wherein the data statistical results comprise the number and the total size of the sent files.
Optionally, the data sending end sets a configuration unit, and the configuration unit configures priority transmission for a specific node or a specific type of data file.
Optionally, a storage disk of the data receiving end sets a threshold, and when the storage rate of the storage disk reaches the set threshold, the data receiving end stops receiving the data file. If the data volume sent by the sending end is too large, the storage rate of the storage disk of the data receiving end is continuously increased, when the storage rate of the disk reaches a set threshold value, the receiving end program can automatically sleep and stop receiving the file, so that the abnormality of the whole operating system caused by the fact that the disk space is used up is avoided.
The data sending end is used for automatically acquiring data from one or more directories and then sending the data to other directories of the local computer or data receiving ends of other hosts according to distribution. Meanwhile, in the whole process, all files passing through the sending end program can be counted in real time, classified counting is carried out according to the node numbers and the data types, files in a fixed format are generated, and the files can monitor the real-time flow of the nodes. The data distribution system provides a stable and reliable mode for transmitting and distributing data, and meanwhile, data statistics and flow control are carried out, so that the data stability of a large data platform can be greatly improved.
The working method of the data distribution system comprises the following steps:
the method comprises the following steps: and configuring and deploying the data receiving end. One or more of the above data receiving end programs may be configured.
Further, if a standard data receiving end program provided by the system needs to be deployed, the IP, the port, the temporary file storage directory, the final received file storage directory, the queue size, the size of the decompression task thread pool, the threshold of the disk utilization rate, and the sleep time after the disk utilization rate exceeds the threshold need to be configured.
Further, if a standard distributed data pool receiving program is to be deployed, the IP and port of the server where the receiving program is located need to be configured.
Further, if a data receiving service end of the SFTP is to be deployed, a SFTP service needs to be deployed separately.
Step two: and configuring a data transmitting end of the data distribution system. Reading source directories, file temporary storage directories, the priority of the source directories, the priority of nodes, the priority of data types, data receiving ends and the like of data need to be configured.
Step three: and starting a data sending end program, and automatically acquiring and shunting data.
The preferred embodiments of the present invention have been described in detail with reference to the accompanying drawings, however, the present invention is not limited to the specific details of the above embodiments, and various simple modifications can be made to the technical solution of the present invention within the technical idea of the present invention, and these simple modifications are within the protective scope of the present invention.
It should be noted that, in the foregoing embodiments, various features described in the above embodiments may be combined in any suitable manner, and in order to avoid unnecessary repetition, various possible combinations are not described herein.
In addition, any combination of the various embodiments of the present invention is also possible, and the same should be considered as the disclosure of the present invention as long as it does not depart from the spirit of the present invention.
Claims (5)
1. Data distribution system, its characterized in that: the data compression method comprises a data sending end and a plurality of data receiving ends, wherein the sending end used for obtaining data compresses the data and then respectively sends the compressed data to the plurality of appointed data receiving ends, and the data receiving ends decompress and store the data;
the data sending end is provided with a distribution unit which correspondingly distributes different data receiving ends for the data of different types or nodes, and the data sending end correspondingly sends the data of different types or nodes to different data receiving ends according to the distribution of the distribution unit.
2. The data splitting system of claim 1, wherein: the data sending end is provided with a statistical unit, the statistical unit is used for counting data files sent by the data sending end to the data receiving end, and recording results according to nodes and types of the data.
3. The data splitting system of claim 1, wherein: the data sending end is provided with a configuration unit, and the configuration unit configures priority transmission for a specific node or a specific type of data file.
4. The data splitting system of claim 1, wherein: the data receiving end is any one or more of the following: a standard data receiving terminal program, a standard distributed data pool data receiving program and an SFTP server.
5. The data splitting system of claim 1, wherein: and setting a threshold value for a storage disk of the data receiving end, and stopping receiving the data file by the data receiving end when the storage rate of the storage disk reaches the set threshold value.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010766890.2A CN111935266A (en) | 2020-08-03 | 2020-08-03 | Data distribution system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010766890.2A CN111935266A (en) | 2020-08-03 | 2020-08-03 | Data distribution system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111935266A true CN111935266A (en) | 2020-11-13 |
Family
ID=73306428
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010766890.2A Pending CN111935266A (en) | 2020-08-03 | 2020-08-03 | Data distribution system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111935266A (en) |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020143926A1 (en) * | 2000-12-07 | 2002-10-03 | Maltz David A. | Method and system for collecting traffic data in a computer network |
| CN1561007A (en) * | 2004-02-27 | 2005-01-05 | 中兴通讯股份有限公司 | Device and method for data compression decompression in data transmission |
| EP1608116A1 (en) * | 2004-06-18 | 2005-12-21 | Agere Systems, Inc. | Method and apparatus for per-service fault protection and restoration in a packet network |
| CN102098301A (en) * | 2011-01-06 | 2011-06-15 | 复旦大学 | Multilink self-adaptation data transmission method and system |
| CN103778212A (en) * | 2014-01-16 | 2014-05-07 | 国网山东省电力公司青岛供电公司 | Data node-based parallel massive data processing method |
| CN103795655A (en) * | 2012-10-30 | 2014-05-14 | 中兴通讯股份有限公司 | Data transmission method and data transmission device |
| CN104283801A (en) * | 2013-07-04 | 2015-01-14 | 中兴通讯股份有限公司 | Method and system for processing service data |
| CN106411872A (en) * | 2016-09-21 | 2017-02-15 | 杭州迪普科技有限公司 | Method and device for compressing messages based on data message classification |
| CN106846065A (en) * | 2017-02-07 | 2017-06-13 | 咪咕互动娱乐有限公司 | A kind of data processing method and device |
| CN107634915A (en) * | 2017-08-25 | 2018-01-26 | 中国科学院计算机网络信息中心 | Data transmission method, device and storage medium |
| CN111241078A (en) * | 2020-01-07 | 2020-06-05 | 网易（杭州）网络有限公司 | Data analysis system, data analysis method and device |
-
2020
- 2020-08-03 CN CN202010766890.2A patent/CN111935266A/en active Pending
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020143926A1 (en) * | 2000-12-07 | 2002-10-03 | Maltz David A. | Method and system for collecting traffic data in a computer network |
| CN1561007A (en) * | 2004-02-27 | 2005-01-05 | 中兴通讯股份有限公司 | Device and method for data compression decompression in data transmission |
| EP1608116A1 (en) * | 2004-06-18 | 2005-12-21 | Agere Systems, Inc. | Method and apparatus for per-service fault protection and restoration in a packet network |
| CN102098301A (en) * | 2011-01-06 | 2011-06-15 | 复旦大学 | Multilink self-adaptation data transmission method and system |
| CN103795655A (en) * | 2012-10-30 | 2014-05-14 | 中兴通讯股份有限公司 | Data transmission method and data transmission device |
| CN104283801A (en) * | 2013-07-04 | 2015-01-14 | 中兴通讯股份有限公司 | Method and system for processing service data |
| CN103778212A (en) * | 2014-01-16 | 2014-05-07 | 国网山东省电力公司青岛供电公司 | Data node-based parallel massive data processing method |
| CN106411872A (en) * | 2016-09-21 | 2017-02-15 | 杭州迪普科技有限公司 | Method and device for compressing messages based on data message classification |
| CN106846065A (en) * | 2017-02-07 | 2017-06-13 | 咪咕互动娱乐有限公司 | A kind of data processing method and device |
| CN107634915A (en) * | 2017-08-25 | 2018-01-26 | 中国科学院计算机网络信息中心 | Data transmission method, device and storage medium |
| CN111241078A (en) * | 2020-01-07 | 2020-06-05 | 网易（杭州）网络有限公司 | Data analysis system, data analysis method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10560350B2 (en) | Systems and methods for monitoring globally distributed remote storage devices | |
| US8528039B2 (en) | System for and method of set-top box memory monitoring | |
| EP3386150B1 (en) | Terminal failure processing method, device and system | |
| CN111143163B (en) | Data monitoring method, device, computer equipment and storage medium | |
| CN103220173B (en) | A kind of alarm monitoring method and supervisory control system | |
| CN108900374B (en) | Data processing method and device applied to DPI equipment | |
| EP3160081A1 (en) | Alert data processing method and network management device | |
| CN110908833A (en) | Data backup method, device and equipment and computer readable storage medium | |
| CN112395156A (en) | Fault warning method and device, storage medium and electronic equipment | |
| CN109710490A (en) | Acquisition method, device, storage medium and the electronic equipment of client log | |
| CN107306200B (en) | Network fault early warning method and gateway for network fault early warning | |
| CN112187513A (en) | Medical Internet of things cloud platform method and system based on big data and storage medium | |
| CN110177024B (en) | Monitoring method of hotspot equipment, client, server and system | |
| US20150215333A1 (en) | Network filtering apparatus and filtering method | |
| CN109788251B (en) | Video processing method, device and storage medium | |
| CN105530658A (en) | Remote diagnosis method of wireless communication module, device and system | |
| CN113794597A (en) | Alarm information processing method, system, electronic device and storage medium | |
| CN111935266A (en) | Data distribution system | |
| CN114979268B (en) | Data transmission method, device, server and system for industrial gas enterprise | |
| CN108989116B (en) | Big data collection and analysis system and method | |
| CN110908956A (en) | Information protection main station system and fault information filing method thereof | |
| CN113157475A (en) | Log processing method and device, storage medium and electronic equipment | |
| CN110134547B (en) | Middleware-based repeated data deleting method and related device | |
| CN109388546B (en) | Method, device and system for processing faults of application program | |
| CN113923272A (en) | Data analysis method and device and server equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20201113 |