CN110460599B - Method for realizing one-way transmission based on multi-level cache and main/standby mechanism - Google Patents
Method for realizing one-way transmission based on multi-level cache and main/standby mechanism Download PDFInfo
- Publication number
- CN110460599B CN110460599B CN201910742496.2A CN201910742496A CN110460599B CN 110460599 B CN110460599 B CN 110460599B CN 201910742496 A CN201910742496 A CN 201910742496A CN 110460599 B CN110460599 B CN 110460599B
- Authority
- CN
- China
- Prior art keywords
- file
- receiving
- main
- standby
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0654—Management of faults, events, alarms or notifications using network fault recovery
- H04L41/0663—Performing the actions predefined by failover planning, e.g. switching to standby network elements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0805—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters by checking availability
- H04L43/0817—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters by checking availability by checking functioning
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/10—Active monitoring, e.g. heartbeat, ping or trace-route
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L49/00—Packet switching elements
- H04L49/90—Buffering arrangements
- H04L49/9015—Buffering arrangements for supporting a linked list
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/16—Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
- H04L69/164—Adaptation or special uses of UDP protocol
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/104—Peer-to-peer [P2P] networks
- H04L67/1074—Peer-to-peer [P2P] networks for supporting data block transmission mechanisms
- H04L67/1078—Resource delivery mechanisms
- H04L67/108—Resource delivery mechanisms characterised by resources being split in blocks or fragments
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Environmental & Geological Engineering (AREA)
- Health & Medical Sciences (AREA)
- Cardiology (AREA)
- General Health & Medical Sciences (AREA)
- Computer Security & Cryptography (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention relates to the technical field of data transmission, in particular to a method for realizing one-way transmission based on multi-level cache and a main and standby mechanism. According to the invention, on the basis of adopting a one-way transmission component, a multi-level memory caching mode is utilized, and a ferrying and proxy technology of a traditional safety isolation technology is combined, so that the information is transmitted and controlled in real time to the maximum extent while the one-way transmission of the information is ensured, a memory layout mode is optimized by utilizing a direct memory and combining technical means such as an array, a pointer, a hash, a linked list, a distributed mode, a main standby mode and the like, the problems of multi-thread random reading and overtime memory overflow are solved, and meanwhile, a distributed thread pool is used for filtering data, so that the data processing capacity is improved.
Description
Technical Field
The invention relates to the technical field of data transmission, in particular to a method for realizing one-way transmission based on a multi-level cache and a main/standby mechanism.
Background
With the rapid development of information technology, the life and working modes of people are greatly changed, the large-scale application of information networks greatly improves the office efficiency, China has built a digital network with a considerable scale after years of construction, but with the continuous popularization of networks, the security problems are increased day by day, the network and information security problems become major hidden dangers threatening the security of the country and the government, with the continuous understanding and understanding of the security problems, especially for the protection of secret-related information, China has already mentioned the information security construction to a considerable height, the connection of an internal secret-related network and an external public network brings serious security problems, such as virus destruction, hacker intrusion and the like, while the firewall, the proxy server, intrusion detection and other security products are realized by logical isolation based on software, and can not meet the physical isolation security requirements between a high-level network department and other networks, therefore, how to control the data flow from the physical layer becomes a problem to be solved urgently.
When data exchange is carried out between a high-security network and a low-security network, how to ensure the self-security of the high-security network and prevent information leakage is very important, when the one-way transmission function of a file is realized, gigabit network transmission and a UDP (user Datagram protocol) transmission protocol are generally used, because the UDP protocol is a connectionless protocol, the reliability is low, and through the traditional technical means, the mutual parameters are asymmetric due to the fact that the UDP protocol is excessively dependent on the conditions of network speed, hard disk speed, CPU dominant frequency and the like, so that the problems of data processing blockage, data loss and the like on one side can be caused.
Disclosure of Invention
Technical problem to be solved
Aiming at the defects of the prior art, the invention provides a method for realizing one-way transmission based on multi-level cache and a main/standby mechanism, which has the advantages of real-time transmission and controllability of information and solves the problems of data processing blockage and data loss on one part caused by asymmetry of parameters caused by excessive dependence on conditions such as network speed, hard disk speed, CPU (central processing unit) main frequency and the like in the traditional technical means.
(II) technical scheme
In order to realize the technical problem, the invention provides a method for realizing one-way transmission based on multi-level cache and a main and standby mechanism, which comprises a sending server and a main and standby receiving server, wherein the sending server divides a file into small sections, codes a section serial number, assembles each small section into a UDP data packet, fills a file name, a file MD5 value, a serial number and the total packet number, and respectively sends the file name, the file MD5 value, the serial number and the total packet number to the main and standby receiving servers through two network cards;
the processing flow of the main and standby receiving servers is as follows:
s1, the master and backup receiving server receives the data sent by the sending server;
s2, formatting main and standby receiving server memories, allocating 4G space memories to be used for caching by each server, formatting the 4G memory spaces according to a structure, and compiling a structure address, wherein the structure comprises a file name, a file MD5, a sequence number, the total packet number and time;
s3, creating a receiving process, wherein the main receiving server and the standby receiving server respectively create the receiving process for receiving data from the sending server;
s4, after receiving UDP data packets, according to the file MD5 value and packet sequence number of each packet, calculating a hash value, namely the corresponding address of the pre-formatted memory, if the address is used, using the next address, if 10 continuous addresses are used, creating a dynamic linked list on the 10 th address;
s5, the receiving process creates a thread after receiving the first data packet of each file;
s6, queue log processes are respectively started on two servers, work tasks are the same, contents are completely synchronous, the two processes judge the main process and the standby process in a heartbeat mode, when the main process is abnormal, the standby process immediately raises the main process, and unfinished work is continued;
and S7, after the file is accepted, firstly, checking whether the processed same file name and MD5 exist in the log, directly deleting the file name and the MD5, and writing the file name and the MD5 into the queue after the log is not written.
And S8, the two filtering thread pools are used for reading the file which is received and needs to be filtered by the de-queue log process, and writing the file into the region to be read after performing antivirus, keyword filtering, feature code matching and other work, so that the forwarding of one file is completed.
Further, in the process of step S4, when the corresponding memory receives data, the UDP buffer does not overflow, and the UDP buffer is directly expanded to a hundred megabytes or more.
Further, in step S5, the thread first creates a file, which has only the read/write permission of the designated user and no executable permission, and thus prevents unexpected execution. And the thread calculates the memory address according to the MD5 value and the sequence number of the file, receives the data packets one by one in sequence, and writes the data packets into the queue log process after all the data packets are completely received and the MD5 value is calculated to be correct. If one data packet is overtime, the file stops receiving, the thread exits, the resource is recycled, the data written into the memory is overtime, and the next writing is not influenced.
(III) advantageous effects
The invention provides a method for realizing one-way transmission based on a multi-level cache and a main/standby mechanism, which has the following beneficial effects:
1. according to the method for realizing the one-way transmission based on the multi-level cache and the main and standby mechanism, the requirement of one-way isolation can be met in the aspects of realizing one-way control, high efficiency, stability and economy in the aspect of optical fiber transmission, meanwhile, on the basis of adopting a one-way transmission component, the real-time transmission and the controllability of information are realized to the maximum extent while the one-way transmission of the information is ensured by utilizing a multi-level memory cache mode and combining a ferrying and proxy technology of the traditional safety isolation technology.
2. The method for realizing one-way transmission based on the multi-level cache and the main and standby mechanisms optimizes a memory layout mode by utilizing a direct memory and combining technical means such as an array, a pointer, a hash, a linked list, a distributed mode, a main and standby mode and the like, solves the problems of multithreading random reading and overtime memory overflow, increases the main and standby modes, reduces an emergency processing mechanism when unilateral faults and data are abnormal, filters data by using a distributed thread pool in the process, and improves the data processing capacity.
Drawings
FIG. 1 is a flow chart of the structure of the present invention;
FIG. 2 is a flow chart of the process of the primary/standby receiving server according to the present invention
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1-2, a method for implementing unidirectional transmission based on multi-level cache and a primary/secondary mechanism includes a sending server and a primary/secondary receiving server, where the sending server divides a file into segments, codes segment numbers, assembles each segment into a UDP packet, fills a file name, a file MD5 value, a sequence number, and a total packet number, and sends the UDP packet to the primary/secondary receiving server through two network cards.
The processing flow of the main and standby receiving servers is as follows:
and S1, the master receiving server receives the data sent by the sending server.
S2, formatting main and standby receiving server memories, allocating 4G space memories for caching by each server, formatting the 4G memory spaces according to a structure, and compiling a structure address, wherein the structure comprises a file name, a file MD5, a sequence number, the total packet number, time and related marks.
And S3, creating a receiving process, wherein the main receiving server and the standby receiving server respectively create the receiving process for receiving the data from the sending server.
S4, after receiving UDP data packets, according to the MD5 value and packet sequence number of each packet, calculating a hash value, namely the corresponding address of the pre-formatted memory, if the address is used, using the next address, if 10 continuous addresses are used, creating a dynamic linked list on the 10 th address, thus writing the data into the corresponding memory, avoiding overflow of UDP buffer, and directly expanding the UDP buffer area to more than hundred megabytes.
S5, the receiving process creates a thread after receiving the first data packet of each file, the thread first creates a file, the file only has the read-write permission of the designated user, but has no executable permission, and the accidental execution is prevented. And the thread calculates the memory address according to the MD5 value and the sequence number of the file, receives the data packets one by one in sequence, and writes the data packets into the queue log process after all the data packets are completely received and the MD5 value is calculated to be correct. If one data packet is overtime, the file stops receiving, the thread exits, the resource is recycled, the data written into the memory is overtime, and the next writing is not influenced.
And S6, the queue log processes are respectively started on the two servers, the work tasks are the same, the contents are completely synchronous, the two processes judge the main process and the standby process in a heartbeat mode, and when the main process is abnormal, the standby process immediately raises the main process and continues unfinished work.
And S7, after the file is accepted, firstly, checking whether the processed same file name and MD5 exist in the log, directly deleting the file name and the MD5, and writing the file name and the MD5 into the queue after the log is not written.
And S8, the two filtering thread pools are used for reading the file which is received and needs to be filtered by the de-queue log process, and writing the file into the region to be read after performing antivirus, keyword filtering, feature code matching and other work, so that the forwarding of one file is completed.
In summary, according to the method for realizing unidirectional transmission based on the multi-level cache and the main/standby mechanism, the optical fiber transmission can meet the requirement of unidirectional isolation in the aspects of realizing unidirectional control, high efficiency, stability and economy, and meanwhile, on the basis of adopting a unidirectional transmission component, the real-time transmission and the controllability of information are realized to the maximum extent while the unidirectional transmission of the information is ensured by utilizing a multi-level memory cache mode and combining a ferry-ferrying-plus-proxy technology of the traditional safety isolation technology.
The method for realizing one-way transmission based on the multi-level cache and the main and standby mechanisms optimizes a memory layout mode by utilizing a direct memory and combining technical means such as an array, a pointer, a hash, a linked list, a distributed mode, a main and standby mode and the like, solves the problems of multithreading random reading and overtime memory overflow, increases the main and standby modes, reduces an emergency processing mechanism when unilateral faults and data are abnormal, filters data by using a distributed thread pool in the process, and improves the data processing capacity.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.
Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Claims (3)
1. A method for realizing one-way transmission based on multi-level cache and a main/standby mechanism is characterized in that: the method comprises the steps that a file is divided into small sections by the sending server, the serial numbers of the small sections are coded, each small section is assembled into a UDP data packet, the file name, the MD5 value, the serial numbers and the total packet number are filled, and the file name, the MD5 value, the serial numbers and the total packet number are respectively sent to the main receiving server and the standby receiving server through two network cards;
the processing flow of the main and standby receiving servers is as follows:
s1, the master and backup receiving server receives the data sent by the sending server;
s2, formatting main and standby receiving server memories, allocating 4G space memories to be used for caching by each server, formatting the 4G memory spaces according to a structure, and compiling a structure address, wherein the structure comprises a file name, a file MD5, a sequence number, the total packet number and time;
s3, creating a receiving process, wherein the main receiving server and the standby receiving server respectively create the receiving process for receiving data from the sending server;
s4, after receiving UDP data packets, according to the file MD5 value and packet sequence number of each data packet, calculating a hash value, namely the address of the corresponding pre-formatted memory, if the address is used, using the next address, if 10 continuous addresses are used, creating a dynamic linked list on the 10 th address;
s5, the receiving process creates a thread after receiving the first data packet of each file;
s6, queue log processes are respectively started on two servers, work tasks are the same, contents are completely synchronous, the two processes judge the main process and the standby process in a heartbeat mode, when the main process is abnormal, the standby process immediately raises the main process, and unfinished work is continued;
s7, after the file is received, firstly checking whether the processed same file name and MD5 exist in the log, if so, directly deleting the file name and MD5 without writing the log, and then writing the file into the queue;
and S8, two filtering thread pools, wherein the file which is received and needs to be filtered is taken by the de-queue log process, and the file is sterilized, filtered by key words, written into the region to be taken after the matching work of the feature codes, and the forwarding of one file is completed.
2. The method according to claim 1, wherein the method comprises: in the process of step S4, when the corresponding memory receives data, the UDP buffer does not overflow, and the UDP buffer is directly expanded to a hundred megabytes space.
3. The method according to claim 1, wherein the method comprises: in step S5, the thread first creates a file, which only has the read-write permission of the designated user and no executable permission, and prevents unexpected execution; the thread calculates the memory address according to the MD5 value and the serial number of the file, receives the data packets one by one in sequence, and writes the data packets into a queue log process after all the data packets are completely received and the MD5 value is calculated to be correct; if one data packet is overtime, the file stops receiving, the thread exits, the resource is recycled, the data written into the memory is overtime, and the next writing is not influenced.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910742496.2A CN110460599B (en) | 2019-08-13 | 2019-08-13 | Method for realizing one-way transmission based on multi-level cache and main/standby mechanism |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910742496.2A CN110460599B (en) | 2019-08-13 | 2019-08-13 | Method for realizing one-way transmission based on multi-level cache and main/standby mechanism |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110460599A CN110460599A (en) | 2019-11-15 |
| CN110460599B true CN110460599B (en) | 2021-11-09 |
Family
ID=68486132
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910742496.2A Active CN110460599B (en) | 2019-08-13 | 2019-08-13 | Method for realizing one-way transmission based on multi-level cache and main/standby mechanism |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110460599B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110912669A (en) * | 2019-11-15 | 2020-03-24 | 珠海市新德汇信息技术有限公司 | Two-channel data transmission method based on file stream and data stream |
| CN112003992A (en) * | 2020-08-14 | 2020-11-27 | 迅镭智能（广州）科技有限公司 | Transmission system and method based on scanning gun |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101989916A (en) * | 2009-08-04 | 2011-03-23 | 西安交大捷普网络科技有限公司 | Separating multi-stage buffer network content filtering system and method |
| CN102035843A (en) * | 2010-12-17 | 2011-04-27 | 北京锐安科技有限公司 | System and method for transmitting data in one direction |
| US8589953B1 (en) * | 2010-08-06 | 2013-11-19 | Open Invention Network, Llc | System and method for transparent consistent application-replication of multi-process multi-threaded applications |
| CN105656902A (en) * | 2016-01-15 | 2016-06-08 | 深圳市利谱信息技术有限公司 | One-way reliable transmission and control system based on light transmission |
| CN106850156A (en) * | 2016-11-28 | 2017-06-13 | 深圳市鑫之淼科技有限公司 | No-feedback one-way data transmission set and transmission method based on network interface |
| CN106992856A (en) * | 2017-03-29 | 2017-07-28 | 山西大学 | The data coordinating method of extensive continuous variable quantum key distribution based on GPU |
-
2019
- 2019-08-13 CN CN201910742496.2A patent/CN110460599B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101989916A (en) * | 2009-08-04 | 2011-03-23 | 西安交大捷普网络科技有限公司 | Separating multi-stage buffer network content filtering system and method |
| US8589953B1 (en) * | 2010-08-06 | 2013-11-19 | Open Invention Network, Llc | System and method for transparent consistent application-replication of multi-process multi-threaded applications |
| CN102035843A (en) * | 2010-12-17 | 2011-04-27 | 北京锐安科技有限公司 | System and method for transmitting data in one direction |
| CN105656902A (en) * | 2016-01-15 | 2016-06-08 | 深圳市利谱信息技术有限公司 | One-way reliable transmission and control system based on light transmission |
| CN106850156A (en) * | 2016-11-28 | 2017-06-13 | 深圳市鑫之淼科技有限公司 | No-feedback one-way data transmission set and transmission method based on network interface |
| CN106992856A (en) * | 2017-03-29 | 2017-07-28 | 山西大学 | The data coordinating method of extensive continuous variable quantum key distribution based on GPU |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110460599A (en) | 2019-11-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR102226257B1 (en) | Method and device for writing service data to a blockchain system | |
| US20200265009A1 (en) | Blockchain-based data migration method and apparatus | |
| CN107391628B (en) | Data synchronization method and device | |
| CN110460599B (en) | Method for realizing one-way transmission based on multi-level cache and main/standby mechanism | |
| CN107391758B (en) | Database switching method, device and equipment | |
| JP6170512B2 (en) | Mediation of disk ownership in a storage pool | |
| EP2863310B1 (en) | Data processing method and apparatus, and shared storage device | |
| WO2020038105A1 (en) | Blockchain-based data storage method and apparatus | |
| WO2015158194A1 (en) | Dns security system and method thereof for processing a failure | |
| TW201800967A (en) | Method and device for processing distributed streaming data | |
| EP4332870A1 (en) | Transaction data processing method and apparatus, computer device and storage medium | |
| WO2019160128A1 (en) | Method for validating transaction in blockchain network and node for configuring same network | |
| CN106155839B (en) | A kind of method and apparatus for Backup Data | |
| CN103685304A (en) | Method and system for sharing session information | |
| US20160026699A1 (en) | Method for Synchronization of UGC Master and Backup and System Thereof, and Computer Storage Medium | |
| CN110474870B (en) | Block chain-based network active defense method and system and computer readable storage medium | |
| WO2015149629A1 (en) | Dns behavior processing method, device and system | |
| WO2021012932A1 (en) | Transaction rollback method and device, database, system, and computer storage medium | |
| CN113568981B (en) | Transaction data processing method, device, equipment and medium | |
| KR20190130933A (en) | Method and apparatus for constructing data based blockchain | |
| CN111935211A (en) | Method, equipment and medium for improving main chain performance by adopting side chain based on block chain | |
| JP6467540B1 (en) | Method for verifying transactions in a blockchain network and nodes for configuring the network | |
| Wang et al. | Data Security Storage Model of the Internet of Things Based on Blockchain. | |
| CN112765137B (en) | Block synchronization method based on block distributed block chain and electronic equipment | |
| CN117591038A (en) | Data access method, device, distributed storage system, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |