CN109842623A - Big data distribution method for industry control safety database auditing system - Google Patents
Big data distribution method for industry control safety database auditing system Download PDFInfo
- Publication number
- CN109842623A CN109842623A CN201910088997.3A CN201910088997A CN109842623A CN 109842623 A CN109842623 A CN 109842623A CN 201910088997 A CN201910088997 A CN 201910088997A CN 109842623 A CN109842623 A CN 109842623A
- Authority
- CN
- China
- Prior art keywords
- data
- data packet
- server
- packet
- destination
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Abstract
The present invention provides a kind of big data distribution method for industry control safety database auditing system, needs to grab the network interface card of data including setting, and obtains and pass through the data packet that mirror-image fashion is sent into network interface card；Data packet IP layer data is parsed, obtains destination and source IP, and abandon non-ip protocol data packet；Data packet TCP layer and UDP layer are parsed, obtains destination and source end port number, and abandon non-TCP and udp data packet；Destination IP, source IP, destination port numbers and four data informations of source end port number in network packet provide the foundation for judging that data packet belongs to the same session, utilize data distribution algorithm, calculating data packet, which will be assigned in which process or thread, goes to handle, the reply data of the request data of client and server end is assigned in the same data processing process by guarantee to be handled, and data corruption caused by request and reply data are handled in different cpu process is avoided the occurrence of.
Description
Technical field
The present invention relates to technical field of information processing, more particularly to a kind of for industry control safety database auditing system
Big data distribution method.
Background technique
Under the network environment of data image, we often to guarantee to belong to a session network request packet frame and
Reply data frame can be distributed in identical process or thread and go to handle, with guarantee request of data and response uniformity and
Integrality, under this requirement, existing data distribution algorithm is unable to satisfy this requirement.We have proposed the distribution algorithms,
Pass through a series of operations using the four-tuple (destination IP, destination slogan, source IP, source port number) of TCP/IP network data frame,
It goes to handle to guarantee that request data and reply data can be distributed in the same process and thread.
Summary of the invention
The technical problems to be solved by the present invention are: in order to overcome the shortcomings in the prior art, the present invention provides a kind of use
In the big data distribution method of industry control safety database auditing system.
The present invention solves its technical problem technical solution to be taken: one kind is for industry control safety database audit system
The big data distribution method of system, including an at least WEB server, at least a database server, Data Audit server
With internal network interchanger, the WEB server is connect by internal network interchanger with database server, and main realization is outer
Portion requests inquiry of data etc., and the data distribution arithmetic server is connect with internal network interchanger, is used for data image,
The data copy portion of interaction between Web server and DB server is sent to data distribution arithmetic server；The WEB service
Network layer protocol is IP agreement between device and database server, and transport layer protocol is TCP or udp protocol；
It is further comprising the steps of:
S1: setting will grab the network interface card of data；
When user before operation data distribution algorithms, needs to configure data network card to be grabbed on data distribution arithmetic server
Parameter, therefore, it is necessary to users to input the title that grab the network interface card of data on data distribution arithmetic server, defeated in user
After the completion of entering, the data distribution program on data distribution arithmetic server will create raw socket (int on the network interface card
Socketfd=socket (PF_PACKET, SOCK_RAW, htons (ETH_P_ALL))), then it will be received on the network interface card
All data all receive in the socket file (recvfrom (and socketfd, buf, sizeof (buf), 0, NULL,
NULL), this method will receive all data that the network interface card is sent by mirror-image fashion on socket.The access of data in the present invention
Mode is mirror-image fashion access, and the data of mirror-image fashion access are that ensure that the request data of client and server end
Reply data has been sent in the dissemination system.
S2: parsing IP layer data obtains destination IP and source IP, and abandon non-ip protocol data packet；
The IP layer data of data distribution arithmetic analysis data packet in data distribution arithmetic server, judges network layer protocol
It whether is IP agreement, if it is not, then not handling the data packet and abandoning, because used between database and WEB server
Agreement is IP agreement, the data only transmitted between processing database server and WEB server here, therefore non-ip protocol data packet
It does not handle.After being judged as IP agreement, pointer (the struct iphdr*pIpHeader that IP agreement head is directed toward in data packet is obtained
=(struct iphdr*) buf), contain in the structural body IP agreement relevant field namely destination IP (dstip) and
Source IP (srcip).
S3: parsing TCP layer and UDP layer obtain the port numbers and source port number of destination, and abandon non-TCP and udp protocol
Data packet；
Judge whether transport layer protocol is TCP or udp protocol, factor according to the value of the protocol field of IP agreement
It is based on both agreements, for the data except both agreements according to the data transmission between library server and WEB server
Packet, will be dropped, disregards.
Obtain the TCP or udp protocol information (struct tcphdr*pTcpHeader=in data packet
(structtcphdr*) buf), which, which contains data, will be sent to the source end port number that destination port numbers and data originate,
Destination port numbers (dstport) and source end port number (srcport) are obtained with this.
S4: utilizing data distribution algorithm, which process will be calculating data packet, which will be assigned in, goes to handle；
Algorithm principle: for the data packet in network, by the request data package and response of the same session (session)
Data packet is placed in the same process or thread and goes to handle, source IP (srcip), destination IP in network packet
(dstip), four data informations of source end port number (srcport) and destination port numbers (destport) provide judgement data
Packet belongs to the foundation of the same session.This four elements constitute a four-tuple, and four-tuple is passed through certain logical AND and shifting
After bit manipulation operation, it is ensured that the operation result for belonging to the same session (session) is a unique value, with this operation
As a result judge that data packet is to belong to some session (session).
Concrete operations: logic " or (|) " operation, operation are done using source IP address (srcip) and destination IP (dstip)
As a result 16 (bit) are moved to right, result at this time is denoted as val1.It is patrolled using source end port value () and destination port value ()
" or (|) " operation is collected, operation result is denoted as val2, val1 and val2 are then done into logic " or (|) " operation, and utilize the fortune
Result is calculated as the final distribution factor and goes distribution data.
S5: data handling procedure；
Data handling procedure be more number of units according on audit server data processor or separate unit Data Audit service
The data handling procedure of one or more processes or thread on device；
If it is more number of units according to the data handling procedure on audit server, it will establish data distribution by function interface
Corresponding data packet and is sent to its corresponding data processing and taken by arithmetic server to the socket connection of Data Audit server
Business device；
If it is multiple processes or the data handling procedure of thread on separate unit Data Audit server, it will creation is multiple
Message queue, the quantity of message queue are equal to the quantity of process or thread, and data packet will distribute the factor according in step S4
The value of iMatchId is assigned in corresponding message queue, and each process or thread will be read from corresponding queue
Data.
The beneficial effects of the present invention are: a kind of big data for industry control safety database auditing system provided by the invention
Distribution method, the method achieve in massive dataflow, according to IP agreement four-tuple (source IP, destination IP, source port,
Purpose port) realize that the network packet for belonging to different sessions (session) is distributed to by the distribution of udp agreement to tcp
It goes to handle in different flow chart of data processing, it is ensured that distribute the reply data of the request data of client and server end
It is handled into the same data processing process, avoids the occurrence of request and reply data and handled in different cpu process and made
At data corruption.
Detailed description of the invention
Present invention will be further explained below with reference to the attached drawings and examples.
Fig. 1 is inventive network deployment architecture diagram.
Fig. 2 is the flow diagram of big flow network data distributing method of the present invention.
Specific embodiment
Presently in connection with attached drawing, the present invention is described in detail.This figure is simplified schematic diagram, is only illustrated in a schematic way
Basic structure of the invention, therefore it only shows the composition relevant to the invention.
As shown in Figure 1, a kind of big flow network data distribution systems for industrial control data security protection of the invention, packet
An at least WEB server, at least a database server, data distributing server, an at least number of units are included according to auditing service
Device and internal network interchanger, the WEB server carry out data friendship by external network interchanger, firewall and outer net equipment
It changes；The WEB server is connect by internal network interchanger with database server (DB server), main to realize that outside is right
The request such as inquiry of data, data distribution arithmetic server one end are connect with internal network interchanger, are used for data image,
The data copy portion of interaction between Web server and DB server is sent to data distribution arithmetic server, the other end and number
It is connected according to audit server, is used for data processing；Network layer protocol is IP association between the WEB server and database server
View, transport layer protocol are TCP or udp protocol.The mode of mirror image data ensure that client and all requests of server-side and
Reply data packet all copies the data distribution that a data have been sent in Fig. 1 by the format of data frame in a manner of backing up and calculates
On the network card equipment of method server, which is that hardware device " network switch (the internal network interchanger in Fig. 1) " is had
Some functions.Data packet in S1 is the resulting data frame of mirror image.
As shown in Fig. 2, a kind of big data distribution method for industry control safety database auditing system of the invention, including
Following steps:
S1: setting will grab the network interface card of data；
When user before operation data distribution algorithms, needs to configure data network card to be grabbed on data distribution arithmetic server
Parameter, therefore, it is necessary to users to input the title that grab the network interface card of data on data distribution arithmetic server, defeated in user
After the completion of entering, the data distribution program on data distribution arithmetic server will create raw socket (int on the network interface card
Socketfd=socket (PF_PACKET, SOCK_RAW, htons (ETH_P_ALL))), then it will be received on the network interface card
All data all receive in the socket file (recvfrom (and socketfd, buf, sizeof (buf), 0, NULL,
NULL), this method will receive all data that the network interface card is sent by mirror-image fashion on socket.The access of data in the present invention
Mode is mirror-image fashion access, and the data of mirror-image fashion access are that ensure that the request data of client and server end
Reply data has been sent in the dissemination system.
S2: parsing IP layer data obtains destination IP and source IP, and abandon non-ip protocol data packet；
The IP layer data of data distribution arithmetic analysis data packet in data distribution arithmetic server, judges network layer protocol
It whether is IP agreement, if it is not, then not handling the data packet and abandoning, because used between database and WEB server
Agreement is IP agreement, the data only transmitted between processing database server and WEB server here, therefore non-ip protocol data packet
It does not handle.After being judged as IP agreement, pointer (the struct iphdr*pIpHeader that IP agreement head is directed toward in data packet is obtained
=(struct iphdr*) buf), contain in the structural body IP agreement relevant field namely destination IP (dstip) and
Source IP (srcip).
S3: parsing TCP layer and UDP layer obtain the port numbers and source port number of destination, and abandon non-TCP and udp protocol
Data packet；
Judge whether transport layer protocol is TCP or udp protocol, factor according to the value of the protocol field of IP agreement
It is based on both agreements, for the data except both agreements according to the data transmission between library server and WEB server
Packet, will be dropped, disregards.
Obtain the TCP or udp protocol information (struct tcphdr*pTcpHeader=in data packet
(structtcphdr*) buf), which, which contains data, will be sent to the source end port number that destination port numbers and data originate,
Destination port numbers (dstport) and source end port number (srcport) are obtained with this.
S4: utilizing data distribution algorithm, which process will be calculating data packet, which will be assigned in, goes to handle；
Algorithm principle: for the data packet in network, by the request data package and response of the same session (session)
Data packet is placed in the same process or thread and goes to handle, source IP (srcip), destination IP in network packet
(dstip), four data informations of source end port number (srcport) and destination port numbers (destport) provide judgement data
Packet belongs to the foundation of the same session.This four elements constitute a four-tuple, and four-tuple is passed through certain logical AND and shifting
After bit manipulation operation, it is ensured that the operation result for belonging to the same session (session) is a unique value, with this operation
As a result judge that data packet is to belong to some session (session).
Concrete operations: logic " or (|) " operation, operation are done using source IP address (srcip) and destination IP (dstip)
As a result 16 (bit) are moved to right, result at this time is denoted as val1 by us.Use source end port value () and destination port value ()
Logic " or (|) " operation is done, operation result is denoted as val2, val1 and val2 are then done into logic " or (|) " operation, and utilize
The operation result goes distribution data as the final distribution factor.
S5: data handling procedure；
Data handling procedure be more number of units according on audit server data processor or separate unit Data Audit service
The data handling procedure of one or more processes or thread on device；
If it is more number of units according to the data handling procedure on audit server, it will establish data distribution by function interface
Corresponding data packet and is sent to its corresponding data processing and taken by arithmetic server to the socket connection of Data Audit server
Business device；
If it is multiple processes or the data handling procedure of thread on separate unit Data Audit server, it will creation is multiple
Message queue, the quantity of message queue are equal to the quantity of process or thread, and data packet will distribute the factor according in step S4
The value of iMatchId is assigned in corresponding message queue, and each process or thread will be read from corresponding queue
Data.
The method of the invention realizes in massive dataflow, according to IP agreement four-tuple (source IP, destination IP, source
Port, purpose port) it realizes to tcp, the distribution of udp agreement, the network packet point of different sessions (session) will be belonged to
It is dealt into different flow chart of data processing and goes to handle, it is ensured that by the reply data of the request data of client and server end
It is assigned in the same data processing process and is handled, avoid the occurrence of request and reply data in different cpu process
Data corruption caused by reason
Taking the above-mentioned ideal embodiment according to the present invention as inspiration, through the above description, relevant staff
Various changes and amendments can be carried out without departing from the scope of the present invention completely.The technical scope of this invention is not
The content being confined on specification, it is necessary to which the technical scope thereof is determined according to the scope of the claim.
Claims (6)
1. a kind of big data distribution method for industry control safety database auditing system, it is characterised in that: including at least one
WEB server, at least a database server, Data Audit server and internal network interchanger, the WEB server are logical
It crosses internal network interchanger to connect with database server, the Data Audit server is connect with internal network interchanger；Institute
Stating network layer protocol between WEB server and database server is IP agreement, and transport layer protocol is TCP or udp protocol；
It is further comprising the steps of:
S1: setting needs to grab the network interface card of data, and obtains all data packets that the network interface card is sent by mirror-image fashion；
The IP layer data of S2: the analyzing step S1 data packet obtained, obtains destination IP and source IP, and abandon non-ip protocol number
According to packet；
S3: parsing the TCP layer and UDP layer of data packet, obtains destination port numbers and source end port number, and abandon non-TCP and UDP
Protocol data packet；
S4: destination IP, source IP, destination port numbers and four data informations of source end port number in network packet are provided
Judge that data packet belongs to the foundation of the same session, four elements constitute a four-tuple, utilize data distribution algorithm, meter
Calculation data packet, which will be assigned in which process or thread, goes to handle.
2. being used for the big data distribution method of industry control safety database auditing system as described in claim 1, it is characterised in that:
The step S1 is specifically included: being determined the network interface card for needing to grab data, and is created raw socket on the network interface card, then by institute
There are the data for being sent into the network interface card by mirror-image fashion all to receive in the socket file, forms data packet.
3. being used for the big data distribution method of industry control safety database auditing system as described in claim 1, it is characterised in that:
The step S2 is specifically included: the IP layer data of the data distribution arithmetic analysis data packet in data distribution arithmetic server is sentenced
Whether the network layer protocol of disconnected data packet is IP agreement, if it is not, then not handling the data packet and abandoning；It is assisted if it is IP
View obtains the pointer that IP agreement head is directed toward in data packet, to obtain destination IP and source IP.
4. being used for the big data distribution method of industry control safety database auditing system as described in claim 1, it is characterised in that:
The step S3 is specifically included: judged according to the value of the protocol field of IP agreement transport layer protocol whether be TCP or
Udp protocol then abandons the data packet if not both agreements, if so, obtaining the TCP or udp protocol in data packet
Information, the information contain the source end port number that data will be sent to the port numbers of destination and data originate, obtain purpose with this
Hold port numbers and source end port number.
5. being used for the big data distribution method of industry control safety database auditing system as described in claim 1, it is characterised in that:
Data distribution algorithm is specially to do logical "or" operation using source IP and destination IP in the step S4, and operation result moves to right
16, result at this time is denoted as val1；Logical "or" operation is done using source end port value and destination port value, by operation knot
Fruit is denoted as val2；Then val1 and val2 are done into logical "or" operation, and using the operation result as the final distribution factor
Go distribution data.
6. the big data distribution method as described in any one in claim 1-5 for industry control safety database auditing system,
It is characterized in that: further including S5: data handling procedure；
Data handling procedure is more number of units according on the data processor or separate unit Data Audit server on audit server
One or more processes or thread data handling procedure；
If it is more number of units according to the data handling procedure on audit server, it will establish data distribution algorithm by function interface
Corresponding data packet and is sent to its corresponding data processing service to the socket connection of Data Audit server by server
Device；
If it is multiple processes or the data handling procedure of thread on separate unit Data Audit server, it will create multiple message
Queue, the quantity of message queue are equal to the quantity of process or thread, and data packet will distribute the factor according in step S4
The value of iMatchId is assigned in corresponding message queue, and each process or thread will be read from corresponding queue
Data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910088997.3A CN109842623A (en) | 2019-01-30 | 2019-01-30 | Big data distribution method for industry control safety database auditing system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910088997.3A CN109842623A (en) | 2019-01-30 | 2019-01-30 | Big data distribution method for industry control safety database auditing system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109842623A true CN109842623A (en) | 2019-06-04 |
Family
ID=66884396
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910088997.3A Pending CN109842623A (en) | 2019-01-30 | 2019-01-30 | Big data distribution method for industry control safety database auditing system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109842623A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111817979A (en) * | 2020-06-23 | 2020-10-23 | 成都深思科技有限公司 | Multi-dimensional flow association data packet processing method based on sniffing mode |
| CN113691608A (en) * | 2021-08-20 | 2021-11-23 | 京东科技信息技术有限公司 | Traffic distribution method, traffic distribution device, electronic equipment and media |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1294159A2 (en) * | 2001-09-13 | 2003-03-19 | PacketLight Networks Ltd. | Method for assigning network addresses |
| CN104063473A (en) * | 2014-06-30 | 2014-09-24 | 江苏华大天益电力科技有限公司 | Database auditing monitoring system and database auditing monitoring method |
| CN105681317A (en) * | 2016-02-03 | 2016-06-15 | 国网智能电网研究院 | Novel business and database auditing engine |
| CN106850319A (en) * | 2017-03-27 | 2017-06-13 | 广州供电局有限公司 | The collecting method and system of power network EMS system |
| CN107357681A (en) * | 2017-06-26 | 2017-11-17 | 杭州铭师堂教育科技发展有限公司 | Zookeeper backup management systems and method based on salt |
| CN108696537A (en) * | 2018-07-09 | 2018-10-23 | 武汉斗鱼网络科技有限公司 | Long connection network data distribution and processing method and system in IOS systems |
-
2019
- 2019-01-30 CN CN201910088997.3A patent/CN109842623A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1294159A2 (en) * | 2001-09-13 | 2003-03-19 | PacketLight Networks Ltd. | Method for assigning network addresses |
| CN104063473A (en) * | 2014-06-30 | 2014-09-24 | 江苏华大天益电力科技有限公司 | Database auditing monitoring system and database auditing monitoring method |
| CN105681317A (en) * | 2016-02-03 | 2016-06-15 | 国网智能电网研究院 | Novel business and database auditing engine |
| CN106850319A (en) * | 2017-03-27 | 2017-06-13 | 广州供电局有限公司 | The collecting method and system of power network EMS system |
| CN107357681A (en) * | 2017-06-26 | 2017-11-17 | 杭州铭师堂教育科技发展有限公司 | Zookeeper backup management systems and method based on salt |
| CN108696537A (en) * | 2018-07-09 | 2018-10-23 | 武汉斗鱼网络科技有限公司 | Long connection network data distribution and processing method and system in IOS systems |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111817979A (en) * | 2020-06-23 | 2020-10-23 | 成都深思科技有限公司 | Multi-dimensional flow association data packet processing method based on sniffing mode |
| CN113691608A (en) * | 2021-08-20 | 2021-11-23 | 京东科技信息技术有限公司 | Traffic distribution method, traffic distribution device, electronic equipment and media |
| CN113691608B (en) * | 2021-08-20 | 2024-02-06 | 京东科技信息技术有限公司 | Traffic distribution method, device, electronic equipment and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8789135B1 (en) | Scalable stateful firewall design in openflow based networks | |
| US8149705B2 (en) | Packet communications unit | |
| JP2003258890A (en) | Method and system for controlling data transfer between client computer system and internet network | |
| US20080162690A1 (en) | Application Management System | |
| US20190215306A1 (en) | Rule processing and enforcement for interleaved layer 4, layer 7 and verb based rulesets | |
| US20150071085A1 (en) | Network gateway for real-time inspection of data frames and identification of abnormal network behavior | |
| CN109842623A (en) | Big data distribution method for industry control safety database auditing system | |
| CN101217494A (en) | A method and network device for operation flow monitoring | |
| KR20100015790A (en) | Device for coalescing messages and method thereof | |
| JP2006512662A (en) | Quality of service for iSCSI | |
| CN105323310A (en) | Network communication method, device and network attached storage device | |
| CN106230870A (en) | Proprietary protocol document transmission system and method | |
| CN105847179B (en) | The method and device that Data Concurrent reports in a kind of DPI system | |
| CN105282128A (en) | Bidirectional calling method and system based on long connection | |
| US11431677B2 (en) | Mechanisms for layer 7 context accumulation for enforcing layer 4, layer 7 and verb-based rules | |
| US20200280506A1 (en) | Enhancing Transmission Control Protocol (TCP) Performance and Scalability on Multicore Processor Architectures | |
| CN101355585B (en) | System and method for protecting information of distributed architecture data communication equipment | |
| US20040148417A1 (en) | Method and system for distinguishing higher layer protocols of the internet traffic | |
| CN106572103A (en) | Hidden port detection method based on SDN network architecture | |
| CN101854366B (en) | Peer-to-peer network flow-rate identification method and device | |
| CN108055273A (en) | A kind of intranet server finds method, system and Network Security Audit System | |
| CN112291076A (en) | Packet loss positioning method, device and system and computer storage medium | |
| CN103227781A (en) | Network diagnose and performance evaluation system and method based on user datagram protocol | |
| EP1575236A1 (en) | Connectivity confirmation method for network storage device and host computer | |
| KR100439732B1 (en) | Apparatus and method of verifying fair racing using QoS measuring system in the client-server network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: Jinhu innovation center, No.8 Taihu Middle Road, Xinbei District, Changzhou City, Jiangsu Province, 213022 Applicant after: CHANGZHOU EVERFORT DATA TECHNOLOGY Co.,Ltd. Address before: 213003 Jinhu innovation center, No.8 Taihu Middle Road, Xinbei District, Xuzhou City, Jiangsu Province Applicant before: CHANGZHOU EVERFORT DATA TECHNOLOGY Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190604 |
|
| RJ01 | Rejection of invention patent application after publication |