RU2694585C1 - Method of creating a secure l2-connection between packet switched networks - Google Patents

Abstract

FIELD: computer equipment.

SUBSTANCE: method comprises steps of: generating key information for all crypto routers; selecting in each network ranges of IP addresses for installed devices, as well as ranges of IP addresses which are tunneled by crypto routers; key information is loaded onto crypt router devices; configuring on the crypto-router IP-addresses; configuring on the crypto-routers allocated ranges of tunneled IP-addresses; creating on each switch two virtual ports: a port for encapsulating / decapsulating network packets into tunneling protocol headers; L3 level port serving as tunnel end IP interface; configuring all IP address switches on network interfaces; setting IP address of controller on switches; creating on the controller a configuration file containing values of the following constants: timeout of rules in flow tables, statistics request interval, duration of one iteration of statistics collection.

EFFECT: possibility of creating a cluster of crypto routers without restrictions on their number.

1 cl, 1 dwg, 10 tbl

Description

The technical field to which the invention relates.

The alleged invention relates to the field of network technologies, software-configured networks and cryptographic protection of information.

The level of technology

To protect communication channels between networks or data processing centers (DPCs), means of encrypting network traffic are used. At the same time, to ensure the passage of data through the global open network segments, encoders are used that operate at the network (L3) level of the ISO / OSI reference model and are able to route encrypted traffic. Such devices are usually called cryptographic routers or crypto-routers. Since crypto routers lose part of their bandwidth due to the need to encrypt and decrypt traffic, scaling technologies of secure communication channels are used to increase the bandwidth of the communication channel and its fault tolerance.

Also relevant is the task of creating a secure connection between networks at the data link layer (L2) of the ISO / OSI reference model, which allows you to combine unrelated local networks into one local network and ensures free flow of data link traffic (ARP, LLDP, etc.), migration of hosts and others. For this, tunneling protocols such as GRE and VXLAN are used.

A known method of load balancing using software-configured networks, which is an integral part of the method (RF patent No. 2576488, priority from 02/17/2015), according to which the transit network, built on the basis of software-configured networks technology (PKS), includes the following components: OpenFlow switches (barrier switches), traffic verification servers coming from outside the network, PKS controller, authentication server. The transit network is an intermediate of packet transmission between user terminals and protected servers.

The structure of the controller includes the following units: a statistics collection unit, a presence control unit, a routing rule generation unit, an address rewriting unit, a switching unit. Load balancing is carried out on the basis of load statistics, which is collected by the statistics collection unit by polling barrier switches and verification servers at equal specified intervals. The statistical data received by the controller from the barrier switches are the values of the incoming packet counters, and from the verification servers, the total processor load and RAM consumption. Based on this data, methods of equal load distribution are implemented, the block of routing rules generation generates routing rules and sends them to barrier switches. The presence control unit is responsible for tracking the appearance of new switches and verification servers in the transit network and putting idle verification servers into sleep mode. The block implements the strategy of maximizing the use of resources, the meaning of which is to condense the tasks of verifying packages on the verification servers and removing idle equipment from the process. The decision to transfer to “sleep” mode is made on the basis of statistics received from the statistics collection unit. The address reassignment block is used during network initialization and is responsible for sending to the barrier switches commands to change its IP address.

Barrier switches are managed switches that support the OpenFlow 1.3 protocol and include the following main components: the routing rules repository, the packet control block, the command processing block from the controller, the address remapping block. The main task of switches is to distribute incoming packets to the verification servers based on the rules contained in the repository of routing rules. The command processing block is central in the switch structure and is intended for processing control commands from the PKS controller.

The principle of operation of the transit network is as follows. The user terminal sends the network packet to one of the barrier switches. Suppose that there is no rule for processing packets from a given user terminal among the barrier switch routing rules. As a result, the switch sends the packet header to the controller. The packet itself is stored on the barrier switch, awaiting control commands from the controller. The latter accepts the packet and, based on the download statistics of the verification servers, generates a routing rule, according to which all packets coming from this user terminal will be redirected to the corresponding verification server. This rule is transmitted through the control channel to the switch. After that, the packet packets stored in the switch cache are sent to the selected verification server. Next, the packet is analyzed on the verification server and, if it is legitimate, the packet is further routed to the target protected server. Reverse traffic from the server is returned to the user terminal along the same route.

The described method has the following disadvantages:

- the method cannot be applied to create a secure L2 connection between networks. In particular, due to the lack of support for the GRE, VXLAN tunneling protocols;

- the method cannot be applied to crypto-gateways with different key sets and, if necessary, send the encrypted packet to the correct crypto-gateway for decryption.

There is also a known method of connecting several data centers at the L2 level using secure high-performance communication channels using L2OverIP technology (article at: https://www.anti-malware.ru/analytics/Technology_Analysis/ViPNet_Coordinator_HW).

The technology of building a secure network L20verIP allows you to organize a secure exchange for distributed local networks using a single address space at the data link layer (L2) of the OSI network model. The technology supports the exchange of data of any network layer protocols via an encrypted IP connection. Using L2OverIP, nodes from different network segments can interact with each other as if they were in the same local network segment with direct visibility by MAC addresses.

The scheme for channel aggregation uses EtherChannel technology, which is supported in most modern switches and ensures the scalability of the encrypted channel. The performance of the aggregated channel can be increased without turning it off by adding new physical channels.

The main components of the scheme in each data center are: the switch on which the channel aggregation is performed; a cluster of crypto routers with L2OverIP technology support; boundary network routers. The traffic coming from the data center is distributed over the links of the aggregated channel, encrypted by a cluster of cryptographic routers and transmitted through the border routers of the network to the open network. In the data center where the traffic recipient is located, the reverse procedure is performed.

This method is adopted as a prototype. However, the known method has the following disadvantages:

- EtherChannel technology has limitations on the number of aggregated channels and, therefore, on the number of connected cryptographic routers;

- aggregated physical channels must be of the same performance, which excludes the possibility of using cryptographic routers with network interfaces of different performance;

- the specified L2OverIP technology is supported only on identical crypto routers (produced, in particular, by Info-TeKS), which excludes the possibility of using crypto-routers from different manufacturers in the scheme;

- the method cannot be applied if it is necessary to migrate hosts between data centers without reconfiguring network equipment.

DISCLOSURE OF INVENTION

The technical result is

1) ensuring the possibility of creating a cluster of crypto routers without restrictions on their number, performance and type;

2) enabling the migration of end devices between connected networks without the need to reconfigure network devices.

To this end, a method is proposed for creating a secure L2 connection between packet-switched networks, with the following networks installed:

- a controller connected to all networks and configured to:

формировать таблицы потоков, содержащие правила обработки сетевых пакетов;

create flow tables containing the rules for processing network packets;

обмениваться управляющими сообщениями с коммутаторами по установленному протоколу управления;

exchange control messages with switches according to the established control protocol;

запрашивать, получать и анализировать информацию от других сетевых устройств;

request, receive and analyze information from other network devices;

- switches, each of which is installed in its network, is connected to it and configured to:

принимать и посылать сетевые пакеты;

receive and send network packets;

обмениваться управляющими сообщениями с контроллером по установленному протоколу управления;

exchange control messages with the controller according to the established control protocol;

хранить в оперативной памяти, по крайней мере, три таблицы потоков и выполнять обработку сетевых пакетов по этим таблицам;

store in memory at least three stream tables and perform network packet processing on these tables;

обрабатывать пакеты при помощи заданной таблицы потоков;

process packets using the specified stream table;

создавать виртуальные порты с поддержкой установленного протокола туннелирования;

create virtual ports with the support of the installed tunneling protocol;

- cryptographic routers (hereinafter referred to as crypto routers), each of which has at least one processor; moreover, each network has at least two cryptoMarsh-routizers connected to its switchboard; each crypto-Russian rutizator is configured to:

принимать и посылать сетевые пакеты;

receive and send network packets;

загружать ключевую информацию;

upload key information;

настраивать диапазоны туннелируемых IP-адресов;

configure tunneling IP address ranges;

шифровать, расшифровывать и маршрутизировать сетевые пакеты;

encrypt, decrypt, and route network packets;

отвечать на запросы информации от контроллера;

respond to requests for information from the controller;

- workstations installed and connected each to its network and made with the ability to exchange network packets with each other;

the way is that:

- generate key information for all cryptographic routers;

- allocate in each network ranges of GR-addresses for installed devices, as well as ranges of cryptographic routers IP-addresses;

- include crypto routers;

- key information is uploaded to crypto routers;

- set up IP-addresses on crypto-routers;

- set up dedicated ranges of tunneling IP addresses on crypto routers;

- include switches;

- create on each switch two virtual ports:

порт для инкапсуляции/декапсуляции сетевых пакетов в заголовки протокола туннелирования (далее туннельный порт);

port for encapsulation / decapsulation of network packets in the tunneling protocol headers (hereinafter the tunnel port);

порт уровня L3, служащий конечным IP-интерфейсом туннеля (далее сетевой интерфейс);

port level L3, which serves as the final IP-interface of the tunnel (hereinafter the network interface);

- configure on the network interfaces of all switches IP-addresses;

- configure the switch IP address of the controller;

- include the controller;

- create on the controller a configuration file containing the values of the following constants: inactivity timeout of rules in the flow tables, statistics request interval, duration of one iteration of statistics collection;

- perform on the controller the registration and initialization of switches, during which three flow tables with the initial set of rules are loaded onto each switch:

таблица 0 - предназначена для распределения по другим таблицам сетевых пакетов;

table 0 - intended for distribution to other tables of network packets;

таблица 1 - предназначена для обработки исходящих с коммутатора сетевых пакетов;

Table 1 is designed to handle outgoing network packets from the switch;

таблица 2 - предназначена для обработки приходящих на коммутатор сетевых пакетов;

Table 2 is intended for processing network packets arriving at the switch;

- calculate in the controller metrics cryptographic routers, performing the following actions:

получают значения загрузки процессоров криптомаршрутизаторов через интервал запроса статистики, в течение длительности одной итерации сбора статистики;

get the values of the load processors cryptographic routers through the interval request statistics, for the duration of one iteration of the collection of statistics;

вычисляют среднюю загрузку L_i процессора каждого криптомаршрутизатора в процентах, где i - номер криптомаршрутизатора;

calculate the average load L _{i of the} processor of each crypto-router in percent, where i is the number of the crypto-router;

вычисляют для каждого криптомаршрутизатора метрику по формуле M_i=(100-L_i)/10

calculate for each crypto router a metric using the formula M _i = (100-L _i ) / 10

and round the obtained M _i to the largest integer;

создают на контроллере для каждого коммутатора пустой одномерный массив (далее массив портов);

create on the controller for each switch an empty one-dimensional array (hereinafter, an array of ports);

записывают в конец массива портов каждого коммутатора номер каждого порта коммутатора, к которому подключен i-й криптомаршрутизатор, в количестве, равном значению M_i;

write to the end of the array of ports of each switch the number of each port of the switch to which the i-th crypto-router is connected, in an amount equal to the value of M _i ;

- include workstations;

- configure IP-addresses on workstations;

- send network traffic (hereinafter the stream) from one workstation (hereinafter referred to as the sender) connected to its switch (hereinafter switch 1) to another workstation (hereinafter referred to as the receiver) connected to its switch (hereinafter switch 2);

- send an ARP request from the sender to determine the MAC address of the recipient by its IP address;

- carry out the selection of the cryptographic router and determining the parameters of the tunnel on the switch 1, performing the following actions:

получают на коммутаторе 1 ARP-запрос от отправителя;

receive on the switch 1 ARP request from the sender;

обрабатывают ARP-запрос на коммутаторе 1 по таблице потоков 0 и отправляют ARP-запрос на контроллер для анализа;

process the ARP request on switch 1 using the flow table 0 and send an ARP request to the controller for analysis;

посылают из контроллера на коммутатор 1 команду обработать ARP-запрос по таблице потоков 1 как широковещательный;

sending from the controller to switch 1 a command to process the ARP request on the flow table 1 as a broadcast;

обрабатывают ARP-запрос на коммутаторе 1 по таблице потоков 1 и снова отправляют ARP-запрос на контроллер для анализа;

process the ARP request on switch 1 according to the flow table 1 and again send the ARP request to the controller for analysis;

определяют криптомаршрутизатор, на который нужно отправить ARP-запрос; путем случайного выбора в контроллере из массива портов коммутатора 1 выходного порта для отправки ARP-запроса;

determine the cryptographic router to which you need to send an ARP request; by randomly selecting in the controller from the array of ports of the switch 1 output port for sending an ARP request;

посылают из контроллера на коммутатор 1 команду установить правило коммутации в таблицу потоков 0 с таймаутом неактивности;

send from the controller to switch 1 a command to set the switching rule to the flow table 0 with an inactivity timeout;

посылают из контроллера на коммутатор 1 команду отправить ARP-запрос в туннельный порт с установленными параметрами туннеля;

send from the controller to switch 1 a command to send an ARP request to the tunnel port with the tunnel parameters set;

определяют на коммутаторе 1 с участием контроллера МАС-адрес удаленного интерфейса туннеля, выполняя следующие действия:

determine on the switch 1 with the participation of the controller MAC address of the remote interface of the tunnel by performing the following actions:

посылают из контроллера на коммутатор 1 команду отправить ARP-запрос из сетевого интерфейса коммутатора 1;

send from the controller to switch 1 a command to send an ARP request from the network interface of switch 1;

устанавливают при помощи искусственной подстановки в качестве МАС-адреса удаленного интерфейса туннеля МАС-адрес выбранного криптомаршрутизатора;

set using artificial substitution as the MAC address of the remote interface of the tunnel MAC address of the selected cryptographic router;

устанавливают с контроллера в таблицы потоков 0 и 1 коммутатора 1 правила коммутации с таймаутом неактивности;

from the controller, in the flow tables 0 and 1 of switch 1, the switching rules with inactivity timeout are set;

посылают из коммутатора 1 на коммутатор 2 инкапсулированный ARP-запрос отправителя;

send from switch 1 to switch 2 an encapsulated sender's ARP request;

- process the sender's ARP request on switch 2 using the controller, performing the following actions:

получают на коммутаторе 2 инкапсулированный ARP-запрос отправителя и согласно начальным правилам таблицы потоков 0 отправляют в туннельный порт;

receive on the switch 2 the encapsulated ARP request of the sender and according to the initial rules the flow table 0 is sent to the tunnel port;

отправляют ARP-запрос отправителя после декапсуляции на туннельном порту коммутатора 2 согласно начальным правилам таблицы потоков 0 в таблицу потоков 2;

send the sender's ARP request after decapsulation on the tunnel port of switch 2 according to the initial rules of flow table 0 to flow table 2;

обрабатывают ARP-запрос на коммутаторе 2 по таблице потоков 2 и отправляют на контроллер для анализа;

process the ARP request on switch 2 according to the flow table 2 and send to the controller for analysis;

посылают из контроллера на коммутатор 2 следующие команды:

send the following commands from the controller to the switch 2:

отправить ARP-запрос в порт, к которому подключен получатель;

send an ARP request to the port to which the recipient is connected;

установить в таблицы потоков 0 и 2 правила коммутации с таймаутом неактивности;

establish in the flow tables 0 and 2 switching rules with an inactivity timeout;

- receive ARP request to the recipient;

- send from the recipient to the sender ARP-response containing the MAC address of the recipient;

- receive ARP response on switch 2;

- carry out the selection of the cryptographic router and determination of the parameters of the tunnel on the switch 2, performing the following actions:

обрабатывают ARP-ответ на коммутаторе 2 по таблице потоков 0 и отправляют ARP-ответ на контроллер для анализа;

process the ARP response on switch 2 according to flow table 0 and send an ARP response to the controller for analysis;

посылают из контроллера на коммутатор 2 команду обработать ARP-ответ по таблице потоков 1 как широковещательный;

sending from the controller to switch 2 a command to process the ARP reply on the flow table 1 as a broadcast;

обрабатывают ARP-ответ на коммутаторе 2 по таблице потоков 1 и снова отправляют ARP-ответ на контроллер для анализа;

process the ARP response on switch 2 according to flow table 1 and again send the ARP response to the controller for analysis;

определяют криптомаршрутизатор, на который нужно отправить ARP-ответ; путем случайного выбора в контроллере из массива портов коммутатора 2 выходного порта для отправки ARP-ответа;

determine the cryptographic router to which you need to send an ARP response; by randomly selecting in the controller from the switch port 2 array of the output port to send an ARP reply;

посылают из контроллера на коммутатор 2 команду установить правило коммутации в таблицу потоков 0 с таймаутом неактивности;

sending from the controller to switch 2 a command to establish a switching rule in the flow table 0 with an inactivity timeout;

посылают из контроллера на коммутатор 2 команду отправить ARP-ответ в туннельный порт с установленными параметрами туннеля;

sending from the controller to switch 2 a command to send an ARP reply to the tunnel port with the tunnel parameters set;

определяют на коммутаторе 2 с участием контроллера МАС-адрес удаленного интерфейса туннеля, выполняя следующие действия:

determine on the switch 2 with the participation of the controller MAC address of the remote interface of the tunnel by performing the following actions:

посылают из контроллера на коммутатор 2 команду отправить ARP-запрос из сетевого интерфейса коммутатора 2;

send from the controller to switch 2 a command to send an ARP request from the network interface of switch 2;

устанавливают при помощи искусственной подстановки в качестве МАС-адреса удаленного интерфейса туннеля МАС-адрес выбранного криптомаршрутизатора;

set using artificial substitution as the MAC address of the remote interface of the tunnel MAC address of the selected cryptographic router;

устанавливают с контроллера в таблицы потоков 0 и 1 коммутатора 2 правила коммутации с таймаутом неактивности;

from the controller, in the flow tables 0 and 1 of switch 2, the switching rules with an inactivity timeout are set;

посылают из коммутатора 2 на коммутатор 1 инкапсулированный ARP-ответ получателя;

send from switch 2 to switch 1 the recipient's encapsulated ARP response;

- receive the recipient's ARP response on switch 1;

- process the recipient's ARP response on switch 1 using the controller by performing the following actions:

отправляют на коммутаторе 1 инкапсулированный ARP-ответ получателя согласно начальным правилам таблицы потоков 0 в туннельный порт;

send on switch 1 the encapsulated recipient ARP response according to the initial rules of the flow table 0 to the tunnel port;

отправляют ARP-ответ получателя после декапсуляции на туннельном порту коммутатора 1 согласно начальным правилам таблицы потоков 0 в таблицу потоков 2;

send the recipient's ARP response after decapsulation at the tunnel port of switch 1 according to the initial rules of flow table 0 to flow table 2;

обрабатывают ARP-ответ на коммутаторе 1 по таблице потоков 2 и отправляют на контроллер для анализа;

process the ARP response on switch 1 according to the flow table 2 and send to the controller for analysis;

посылают из контроллера на коммутатор 1 следующие команды:

The following commands are sent from the controller to switch 1:

отправить ARP-ответ в порт, к которому подключен отправитель;

send an ARP reply to the port to which the sender is connected;

установить в таблицы потоков 0 и 2 правила коммутации с таймаутом неактивности;

establish in the flow tables 0 and 2 switching rules with an inactivity timeout;

- receive an ARP response to the sender;

- transmit flow data between the sender and the recipient on the formed L2 connection without new calls to the controller.

Distinctive features of the invention are as follows.

The proposed method is based on the PKS technology, according to which all the traffic processing logic is transferred to a centralized controller, and the switches are only involved in switching traffic flows based on the flow tables loaded by the controller according to the installed protocol (for example, OpenFlow).

In the proposed method, each cryptographic router has a unique key set and a tunneled range of IP addresses. In order to correctly transfer an encrypted IP packet between a pair of cryptographic routers with pairing keys, IP addresses are substituted in the header of the original IP packet.

The method uses a tunneling protocol (GRE) to connect multiple networks into a single local area network, as well as to enable virtual machines to migrate between networks, allowing traffic of any level to be transmitted in an encapsulated form over an open network.

Traffic distribution on each switch is based on crypto-routers metrics, which, in turn, are based on the average load of crypto-routers for a given period of time.

The proposed method allows servicing virtual machine migrations between networks through the use of an inactive timeout for switching rules that are dynamically installed into traffic flow tables that distribute traffic to the switches.

The stated technical result is achieved due to the following.

The number of connected cryptographic routers to which the useful traffic is balanced depends only on the number of free ports on the switch. In the case of combining several switches into a stack, the number of free ports for connecting crypto routers will significantly exceed the number of crypto routers needed to create a secure 10/40 / 100G channel, which fully meets current business requirements.

The possibility of clustering crypto routers of different performance is also determined only by the presence of ports of appropriate performance on the switch. For balancing the distribution of traffic uses cryptographic router metrics that are proportional to their performance and load.

In the proposed method, crypto routers do not have any special functional requirements. Therefore, the type and manufacturer of the cryptographic router do not matter. The only condition is that the pair of cryptographic routers involved in the encryption / decryption of this traffic stream have agreed key sets and encryption algorithms.

In the case of the migration of the end device from one network (data center) to another, the flow of traffic from this end device from the network in which it was originally located is stopped. After the inactivity timeout expires, the rules that switch traffic flows from this end device, all rules related to this end device are deleted on all switches. Next, the controller re-forms and loads the new rules on the switches, which send traffic from and to the new location of the migrated end device.

Brief Description of the Drawings

The figure of the graphic image shows the scheme of combining three networks through a secure communication channel.

The following notation is used:

1, 2, 3 - connected networks;

4, 15, 20 - end devices;

5, 14, 19 - switches;

6 - the controller;

7, 8, 12, 13, 17, 18 — crypto routers;

9, 11, 16 — boundary network routers (insignificant for the proposed method);

10 is a global network.

The implementation of the invention

The proposed method can be implemented in the following software and hardware environment.

As a switch 5, 14, 19 a server with the following hardware characteristics can be used:

- processor: one Intel Xeon E5-2620 v3 processor (2.40 GHz, 6 cores);

- RAM: 8 GB;

- 10G network adapters: 3 Intel 82599ES adapters;

- 1G network adapters: 2 Intel I350-AM2 adapters.

This configuration includes one NUMA node with 12 logical cores with Hyper-Threading technology enabled.

10G network adapters are used to transmit useful traffic. A mandatory requirement for them is support for Intel DPDK technology.

1G network adapters are used for control communication with the controller and switch management.

The type and volume of the hard disk of the switch does not have a critical value, since during the operation of the switch there are no frequent or regular read-write operations. A 250 GB hard disk, SATA type, is sufficient.

Switch software option:

- Debian Server 8.7.1 (minimal install, kernel 4.10.5);

- Open vSwitch 2.7.0;

- Intel DPDK 16.11.

The hardware of the controller 6 can vary considerably depending on the maximum possible load on the controller. The load on the controller depends on the number of new traffic flows per second that the controller must handle, and on the complexity of the processing logic of each flow. A key hardware characteristic of the controller is the power of the processor.

As the controller 6, a server with the following hardware characteristics can be used:

- processor: one Intel Xeon E5-2620 v3 processor (2.40 GHz, 6 cores);

- RAM: 4 GB;

- 1G network adapters: 2 Intel I350-AM2 adapters;

- hard drive: 500 GB, SATA.

1G network adapters are used to manage communication with switches and administer the controller.

Controller software option:

- Debian Server 8.7.1;

- SDN-controller RYU;

- programs that perform the functions of creating L2 connections and balancing traffic.

To create programs that perform the functions of creating L2-connections and balancing traffic, can be an expert in the field of programming (programmer), competent in the field of network technologies and software-configured networks.

The software and hardware of the end devices 4, 15, 20 is not essential for the implementation of the method. The end device can be used by any personal computer with an installed operating system and connected to the network. The only requirement for end devices is the ability to send network traffic between them.

Software and hardware of crypto routers 7, 8, 12, 13, 17, 18 can be different, since the proposed method is intended for simultaneous use of crypto routers of different performance. Variants of crypto routers that can be used in the proposed scheme can be found (material at: https://infotecs.ru/product/setevye-komponenty/vipnet-coordinator-hw/).

The figure of the graphic image shows the connection scheme of three networks 1, 2, 3 through secure communication channels.

On the OF switch 5, 14, 19 (hereinafter the switch) each network is allocated one port to which the internal local network is connected. We assume that this port has the number 01, and in the future we will call this port an access port and denoted as eth01.

Also, a special port is allocated on the switch, which performs encapsulation / decapsulation of Ethernet frames going from / to the local network, respectively. We will call this port a tunnel port and assume that the tunnel port is number 02.

Tunnel creation also requires an IP interface on the switch. We assume that this interface will have the number 03, we will call this interface network.

The switch port to which the boundary router of the network is connected will be called external (by default, number 04).

In the diagram presented in the figure of the graphic image, each switch uses two virtual ports, veth04.1 and veth04.2, since two cryptographic routers are connected to each switch. In general, the number of switch virtual ports is equal to the number of crypto routers connected to the switch. These virtual ports do not really exist in switch configurations; however, they will be used in OF-tables (flow tables). Each virtual port is responsible for communication between a pair of networks. So we will assume that the following virtual ports are interconnected:

- veth04.1 network switch 1 and veth04.1 network switch 2;

- veth04.2 network switch 1 and veth04.2 network switch 3;

- veth04.2 network switch 2 and veth04.1 network switch 3.

The connection between each two networks is direct, i.e. Breaking a channel between a pair of networks does not imply a change in the traffic path through the third network Further, instead of the term “network”, we will use the term “Data Center” in the same way as the prototype.

We describe the communication scheme between data center 1 and data center 2 through a GRE tunnel. We set the following IP addresses on the interfaces. A GRE tunnel will be created between the network interfaces of the OF switches. One border of the GRE tunnel will have a network address of 172.16.101.2, and the second - 172.16.102.2.

Next, we consider the logic of the SDN controller operation by the example of ICMP traffic passing between the end device 4 client in DC1 and the end device 15 server in DC2. Let's start with filling in ARP tables.

1. The client's client workstation sends an ARP request to the OF-switch 5 to establish the hardware address of the server server.

2. ARP request arrives at port eth01 of the OF switch. The switch, in turn, must send it to other data centers, because at the initial moment of time the location of the server is unknown. Next, we consider the passage of traffic only to DC 2.

3. To send an ARP request, the OF switch sends it to the tunnel port eth02. The tunnel port instructed by the SDN controller dynamically creates the GRE tunnel, setting the sender address 172.16.101.2, and the destination address 172.16.102.2.

4. Assuming that all routes to other data centers pass through interface eth03, an ARP request should be sent to this interface after packaging into the headers of the tunnel protocol. However, a problem will arise here: to form an Ethernet frame containing an IP packet with the destination address 172.16.102.2, you need to know the hardware address of the destination. As a result, the controller 6 will form an ARP request on behalf of the address 172.16.101.2. This ARP request will also be sent through the eth03 interface.

5. To process this ARP request, the OF switch must replace the IP address for which the hardware address is being requested with the address of the gateway 9 of the local data center1 172.16.101.1 network and then send the request to the external interface eth04.

6. As a result, the gateway 9 will receive a request to send the hardware address of the gateway to node 172.16.101.2. The gateway will generate an ARP response towards the node 172.16.101.2. This response should go to interface eth04 of the switch.

7. On the OF switch in the ARP reply received from the gateway, you must replace the sender's address with the address 172.16.102.2 for which the ARP request was actually sent, and also send the modified answer to the eth03 interface, through which the node's 172.16.102.2 hardware address was requested.

8. After receiving the ARP response for the address 172.16.102.2, the GRE interface comes to the eth03 network interface and encapsulates the original ARP request from the client. Note that the hardware address of the receiver in this GRE packet will be equal to the hardware interface address of the router 9.

9. The GRE packet arrives at Data Center 2 and goes to the external port eth04 of the OF switch 14 and then is sent to the tunnel port of eth02, where decapsulation occurs, and the original ARP client request is sent by the OF switch to the access port eth01, from which it gets to server server.

10. The reverse ARP response from the server passes the described path with obvious changes in IP and hardware addresses.

Consider the configuration of the OF-tables and the actions of the controller 6. Each OF-switch will use three tables: a table with the numbers 0, 1 and 2. We describe the table with the number 0 after the initialization of the OF-switch. We divide the flow table into several logical parts. The part that describes the processing of GRE traffic looks as shown in Table. 1 (hereinafter, the parameters of OF records that are not essential for the algorithm are omitted).

The first rule of the table (recall that the lower the rule in the table, the higher its priority) means that each incoming packet must be considered a packet of a new stream and sent to the controller to make a decision on processing a new stream.

The second rule defines how outgoing tunnel traffic should be handled (in this case, GRE). It must be sent to the output interface of the OF switch in the direction of the border router.

Rule 3 sends incoming tunnel traffic to the tunnel interface for decompression.

Rules 4 and 5 determine the processing of unpacked GRE traffic, redirecting it to the appropriate ports of Table 2, depending on the sender's address. This is done using the resubmit action, which starts the procedure for processing the OF-switch traffic with table number 2.

The resubmit rule allows you to specify not only the table number, but also the port number. In this case, the flow will be processed by a new table, and in doing so, the switch will assume that the flow has come from the port specified in the resubmit command. For example, if the resubmit (2, 4.1) command is executed, the switch will start processing the flow with table 2 and in the rules it will “assume” that the flow came from port 4.1. We can say that the resubmit action allows us not to send the stream to another switch, but to send it to the same, but to a different port and another table. Note that the additional classification of incoming traffic is necessary to determine the data center number from which traffic came.

The second part of the stream table 0 is responsible for processing ARP traffic (Table 2).

The first line indicates that it is necessary in all ARP requests coming from the network interface of the switch to replace the destination IP address with 172.16.101.1 (gateway address), and send the request to the external port eth04 to the border router. The second line sends ARP responses from the border router with a reverse substitution of the target address, the third allows you to forward all ARP requests coming from the border router to the OF switch network interface, and the fourth is responsible for sending all ARP responses from the eth03 network interface to the border the router.

The flow table with the number 1 handles traffic coming from the local network. It describes the traffic tunneling rules and depends on the tunneling protocol used. For example, for a GRE tunnel, the local tunnel address, remote tunnel address, and tunnel identifier can be specified. In tab. 3 shows the flow table 1 for the OF switch in the case of using the GRE protocol.

In the flow table 1 for each OF switch, you must correctly set the GRE parameters of the tunnel: the address of the recipient and the sender. Each entry actually indicates that the traffic coming to Table 1 from the IN_PORT port should be sent to the tunnel port, during encapsulation in which the parameter values specified in the instructions set_field should be used. The first entry provides communication with the data center 2 through the port veth04.1, and the second - with the data center 3 through the port veth04.2. Thus, after passing through the flow table 1, the stream will be encapsulated in the GRE protocol headers and sent to the border router through the tunnel port eth02. However, as already noted, the Ethernet traffic will not come out of the tunnel port, but from the eth03 network interface, where it will be sent via the TCP / IP stack after GRE encapsulation.

The flow table of the 2 OF switch (Table 4) serves to process the traffic going to the local network. After initialization, this table contains only one rule.

We describe the basic algorithm of operation of the controller 6. The algorithm uses the flow tables №0, 1, 2 (hereinafter referred to as the tables) described above. The reference to the example of the discussed flow table is referred to as “table.”

1. At the initial moment of time at the client workstation, the user enters the ping server command.

2. We assume that before there was no connection between the two nodes, and the workstation is accessing the server for the first time. In this case, the workstation to form the Ethernet frame should find out the MAC address of the server by its IP address. Consequently, a broadcast ARP request is sent from the workstation, which goes to the OFC data center switch1.

3. As previously indicated, at the initial moment of time in table 0 of the OF switch there is a rule that sends the first packet of each new flow to the controller. The switch sees the ARP request from the workstation as a new flow and therefore calls the controller for instructions on how to handle this new flow.

4. Having received information about the new stream, the controller determines that the recipient's address is unknown to it, therefore, it must send the command to the switch to send this stream to all ports except the incoming one, i.e. start the process of studying the topology. However, in the case of using tunnels, it is not enough to send traffic to all ports using table 0. The controller indicates the OFC data center switch1 using the resubmit rule to start the traffic processing procedure on table number 1. Thus, the controller sends the command to process the stream according to table 1, counting that it came consistently from all virtual ports: eth04.1 and eth04.2. As a result, the package will be sent in all possible directions: to data center2 and data center3.

5. Having received a command from the controller, the OF-DDC1 sends traffic to table 1. For each port with the number N in this table there is an entry that indicates that if the flow came to the port with the number N (recall that in table 1 this is virtual ports), then this stream should be sent to GRE / VXLAN port eth02 with the specified tunnel parameters. The packet after entering the tunnel port will be packaged, and the new tunnel packet must be sent through the eth03 network interface. However, there is not enough information to form an Ethernet frame, namely, there is no indication of which hardware address to use. As a result, an ARP request is sent to 172.16.102.2 (or to 172.16.103.2) on behalf of the address 172.16.101.2. This request with the replacement of the target address arrives at the border router, which in turn sends an ARP response processed on the switch using Table 0. As a result, an Ethernet frame is formed on the switch of DPC1 and sent to output port eth04.

6. Tunnel traffic through the global network reaches the border routers of data center 2 and data center 3 and then to the eth02 tunnel port of the boundary OF switch of each data center. Further we will consider only DPC2, since in DPC3, events will occur similarly, with the only exception that the broadcasting ARP request to the server will be dropped in the local data center network3 due to the fact that no one will respond to it. The OFC data center switch2, in accordance with table 0, sends decapsulated GRE traffic for processing to table number 2.

7. Initially, in Table 2, there is a single entry indicating that each incoming packet is considered to be the beginning of a new flow and send it to the controller for analysis. Thus, after getting from the tunnel to the appropriate port of Table 2, the packet will be sent to the controller for analysis.

8. The controller, having received the packet, determines the sender's MAC address and records in its MAC table that this address is available behind a certain virtual port (veth04.1 or veth04.2) of the data center's OF2 switch. The port is determined on the basis of information received from the OF switch, which indicates exactly which port of Table 2 the new stream packet came to. In the example, the packet with the client MAC address will come to port 4.1 of table 2. Therefore, the controller will record the client record: 4.1.

9. The controller sends the following commands to the OFC data center 2 switch: send the received packet to the output port eth01, set an entry in table 0 that tells you to send all traffic with the MAC address of the client client to port 4.1 of table 1, which is connected to the external port eth04 that responds for communication with DPC1, and in Table 2 to establish an entry, on the basis of which the OF switch must send all traffic coming from the hardware client address to the server address to the eth01 port of the local network. Thus, the last rule will prevent calls to the controller for all subsequent packets of this stream. Note that the rule should be set with a timeout of inactivity to track changes in the network topology. We present the summary records in tables 0 and 2 of the OF-switch Ts0D2. New entries in the flow table 0 look as shown in table. 5. New entries in the table of streams 2 look as shown in table. 6

10. As a result, the ARP request from the client will go to the local data center network2 and then to the server.

11. The server will send an ARP reply to the client, which will be returned to the OFC data center 2 switch.

12. The received ARP-response OF-switch can process, because in table 0 there is already an entry corresponding to the ARP response: the MAC address of the recipient in the response is equal to client. This record dictates sending the packet for processing to port 4.1 of table 1. And further, according to the rules of table 1, the packet will be encapsulated and sent to Data Center1.

13. As a result, the ARP response will go to the OF-datacenter OF1. For this ARP response, the same sequence of events will be repeated as for the ARP request on the OFC data center switch2.

14. An ICMP request from a client towards the server, like an ICMP response from a server, will no longer call the controller, since Each OF switch already has packet data handling rules.

Next, we describe the balancing of traffic to cryptographic routers using the example shown in the figure of a graphic image. An additional port will be added to the switch configuration for connecting a second crypto-router.

Each of the OF switch will use tables with numbers 0, 1, and 2. Table 0 will serve to classify traffic and its distribution between crypto routers, tables 1 and 2 will handle traffic, respectively, outgoing and incoming to the local network. The table with the number 0 in the part of the GRE traffic processing after the initialization of the OF-DDC1 switch looks as shown in Table. 7

Partially explain the table entries. In lines 2-5 in the IP_DST addresses of the form 172.16. M.P. the letter M means 100 + data center number, P is the port number to which the crypto-router is connected. When sending GRE traffic to the global network, the destination IP address of the GRE tunnel is replaced. This is done to simplify the configuration of crypto routers. The last two rows of the table are intended for processing the tunnel traffic entering the data center switch 1.

The part of table 0 responsible for processing ARP traffic (see Table 8) provides ARP requests from crypto routers to the eth03 of the switch network interface, as well as return transmission of ARP responses. Table number 1 will look as shown in Table. 9.

We give some explanations. To control the traffic passing through the crypto router, we will use different addresses of the remote GRE tunnel interface. Thus, the sender's IP address uniquely identifies the sending data center, and the recipient's IP address determines the destination data center, and through which cryptographic router the traffic passed. So, for example, a GRE tunnel like:

172.16.102.2 ← → 172.16.103.4

means that the traffic comes from Data Center 2, goes to Data Center 3 and passes through a cryptographic router that is connected to port eth04 of the Data Center 2 OF switch. In the following table, for each remote data center, and for each cryptographic router leading to it, their addresses of the end points of the GRE tunnel are specified. The input virtual ports of the OF-switch (the first column of Table 7) are formed according to the scheme: the first digit is the data center number, the second digit is the port number to which the crypto-router is connected.

The following basic algorithm of the controller operation when balancing ICMP traffic has been adopted.

1. At the initial moment of time at the workstation, the user enters the ping server command.

3 .. Next, repeat steps 2-4 of the previously described traffic flow scheme without balancing. As before, we consider only the traffic going to DC2.

4. At the initial moment of time in Table 1 (DPC1), the packet arriving at ports 4.1 and 4.2 is the first packet of the new flow, and the OF switch addresses the controller for instructions.

5. The controller, receiving a packet from port 4.1 of table 1 (in the direction of data center 2), selects at random (or in accordance with a given balancing algorithm) a crypto-router which the traffic must pass through. Let the crypto-router with number 2 be selected. In this case, the controller sends the OF-switch command to set the rule: send the entire given stream using the resubmit command from table 0 to table 1, but to port 5.1.

6. After receiving a command from the controller, the OF switch sends traffic to the tunnel port, indicating which tunnel parameters should be set. However, there is not enough information to create packet traffic, namely, there is no indication of which one to use the recipient's hardware address. As a result, an ARP request is sent to 172.16.102.5 on behalf of the address 172.16.101.2 and then using the rules of table 0, it will get to the controller, since only the entry with the lowest priority will apply to it.

7. The controller, upon receiving an ARP request, sends two OpenFlow rules to the switch: the first rule indicates that the source address and the destination address in the ARP request must be changed and sent to port eth05, which the border router is located at, and the second one indicates reverse substitution and send to port eth03. The rules are given in table. ten.

We describe the rule of formation of addresses for replacement. Let the controller receive an ARP request with the following parameters:

arp_spa = 172.16.10N.2, arp_tpa = 172.16.10M.R,

where N is the sender's data center number;

M - recipient data center number;

P is the port number of the OFS of the data center of the sender, to which connect the crypto-router selected during balancing.

Then the request is transformed into the following:

arp_spa = 172.16.10N. (M + 1), arp_tpa = 172.16.10N.1

and sent to port eth0P.

The reverse transformation is performed according to the rule:

ARP response with parameters

arp_spa = 172.16.10N.1, arp_tpa = 172.16.10N. (M + 1),

incoming eth0P port is converted to an ARP response:

arp_spa = 172.16.10M.R, arp_tpa = 172.16.10N.2

and sent to port eth03.

8. After the specified replacement, the network port eth.03 will receive an ARP reply with the hardware address of the selected border router. Now the Ethernet frame containing the tunnel packet will be generated and sent to the output interface eth05.

9. Tunnel traffic through the global network goes to the border router Data Center 2 and Data Center 3 and then to the tunnel port of the boundary OF switch of each data center. This is ensured by the initial rules for processing ARP requests in the OF-table with the number 0. Next, we will consider only DC2, since in DPC3, events will occur similarly, with the only exception that the broadcasting ARP request to the server will be dropped in the local data center network3 due to the fact that no one will respond to it. The OF-switch in accordance with table 0 sends the unpacked GRE-traffic for processing in the table with the number 2.

10. The next steps completely coincide with steps 7-11 of the scheme without load balancing.

11. As a result, an ARP response from the server server will be sent to the OFOD1 data center switch.

12. Further steps almost completely repeat steps 12-14 of the circuit without load balancing. The only difference is that the ARP response after hitting the data center switch 2 in table 1 also passes the described balancing procedure.

The settings of each cryptographic router matter. In each data center, the network settings of the cryptographic routers on the LAN side are identical. The list of access addresses on cryptographic routers is also identical. In fact, only one access address is specified in the settings - the local point of creation of the GRE tunnel. So, for data center 1 crypto routers, this address is accepted as 172.16.101.2.

In addition to the above algorithm, we consider the process of forming metrics, on the basis of which the traffic is balanced on crypto routers.

1. Every second for 15 seconds, a special process running on the controller, using the SNMP protocol, polls the processor load of all crypto-routers to which the traffic is balanced.

2. After 15 seconds, the statistics collection is temporarily stopped, and the average processor load of each crypto-router for a specified period of time is calculated. Denote it by L _i , where i is the number of the cryptographic router. This parameter will be measured in percent. If the crypto-router contains several processors, then the average load of all the processors of the crypto-router is calculated.

3. For each crypto router, the metric is calculated using the formula:

M _i = (100-L _i ) / 10

The resulting number is rounded to a larger integer. Note that the calculation of the metric does not use the load of the RAM memory of crypto routers, since it is not a critical parameter for the operation of these devices in the context of the algorithm.

4. Next, on the controller for each balancing OF-switch traffic, an array is formed (hereinafter an array of ports) in which each switch port leading to the corresponding crypto-router is repeated as many times as the metric M of the crypto-router.

5. After the array is formed, for each new flow on the controller, a random selection of the array element occurs, which, as already mentioned, is the switch port leading to the corresponding cryptographic router. Also, immediately after the port arrays for all switches are formed, the described process of generating metrics is restarted from point 1.

When several data centers are merged, as mentioned earlier, it is required to ensure the migration of a host from one data center to another while preserving the possibility of establishing a connection with it of other hosts. The described balancing scheme takes into account such migration, which is ensured by the use of inactivity timeouts in OpenFlow rules.

So in tables 0 and 2, the rules are set with an inactivity timeout (idle_timeout). If the host migrates to another data center, the traffic stops coming to the local OF-switch of the data center, from which the node has migrated. As a result, there will also be no incoming traffic to the migrated node on the switch in another data center, therefore irrelevant rules will be deleted after the inactivity timeout on both the local OF switch and the OF switches of other data centers.

After the removal of the rules in the tables described above, when a new traffic arrives at an OF switch, going in the direction of the migrated node, the switch will contact the controller, which will instruct it to broadcast traffic to all tunnels. As a result, the traffic will reach the node that has migrated, and all OF-switches will have new rules for processing incoming packets according to the algorithms described above.

Claims (83)

A way to create a secure L2 connection between packet-switched networks, with the following installed on the networks: a controller connected to all networks and configured to: create flow tables containing the rules for processing network packets; exchange control messages with switches according to the established control protocol; request, receive and analyze information from other network devices; switches, each of which is installed in its network, is connected to it and configured to: receive and send network packets; exchange control messages with the controller according to the established control protocol; store in memory at least three stream tables and perform network packet processing on these tables; process packets using the specified stream table; create virtual ports with the support of the installed tunneling protocol; cryptographic routers (hereinafter referred to as crypto routers), each of which has at least one processor; moreover, each network has at least two crypto-routers connected to its switchboard; Each crypto-router is configured to: receive and send network packets; upload key information; configure tunneling IP address ranges; encrypt, decrypt, and route network packets; respond to requests for information from the controller; Workstations installed and connected each to its network and made with the ability to exchange network packets with each other; the way is that: form key information for all crypto routers; allocate in each network ranges of IP addresses for installed devices, as well as ranges of IP addresses encrypted by cryptographic routers; include crypto routers; key information is uploaded to crypto routers; set up IP addresses on crypto routers; set up dedicated ranges of tunneled IP addresses on crypto routers; include switches; create on each switch two virtual ports: port for encapsulation / decapsulation of network packets in the tunneling protocol headers (hereinafter the tunnel port); port level L3, which serves as the final IP-interface of the tunnel (hereinafter the network interface); configure on the network interfaces of all switches IP addresses; configure the switches with the IP address of the controller; include the controller; create on the controller a configuration file containing the values of the following constants: inactivity timeout of rules in the flow tables, statistics request interval, duration of one iteration of statistics collection; perform on the controller the registration and initialization of switches, during which three flow tables with the initial set of rules are loaded onto each switch: table 0 - intended for distribution to other tables of network packets; Table 1 is designed to handle outgoing network packets from the switch; Table 2 is intended for processing network packets arriving at the switch; calculate in the controller metrics cryptographic routers, performing the following actions: get the values of the load processors cryptographic routers through the interval request statistics, for the duration of one iteration of the collection of statistics; calculate the average load L _{i of the} processor of each crypto-router in percent, where i is the number of the crypto-router; calculate for each crypto router a metric using the formula M _i = (100-L _i ) / 10 and round the obtained M _i to the largest integer; create on the controller for each switch an empty one-dimensional array (hereinafter, an array of ports); write to the end of the array of ports of each switch the number of each port of the switch to which the i-th crypto-router is connected, in an amount equal to the value of М _i ; include workstations; configure IP addresses on workstations; send network traffic (hereinafter the stream) from one workstation (hereinafter referred to as the sender) connected to its switch (hereinafter switch 1) to another workstation (hereinafter the receiver) connected to its switch (hereinafter switch 2); send an ARP request from the sender to determine the MAC address of the recipient by its IP address; select a cryptographic router and determine tunnel parameters on switch 1 by performing the following actions: receive on the switch 1 ARP request from the sender; process the ARP request on switch 1 using the flow table 0 and send an ARP request to the controller for analysis; sending from the controller to switch 1 a command to process the ARP request on the flow table 1 as a broadcast; process the ARP request on switch 1 according to the flow table 1 and again send the ARP request to the controller for analysis; determine the cryptographic router to which you need to send an ARP request by randomly selecting in the controller from the port array of switch 1 of the output port to send an ARP request; send from the controller to switch 1 a command to set the switching rule to the flow table 0 with an inactivity timeout; send from the controller to switch 1 a command to send an ARP request to the tunnel port with the tunnel parameters set; determine on the switch 1 with the participation of the controller MAC address of the remote interface of the tunnel by performing the following actions: send from the controller to switch 1 a command to send an ARP request from the network interface of switch 1; set using artificial substitution as the MAC address of the remote interface of the tunnel MAC address of the selected cryptographic router; from the controller, in the flow tables 0 and 1 of switch 1, the switching rules with inactivity timeout are set; send from switch 1 to switch 2 an encapsulated sender's ARP request; process the sender's ARP request on switch 2 using the controller by performing the following steps: receive on the switch 2 the encapsulated ARP request of the sender and according to the initial rules the flow table 0 is sent to the tunnel port; send the sender's ARP request after decapsulation on the tunnel port of switch 2 according to the initial rules of flow table 0 to flow table 2; process the ARP request on switch 2 according to the flow table 2 and send to the controller for analysis; send the following commands from the controller to the switch 2: send an ARP request to the port to which the recipient is connected; establish in the flow tables 0 and 2 switching rules with an inactivity timeout; receive an ARP request at the recipient; send from the recipient to the sender an ARP reply containing the MAC address of the recipient; receive an ARP response on switch 2; select a cryptographic router and determine tunnel parameters on switch 2 by performing the following actions: process the ARP response on switch 2 according to flow table 0 and send an ARP response to the controller for analysis; sending from the controller to switch 2 a command to process the ARP reply on the flow table 1 as a broadcast; process the ARP response on switch 2 according to flow table 1 and again send the ARP response to the controller for analysis; determine the cryptographic router to which you need to send an ARP response, by randomly selecting in the controller from the switch port 2 array of the output port to send an ARP reply; sending from the controller to switch 2 a command to establish a switching rule in the flow table 0 with an inactivity timeout; sending from the controller to switch 2 a command to send an ARP reply to the tunnel port with the tunnel parameters set; determine on the switch 2 with the participation of the controller MAC address of the remote interface of the tunnel by performing the following actions: send from the controller to switch 2 a command to send an ARP request from the network interface of switch 2; set using artificial substitution as the MAC address of the remote interface of the tunnel MAC address of the selected cryptographic router; from the controller, in the flow tables 0 and 1 of switch 2, the switching rules with an inactivity timeout are set; send from switch 2 to switch 1 the recipient's encapsulated ARP response; receive the recipient's ARP response on switch 1; process the recipient's ARP response on switch 1 using the controller by performing the following actions: send on switch 1 the encapsulated recipient ARP response according to the initial rules of the flow table 0 to the tunnel port; send the recipient's ARP response after decapsulation at the tunnel port of switch 1 according to the initial rules of flow table 0 to flow table 2; process the ARP response on switch 1 according to the flow table 2 and send to the controller for analysis; The following commands are sent from the controller to switch 1: send an ARP reply to the port to which the sender is connected; establish in the flow tables 0 and 2 switching rules with an inactivity timeout; receive an ARP reply from the sender; transmit the flow data between the sender and the receiver over the formed L2 connection without new calls to the controller.

Images