WO2016164061A1

WO2016164061A1 - Big data transfer

Info

Publication number: WO2016164061A1
Application number: PCT/US2015/037802
Authority: WO
Inventors: Yogesh DUJODWALA; Pramod Kumar RAMACHANDRA
Original assignee: Hewlett Packard Enterprise Development LP
Current assignee: Hewlett Packard Enterprise Development LP
Priority date: 2015-04-08
Filing date: 2015-06-25
Publication date: 2016-10-13
Anticipated expiration: 2017-10-08

Abstract

Examples relate to techniques for Software Defined Networking (SDN) based big data transfer. In an example, a SDN controller receives an API request for big data transfer from an application, wherein the API request contains an amount of data to be transferred and an identity of each node of a plurality of nodes to which the data is to be transferred. Upon receiving the API request for data transfer, the SDN controller performs discovery of the network topology to identify QoS parameters. Next, the SDN controller activates a network template on the network topology based on the identified QoS parameters to generate a changed network topology. Then, the SDN controller enables the data to be transferred over the changed network topology.

Description

] BIG DATA TRANSFER

BACKGROUND

[0001] Cloud has become the natural choice for big data processing because any number of Virtual Machines can be created and destroyed on demand. By nature of the use case, big data processing requires large amount of computing power, storage and network bandwidth which can be provided by a cloud environment. An important aspect to take into consideration in big data processing is the network bandwidth for the transmission of the data.

BRIEF DESCRIPTION OF THE DRAWINGS

[0002] [0] 00 The present application may be more fully appreciated in connection with the following detailed description taken in conjunction with the accompanying drawings, in which like reference characters refer to like parts throughout, and in which:

[0003] FIG. 1 illustrates a network architecture of the Software Defined Networking

(SDN) based routing for big data transfer in accordance with various examples of the disclosure;

[0004] FIG. 2 illustrates a protocol diagram describing the SDN based routing for big data transfer in accordance with various examples of the disclosure.

[0005] FIG. 3 illustrates an example method for SDN based big data transfer in accordance with various examples of the disclosure;

[0006] FIG, 4A-C illustrate the change of network topology when applying different network templates in accordance with various examples of the disclosure;

[0007] FIG. 5 illustrates a SDN based big data transmission system in accordance with various examples of the disclosure; and

[0008] FIG. 6 illustrates a network component for implementing various examples of the disclosure.

DETAILED DESCRIPTION

[0009] In the following detailed description of examples of the disclosure, numerous specific details are set forth in order to provide a more thorough understanding of the disclosure. However, it will be apparent to one of ordinary skill in the art that the disclosure may be practiced without these specific details. In other instances, well-known features have not been described in detail to avoid unnecessarily complicating the description. [0010] The need to transfer and process the data efficiently is one of the critical aspect for big data processing. There is a lot of time consumed by the big data deployment engine to transfer the data to the nodes for computation. The time consumed to transfer the data to the distributed set of nodes is directly related to QoS (Quality of Service) factors of the network infrastructure. Since the big data transfer is required only at a specific period in time (before data processing), the Rot (Return on Investment) for such network infrastructure is not justified. There is a need for on-demand network infrastructure which can adapt with the data requirement before the big data is transmitted. In accordance with various examples of the disclosure, a dynamic network topology that flex-up and flex-down based on the data transfer requirement is realized.

[0011] SDN (Software Defined Networking): SDN is an emerging architecture purporting to be dynamic, manageable, cost-effective, and adaptable, seeking to be suitable for the high-bandwidth, dynamic nature of today's applications. SDN architectures decouple network control and forwarding functions, enabling network control to become directly programmable and the underlying infrastructure to be abstracted from applications and network services.

[0012] Network Quality Of Service: In packet-switched networks, quality of service is affected by various factors, which can be divided into "human" and "technical" factors. Human factors include: stability of service, availability of service, delays, user information. Technical factors include: reliability, scalability, effectiveness, maintainability, etc. Low throughput due to varying load from disparate users sharing the same network resources and the bit rate (the maximum throughput) that can be provided to the big data to enable maximum efficiency are particularly concerned about here.

[0013] Virtual Renters: Virtual Routers (Software Networking devices) can be created as a part of cloud infrastructure on a need basis. This allows the network traffic to be segregated and the Network QoS can be improved. These Virtual Routers can be created and destroyed on demand.

[0014] Network Templates: Network templates can be created for the network topology according to the data transmission requirement. These templates are configured by the network administrator which has the following characteristics:

[0015] 1. Network templates consists of various routers and routing configuration (such as QoS) for the nodes to be used by the big data processing. [0016] 2. The existing network infrastructure QoS can be changed with the configuration mentioned in the templates.

[0017] 3. New virtual Routers mentioned in the network template can he created on demand to provide efficient networking.

[0018] R1eferring now to FIG, 1, a network architecture of the SDN based routing for big data transfer in accordance with various examples of the disclosure is described.

As shown in FIG. 1 , a SDN controller 100 is located between the big data deployment engine 102 (which is an example of an application) and the network topology 104. The big data deployment engine 102 sends an API request for big data transfer to the SDN controller 100 which in turn performs discovery of the network topology 104 to identify various QoS parameters, activates a network template on the network topology based on the identified QoS parameters to generate a changed network topology. Once the network template is activated on the network topology, the big data transfer is enabled from the big data deployment engine 102 to the changed network topology 104. Alternatively, the SDN controller 100 and the big data deployment engine 102. communicates with the network topology 104 via a SDN router 108.

[0019] In an example, after the big date transfer is completed, the SDN controller 100 may restore the network topology 104 into its original form. Alternatively, the SDN controller 100 may not change the network topology 104 after the big date transfer is completed until it receives another API request for big data transfer and activates another network template on the network topology 104.

[0020] Referring now to FIG. 2, a protocol diagram describing the SDN based routing for big data transfer in the network architecture of FIG. 1 is described. At 201 , the big data deployment engine 102 identifies each node of a plurality of nodes to which data needs to be transferred and the amount of data that needs to be transferred, and at 202, it sends an API request for big data transfer to the SDN controller 100, in an example, the API request may contain amount of data that needs to be transferred and an identity of each node of a plurality of nodes to which the data is to be transferred such as IP address and/or hostname. At 203, upon receiving the API request from the data deployment engine 102, the SDN controller 100 performs discovery of the network topology 104 to identity various QoS parameters, and the network topology 104 returns discovery results i.e. QoS parameters to the SDN controller 100. At 204, the SDN controller 100 activates a network template on the network topology based on the identified QoS parameters to generate a changed network topology. Then at 205, once the selected network template is activated on the network topology, the big data transfer is enabled from the big data deployment engine 102 to the changed network topology 104. After the big data transfer is completed, at 206, the SDN controller 100 restores the network topology 104 into its original form. Alternatively, the SDN controller 100 may not change the network topology 104 after the big data transfer is completed until it receives another API request for big data transfer and activates another network template on the network topology 104.

[0021] In an example, the network template may comprise routers and routing configuration for the nodes to which the data is to be transferred. In another example, the network templates may further comprise virtual routers which can be created as a part of network topology and can be destroyed on demand.

[0022] In another example, the API request sent by the big data deployment engine 102 may comprise at least one QoS parameter requirement which should be satisfied for the big data transfer, and accordingly the selection of a network template by the SDN controller 100 at 204 may be based partly on the at least one QoS parameter requirement. In an example, the QoS parameter may be throughput. Alternatively, the QoS parameter may be selected from a group of QoS parameters comprising human factors which include stability of service, availability of service, delays, user information and technical factors which include reliability, scalability, effectiveness, maintainability, etc.

[0023] Referring now to FIG. 3, an example method 300 for SDN based big data transfer in accordance with various examples of the disclosure is described. At block 301, a SDN controller receives an API request for big data transfer from an application, wherein the API request contains amount of data that needs to be transferred and an identity of each node of a plurality of nodes to which the data is to be transferred such as IP address and/or hostname thereof. At block 302, the SDN controller performs discovery of the network topology to identify current QoS parameters. At block 303, based on the identified QoS parameters, the SDN controller activates a network template on the network topology to generate a changed network topology. At block 304, the SDN controller enables the data to be transferred over the changed network topology.

[0024] In an example, the method 300 may further comprise block 305 at which the SDN controller restores the network topology into its original form after data transfer is completed. Alternatively, the SDN controller may not change the network topology after the date transfer is completed until it receives another API request for data transfer and activates another network template on the network topology,

[0025] In another example, the network template may comprise routers and routing configuration for the nodes to which the data is to be transferred. In another example, the network template may further comprise virtual routers which can be created as a part of network topology and can be destroyed on demand, such as the SDN router 108 shown in FIG. 1.

[0026] In another example, the API request sent by the application may comprise at least one QoS parameter requirement which should be satisfied for the data transfer, , and the network template is created based on the at least one QoS parameter requirement. In an example, the QoS parameter may be throughput. Alternatively, the QoS parameter may be selected from a group of QoS parameters comprising human factors which include stability of service, availability of service, delays, user information and technical factors which include reliability, scalability, effectiveness, maintainability, etc.

[0027] Referring now to FIG. 4A-C, the change of network topology when apply ing different network templates in accordance with various examples of the disclosure is described. The original network topology 400 as shown in FIG. 4A may comprise a plurality of big data nodes A, B and C as well as a plurality of routers Rl, R2 and R3.

[0028] In an example, for big data which needs to be transferred to a set of distribiEted big data nodes A, B and C, different network templates can be created based on the amount of the big data that needs to be transferred and the QoS requirements. In this example, two changed network topologies when applying two different network templates to the network topology are shown in FIG. 4B and 4C respectively.

[0029] As shown in FIG. 4B, the router R3, which is labeled by the dashed circle, is omitted when compared to the original network topology. The network template applied in Fig. 4B is suitable for big data transfer which does not require high bandwidth.

[0030] As shown in FIG. 4C, virtual router R4, which is labeled by the thick circle, is added when compared to the original network topology. The network template applied in Fig. 4C is suitable for big data transfer which requires high bandwidth.

[0031] In the above examples of the network templates, only network bandwidth which reflects throughput is contemplated, however, those skilled in the art may also take human factors which include stability of service, availability of service, delays, user information and technical factors which include reliability, scalability, effectiveness, maintainability, etc into account when creating network templates,

[0032] It can be seen from above that by means of applying different network templates, different QoS can be provided by a same network infrastructure through realizing a dynamic network topology that flex-up and flex-down based on the big data transfer requirement.

[0033] Referring now to FIG. 5, a SDN based data transmission system 500 in accordance with various examples of the disclosure is described. The SDN based data transmission system 500 may comprise a detection module 501 , a discovery module 502, an activation module 503, and a transmission enable module 504.

[0034] The detection module 501 detects whether an API request for data transmission is arrived from an application, wherein the API request contains amount of data that needs to be transmitted and an identity of each node of a plurality of nodes to which the data is to be transmitted.

[0035] The discovery module 502. carries out discovery of the network topology to determine current QoS parameters upon detecting by the detection module 501 that an API request for data transmission is arrived from an application.

[0036] The activation module 503 activates a network template on the network topology according to the determined QoS parameters to generate a changed network topology.

[0037] The transmission enable module 504 enables the data transmission from the application to the changed network topology when the network template is activated on the network topology.

[0038] In an example, the SDN based data transmission system 500 may further comprise a recovery module 505 which recovers the network topology after data transmission is completed.

[0039] In another example, the network templates may comprise routers and routing configuration for the nodes to which the data is to be transmitted. In another example, the network templates may further comprise virtual routers which can be created as a part of network topology and can be destroyed on demand, such as the SDN router 108 shown in FIG. 1.

[0040] In another example, the arrived API request may comprise at least one QoS parameter requirement which should be satisfied for the data transmission, and , and the network template is created based on the at least one QoS parameter requirement. In an example, the QoS parameier may be throughput. Alternatively, the QoS parameter may be selected from a group comprising human factors which include stability of service, availability of service, delays, user information and technical factors which include reliability, scalability, effectiveness, maintainability, etc.

[0041] At least some of the features/methods described in the disclosure may be implemented in a general computing device and/or a network apparatus or component, such as a computer or network component with sufficient processing power, memory resources, and network throughput capability to handle the necessary workload placed upon it. For instance, the features/methods of the disclosure may be implemented using hardware, firmware, and/or software installed to run on hardware. FIG, 6 illustrates a network component 600 suitable for implementing various examples of the disclosure. The network component 600 includes a processor 601 (which may be referred to as a central processor unit or CPU), an associated memory 602 (e.g., random access memory (RAM), cache memory, flash memory, etc.), a storage device 603, I/O devices 604 such as a keyboard, a mouse, a microphone (not shown) or a monitor and network connectivity devices 605, which couple with each other with a bits 606.

[0042] The processor 601 may be implemented as a CPU chip, or may be part of an application specific integrated circuit (ASIC) and/or digital signal processor (DSP). The storage device 603 is typically a hard disk, an optical drive such as a compact disk drive or digital video disk (DVD) drive, a flash memory stick, etc and is used for non-volatile storage of data. The storage device 603 may be used to store programs that are loaded into memory 602 when such programs are selected for execution. The memory 602 is used to store instructions and perhaps data that are read during program execution.

[0043] The memory 602 includes, among others, a data transfer module 607 storing machine readable instructions which, when executed by the processor 601, cause the processor to perform the following operations. In an example, the operations include performing the method as explained with respect to Fig. 3, In another example, the data transmission system 500 as shown in FIG, 5 may be implemented as the data transfer module 607. Another example of the data transfer module 607 includes instructions that cause the processor 601 to realize the data transmission system 500 in FIG, 5 and the method illustrated in Fig. 3. [0044] Thanks to the SDN based data transfer solution described above, a dynamic network topology that flex-up and flex-down based on the data transfer requirement is realized. Due to such dynamic network topology, the following technical effects can be achieved: 1. Time involved to transmit the data is reduced. 2. The big data transmission doesn't cause any congestion on the network. 3. Data processing is performed efficiently. 4. On demand change in the network topology for effective RoL 5. Packet loss is reduced, 6. Dynamic network topology is realized to handle varying network load.

[0045] While the disclosure has been described with respect to a limited number of examples, those skilled in the art, having benefit of this disclosure, will appreciate that other example embodiments can be devised which do not depart from the scope of the disclosure as disclosed herein. Accordingly, the scope of the disclosure should be limited only by the attached claims.

Claims

I/We Claim:

1. A method for Software Defined Networking (SDN) based big data transfer, comprising: receiving, at a SDN controller in a network, an API request for big data transfer from an application, wherem the API request contains an amount of data that needs to be transferred and an identity of each node of a plurali ty of nodes to which the data is to be transferred; performing discovery of a network topology of the network to identify QoS parameters; based on the identified QoS parameters, activating a network template on the network topology to generate a changed network topology; and

enabling the data to be transferred over the changed network topology.

2. The method according to claim 1 , further comprising restoring the network topology after the big data transfer is completed.

3. The method according to claim 1, wherein the network template comprises routers and routing configuration for the nodes to which the data is to be transferred.

4. The method according to claim 3, wherein the network template further comprises virtual routers which can be created as a part of the network topology and can be destroyed on demand.

5. The method according to claim 1 , wherein the API request comprises at least one QoS parameter requirement, and the network template is created based on the at least one QoS parameter requirement.

6. A SDN based data transmission system, comprising:

a detection module to detect whether an API request for big data transmission is arrived from an application, wherein the API request contains an amount of data to be transmitted and an identi ty of each node of a plurality of nodes to which the data is to be transmi tted; a discovery module to cany out discovery of a network topology to determine QoS parameters;

an activation module to activate a network template on the network topology according to the determined QoS parameters to generate a changed network topology; and a transmission enable module to enable data transmission from the application to the changed network topology when the network template is activated on the network topology.

7. The system according to claim 6, further comprising:

a recovery module to recover the network topology after the big data transmission is completed.

8. The system according to claim 6, wherein the network template comprises routers and routing configuration for the nodes to which the data is to be transmitted.

9. The system according to claim 8, wherein the network template further comprises virtual routers which can be created as a part of the network topology and can be destroyed on demand.

10. The system according to claim 6, wherein the API request comprises at least one QoS parameter requirement, and the network template is created based on the at least one QoS parameter requirement.

1 1. An apparatus in a network comprising:

a processor; and

a memory storing computer readable instructions executable by the processor to:

receive an API request for big data transfer from an application, wherein the API. request contains an amount of data to be transferred and an identity of each node of a plurality of nodes to which the data is to be transferred;

perform discovery of a network topology of the network to identify QoS parameters; based on the identified QoS parameters, activate a network template on the network- topology to generate a changed network topology; andenable the data to be transferred over the changed network topology.

12. The apparatus according to claim 1 1, wherein the memory further stores instructions which, when executed by the processor, cause the processor to restore the network topology- after big data transfer is completed.

13. The apparatus according to claim i 1, wherein the network template comprises routers and routing configuration for the nodes to which the data is to be transferred.

14. The apparatus according to claim 13, wherein the network template further comprises virtual routers which can be created as a part of network topology and can be destroyed on demand.

15. The apparatus according to claim 1 1, wherein the API request comprises at least one QoS parameter requirement, and the network template is created based on the at least one QoS parameter requirement.