Luận án Các phương pháp tiết kiệm năng lượng sử dụng công nghệ mạng điều khiển bằng phần mềm trong môi trường điện toán đám mây

MINISTRY OF EDUCATION AND TRAINING
HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY
TRAN MANH NAM
CÁC PHƯƠNG PHÁP TIẾT KIỆM NĂNG LƯỢNG SỬ DỤNG CÔNG NGHỆ MẠNG ĐIỀU KHIỂN BẰNG PHẦN MỀM TRONG MÔI TRƯỜNG ĐIỆN TOÁN ĐÁM MÂY
SDN-BASED ENERGY-EFFICIENT NETWORKING IN CLOUD COMPUTING ENVIRONMENTS
DOCTORAL THESIS OF TELECOMMUNICATIONS ENGINEERING
HANOI - 2018
MINISTRY OF EDUCATION AND TRAINING
HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY
TRAN MANH NAM
CÁC PHƯƠNG PHÁP TIẾT KIỆM NĂNG LƯỢNG SỬ DỤNG CÔNG NGHỆ MẠNG ĐIỀU KHIỂN BẰNG PHẦN MỀM TRONG MÔI TRƯỜNG ĐIỆN TOÁN ĐÁM MÂY
SDN-BASED ENERGY-EFFICIENT NETWORKING IN CLOUD COMPUTING ENVIRONMENTS
Specialization: Telecommunications Engineering
Code No: 62520208
DOCTORAL THESIS OF TELECOMMUNICATIONS ENGINEERING
Supervisor: Assoc.Prof. Nguyen Huu Thanh
HANOI - 2018
PREFACE
I hereby assure that the results presented in this dissertation are my work under the guidance of my supervisor. The data and results presented in the dissertation are completely honest and have not been disclosed in any previous works. The references have been fully cited and in accordance with the regulations. 
Tôi xin cam đoan các kết quả trình bày trong luận án là công trình nghiên cứu của tôi dưới sự hướng dẫn của giáo viên hướng dẫn. Các số liệu, kết quả trình bày trong luận án là hoàn toàn trung thực và chưa được công bố trong bất kỳ công trình nào trước đây. Các kết quả sử dụng tham khảo đều đã được trích dẫn đầy đủ theo đúng quy định.
Hà Nội, Ngày 19 tháng 01 năm 2018
Tác giả
Trần Mạnh Nam
ACKNOWLEDGEMENTS
First and foremost, I would like to thank my advisor, Associate Prof. Dr. Nguyen Huu Thanh, for providing an excellent researching atmosphere, for his valuable comments, constant support and motivation. His guidance helped me in all the time and also in writing this dissertation. I could not have thought of having a better advisor and mentor for my PhD. 
Moreover, I would like to thank Associate Prof. Dr. Pham Ngoc Nam, Dr. Truong Thu Huong for their advices and feedbacks, also for many educational and inspiring discussions. 
My sincere gratitude goes to the members (present and former) of the Future Internet Lab, School of `Electronics and Telecommunications, Hanoi University of Science and Technology. Without their support and friendship it would have been difficult for me to complete my PhD studies.
Finally, I would like to express my deepest gratitude to my family. They are always supporting me and encouraging me with their best wishes, standing by me throughout my life.
Hanoi, 19th Jan 2018
CONTENTS
ABBREVIATIONS
APCI
Advanced Configuration & Power Interface
APEX
Capital expenditure
ASIC
Application specific integrated circuits
BAU
Business-as-usual
BFS
Breadth-first Search
CAPEX
Capital Expenditure 
DC
Data center
DCN
Data center network
D-ITG
Distributed internet traffic generator
EA-NV
Energy-aware network virtualization
EA-VDC
Energy-aware Virtual Data Center
ECO
Eco sustainable
FM
Full migration
FPGA
Field programmable gate arrays
GH
GreenHead
HEA-E
Heuristic Energy-aware VDC Embedding
HEE
Heuristic energy-efficient
IaaS
Infrastructure-as-a-service
ICT
Information and communication technologies
ISP
Internet service provider
MoA
Migrate on arrival
MST
Minimum spanning tree
NaaS
Network-as-a-service
NFV
Network function virtualization
NV
Network virtualization
OLD
OpenDayLight
OPEX
Operating expenses
PaaS
Platform-as-a-service
PCS
Power-Control System
PM
Partial migration
POD
Optimized data centers
PSnEP
Power scaling and energy-profile-aware
RMD-EE
Reducing middle node energy efficiency
SaaS
Software-as-a-service
SDSN
Software-Defined Substrate Network
SN
SecondNet
SNMP
Simple network management protocol
TCAM
Ternary content-addressable memory
VDC
Virtual data center
VDCE
Virtual data center embedding
VLiM
Virtual link mapping
VM
Virtual Machine
VmM
Virtual machine mapping
VNE
Virtual network embedding
VNoM
Virtual node mapping
VNR
Virtual network requests
LIST OF FIGURES
Figure 1.1: Estimate of the global carbon footprint of ICT (including PCs, telcos’ networks and devices, printers and datacenters) [15].	7
Figure 1.2: Energy consumption estimation for the European telcos’ network infrastructures in the”Business-As-Usual” (BAU) and in the Eco-sustainable (ECO) scenarios, and cumulative energy savings between the two scenarios [16].	7
Figure 1.3: Operating Expenses (OPEX) estimation related to energy costs for the European telcos’ network infrastructures in the ”Business-As-Usual” (BAU) and in the Eco-sustainable (ECO) scenarios, and cumulative savings between the two scenarios [17]	8
Figure 1.4: SDN Architecture	11
Figure 1.5: OpenFlow controller and switches	12
Figure 2.1: DCN Architecture [43]	18
Figure 2.2: Three-tier DCN Architecture [45]	18
Figure 2.3: Fat-tree DCN Topology	19
Figure 2.4: Dcell DCN Architecture [53]	19
Figure 2.5: BCube DCN Architecture [54]	20
Figure 2.6: Fat-tree architecture with k = 4	21
Figure 2.7: Diagram of the ElasticTree system [57]	22
Figure 2.8: Energy – Utilization relation of a network [58]	23
Figure 2.9: Power-control System of a Network	26
Figure 2.10: Fat-tree topology with Minimum Spanning Tree	28
Figure 2.11: Power Scaling Algorithm	32
Figure 2.12: Power Scaling and Energy-Profile-Aware - PSnEP algorithm (Proposed Algorithm 1). The flowchart describes the process between Edge and Aggregation switches	34
Figure 2.13: use-case with PSnEP algorithm in a DCN	35
Figure 2.14: PSnEP vs Power scaling (PS) with k=6 Fat-tree, mix scenario	38
Figure 2.15: Energy-saving level ratio of the PSnEP algorithm to the PS algorithm in different sizes	39
Figure 2.16: Extended Power-Control system (Ext-PCS)	40
Figure 2.17: Example	42
Figure 2.18: First-fit Migration [67] Algorithm	42
Figure 2.19: Topology-Aware Placement Algorithm	43
Figure 2.20: K=8, comparison with full mesh scenario	46
Figure 2.21: K=16, comparison with full mesh scenario	47
Figure 2.22: K=8, comparison with Honeyguide	47
Figure 2.23: K=16, comparison with Honeyguide	48
Figure 3.1: FlowVisor – Hypervisor-like Network Layer [71]	50
Figure 3.2: Example of a virtual network on top of a physical network	51
Figure 3.3: Energy-Aware Network Virtualization system’s Diagram	52
Figure 3.4: Online VNE mapping method	57
Figure 3.5: Online using Time Window method	58
Figure 3.6: The GUI of an Energy-aware network virtualization platform	64
Figure 3.7 AR– Online	65
Figure 3.8: AR – Online using Time Windows	65
Figure 3.9: Percentage of Power Consumption to Full State in Online Strategy	65
Figure 3.10 Percentage of Power Consumption to Full State in OuTW Strategy	65
Figure 3.11: Comparison of comsumed energy between Online and OuTW strategies	66
Figure 3.12: Comparison of acceptance ratio between Online and OuTW strategies	66
Figure 4.1: Traditional cloud service provider vs NaaS	68
Figure 4.2: Embedding virtual data center requests on a physical data center	70
Figure 4.3: Virtual data center embedding - Static mapping;	72
Figure 4.4: Virtual data center embedding - Dynamic mapping	72
Figure 4.5: Energy proportional property of energy-aware data centers	73
Figure 4.6: Energy-Aware VDC Architecture	78
Figure 4.7: VDC Embedding Flowchart	79
Figure 4.8: Flowchart of Partial Migration (PM)	83
Figure 4.9: Migration on Arrival	84
Figure 4.10: Fluctuation of system utilization (SecondNet)	86
Figure 4.11: DC Utilization per Load	87
4.12: Acceptance Ratio per VM	87
Figure 4.13: Acceptance Ratio per VDC	88
Figure 4.14: Total power consumption of the physical DC	88
Figure 4.15: Average consumed power per serving VDC	89
Figure 4.16: Number of migrations for different strategies	90
Figure 4.17: Comparison of embedding - migration strategies	90
4.18: Different embedding-magrition strategies: (a) GreenHead, (b) SecondNet, (c) Partial Migration, (d) Migration on Arrival, (e) Full Migration	91
LIST OF TABLES
Table 1.1: The Internet’s users in the world [1]	6
Table 1.2: Estimated power consumption sources in a generic platform of IP router	8
Table 1.3: Classification of energy-efficient approaches of the future Internet [4]	9
Table 2.1: Power Summary For A 48-Port Pronto 3240	30
Table 2.2: Energy consumption of NetFPGA-Based OpenFlow Switch	31
Table 2.3: Energy-saving ratio of PSnEP to Power scaling algorithm in different topology’s sizes	39
Table 2.4: Traffic demand	41
Table 2.5: Power profile of server Dell PowerEdge R710	46
Table 3.1: Virtual Network Embedding Terminology	54
Table 3.2: Acceptance ratio and power consumption of the system under different window size in OuTW	67
Table 4.1: Standard deviation of system utilization	86
INTRODUCTION
Overview of Network Energy Efficiency in Cloud Computing Environments
The advances in Cloud Computing services as well as Information and Communication Technologies (ICT) in the last decades have massively influenced economy and societies around the world. The Internet infrastructure and services are growing day by day and play a considerable role in all aspects including business, education as well as entertainment. In the last four years, the percentage of people using Internet witnesses an annual growth of 3.5%, from 39% world population’s percentage in Dec-2013 to 51.7% in June-2017 [1]. 
To support the demand of cloud network infrastructure and Internet services in the rapid growth of users, it is necessary for the Internet providers to have a large number of devices, complex design and architecture that have the capacity to perform increasingly number of operations for a scalability. Consequently, many huge cloud infrastructures have been employed by Telcos, Internet Service Providers (ISPs) and enterprises for the exploded demand of various applications and data cloud-services such as YouTube, Dropbox, 
e-learning, cloud office etc. To meet the requirements of these booming services all around the world, cloud network infrastructures have been built up in a very large scale, even geographically distributed data centers with a huge number of network devices and servers. In addition, the maintenance of the systems with high availability and reliability level requires a notable redundancy of devices such as routers, switches, links etc. As a result, having such a large infrastructure consumes a huge volume of energy, which leads to consequent environmental and economic issues:
Environmentally, the amount of energy consumption and carbon footprint of the 
ITC-sector is remarkable. The manufacture of ICT equipment is estimated its use and disposal account for 2% of global CO2 emissions, which is equivalent to the contributions from the aviation industry [2]. The networking devices and components estimate around 37% of the total ICT carbon emission [3]; 
Economically, the huge consumed power leads to the costs sustained by the providers/operators to keep the network up and running at the desired service level and their need to counterbalance ever-increasing cost of energy.
Although network energy efficiency has recently attracted much attention from communities [4], there are still many issues in realization of the energy-efficient network including inflexibility and the lack of an energy-aware network. The main difficulties of the network energy efficiency as well as its research motivations are shortly described as follows: 
Inflexible network: first, one important point the network in cloud data centers (DC) nowadays is the inflexibility issue. For changing the processing algorithm and the control plane of a network, its administrators should carefully re-design, 
re-configure and migrate the network for a long time. In many cases, there is a technical challenge for an administrator to apply new approaches and evaluate their efficiency. Consequently, the flexible and programmable network is strictly necessary. Secondly, there are difficulties in evaluating the energy-saving levels of new energy-efficient approaches in a network due to the lack of the centralized power-control system. This system allows administrators and developers to monitor, control and managing the working states as well as power consumption of all network devices in real-time.
Energy-aware networking for virtualization technologies in cloud environments: cloud computing has emerged in the last few years as a promising paradigm that facilitates such new service models as Infrastructure-as-a-Service (IaaS), Storage-as-a-Service (SaaS), Platform-as-a-Service (PaaS), Network-as-a-Service (NaaS). For such kinds of cloud services, virtualization techniques including network virtualization [5] [6] [7] and data center virtualization [8] [9] [10] have quickly developed and attracted much attention of research and industrial communities. Currently, research in virtualization technologies mainly focuses on the resource optimization and resource provisioning approaches [8] [9]. There are very few works focusing on the energy efficiency of a network. With the benefits of flexible controlling and resource management of virtualization technologies as well as new network technologies such as Software-defined Networking (SDN) [11] [12] [13], researching in network energy efficiency in virtualization is an important and promising approach.
Additionally, the SDN technology, the emergence of new trends in networking technology, provides new way to realize and optimize network energy efficiency. Software-defined networking [11] aims to change the inflexible state networking, by breaking vertical integration, separating the network’s control logic from the underlying routers and switches, promoting (logical) centralization of network control, and introducing the ability to program the network. Consequently, SDN is an important key for resolving aforementioned difficulties.
Research Scope and Methodology
Research Scope
The scope of this research focuses on the network energy efficiency in cloud computing environments, including: (1) energy efficiency in centralized data center network; (2) energy efficiency in network virtualization; and (3) energy efficiency in data center virtualization. The proposed energy-efficient approaches are based on the Software-defined Networking technology [11] [12] [13].
Research Methodology: the research methodology is used following the reference [14].
Step 1: Problem formulation:
Interrogative form
Describe relations among constructs
Step 2: Hypothesis formulation: answering to problem statements
Step 3: Research design: building research plan for a research process including survey, related work and experiments
Step 4: Sampling and Data Collection
Step 5: Data analysis
Step 6: Manuscript Writing
Contributions and Structure of the Dissertation
Recently, Software-defined Networking technology [5] is likely an evolutionary step in Internet technologies that makes networking become more flexible and programm ...  no. 3, pp. pp. 243-252, Feb 2009. 
[20] 
D. Neilson, "Photonics for Switching and Routing," IEEE Journal of Selected Topics in Quantum Electronics (JSTQE), vol. 12, no. 4, pp. 669-678, July-Aug. 2006.. 
[21] 
Bianzino, A. P., Chaudet, C., Rossi, D., & Rougier, J., "A Survey of Green Networking Research," IEEE Communications Surveys Tutorials, vol. 14, no. 1, 2012. 
[22] 
Barr, Keith, ASIC Design in the Silicon Sandbox: A Complete Guide to Building Mixed-Signal Integrated Circuits, McGraw Hill Professional, 2007. 
[23] 
Józef KORBICZ et al., Synthesis of Compositional Microprogram Control Units for Programmable Devices, Poland: University of Zielona Góra Press, Poland, 2009. 
[24] 
J. Baliga, R. Ayre, K. Hinton, and R. S. Tucker, "Photonic Switching and the Energy Bottleneck," in Internat. Conf. Photonics in Switching 2007, San Francisco, CA, USA, 2007. 
[25] 
L. G. Roberts, "A Radical New Router," IEEE Spectrum, vol. 46, no. 7, pp. 34-39, 2009. 
[26] 
M. Baldi, Y. Ofek, "Time for a ”Greener” Internet,”," in Proc. Green Communications Workshop in conjunction with IEEE ICC’09 (GreenComm09), Dresden, Germany, 2009. 
[27] 
J. Noguera, I.O. Kennedy, "Power Reduction in Network Equipment Through Adaptive Partial Reconfiguration," in Internat. Conf. on Field Programmable Logic and Applications (FPL 2007), Amsterdam, Netherlands, 2007. 
[28] 
M. B. Srivastava, A.P. Chandrakasan, R.W. Brodersen, "Predictive System Shutdown and Other Architectural Techniques for Energy Efficient Programmable Computation," IEEE Trans. Very Large Scale Integr. (VLSI), vol. 4, no. 1, pp. 42-55, 1996. 
[29] 
M. Allman, K. Christensen, B. Nordman, V. Paxson, "Enabling an Energy-Efficient Future Internet Through Selectively Connected End Systems," in Proc. ACM SIGCOMM HotNets Workshop (HotNets 07), Atlanta, GA, USA, 2007. 
[30] 
M. Jimeno, K. Christensen, B. Nordman, "A Network Connection Proxy to Enable Hosts to Sleep and Save Energy," in Proc. IEEE Internat. Performance Computing and Communications Conf, Austin, Texas, USA, 2008. 
[31] 
Han, B., Gopalakrishnan, V., Ji, L., & Lee, S., "Network function virtualization: Challenges and opportunities for innovations," IEEE Communications Magazine, vol. 53, no. 2, p. 90–97, 2015. 
[32] 
Correa, E. S., Fletscher, L. A., & Botero, J. F., "Virtual Data Center Embedding: A Survey.," IEEE Latin America Transactions, vol. 13, no. 5, p. 1661–1670, 2015. 
[33] 
B. Raghavan et al, "Software-defined internet architecture: Decoupling architecture from infrastructure," in Proc. 11th ACM Workshop Hot Topics Network, Redmond, Washington, USA, 2012. 
[34] 
N. McKeown et al, "OpenFlow: Enabling innovation in campus networks," SIGCOMM Comput. Commun. Rev, vol. 38, no. 2, p. 69–74, 2008. 
[35] 
"Open Networking Foundation (ONF)," 2014. [Online]. Available: https://www.. [Accessed Jan 2015].
[36] 
Gude, N., Koponen, T., Pettit, J., Pfaff, B., Casado, M., McKeown, N., & Shenker, S., "NOX: Towards an Operating System for Networks," ACM SIGCOMM Computer Communication Review, vol. 38, no. 3, pp. 105-111, 2008. 
[37] 
"POX," [Online]. Available:  about-pox/. [Accessed Jan 2014].
[38] 
D. Erickson, "Beacon Home," [Online]. Available: https://openflow.stanford.edu/display/Beacon/Home/. [Accessed May 2015].
[39] 
Floodlight. [Online]. Available: Available:  [Accessed May 2015].
[40] 
"The OpenDaylight Platform," OpenDaylight, [Online]. Available: https://www.opendaylight.org/. [Accessed Dec 2015].
[41] 
G. V. N. T. i. O.-. . ‐. B. N. Roteiro, "Efficient Resource Provisioning Using Virtualization Technology in Cloud Environment," International Journal of Innovative Research in Science, Engineering and Technology, vol. 3, no. 3, pp. 2200-2205, 2014, Oct. 
[42] 
Theophilus Benson, Aditya Akella and David A. Maltz, "Network Traffic Characteristics of Data Centers in the Wild," in IMC 10, Melbourne, Australia., 2010. 
[43] 
Z. Q. R. M. H. G. A. V. V. Bin Wang, "A survey on data center networking for cloud computing," Computer Networks, vol. 91, p. 528–547, 2015. 
[44] 
A. M. N. T. Brian Lebiednik, "A Survey and Evaluation of Data Center Network Topologies," Distributed, Parallel, and Cluster Computing, 2016. 
[45] 
D. Kliazovich, P. Bouvry, S.U. Khan, "GreenCloud: a packet-level simulator for energy-aware cloud computing data centers," Journal of Supercomputing, vol. 62, no. 3, p. 1263–1283, 2012. 
[46] 
"Cisco Data Center Infrastructure 2.5," Design Guide, Cisco Validated Design, March 9, 2010. 
[47] 
S. K. L. Z. H. L. K. H. S. M. N. M.-A. L. W. a. D. C. K. Bilal, "Quantitative Comparisons of the State of the Art Data Center Architectures," Concurrency and Computation: Practice and Experience, vol. 25, no. 12, pp. 1771-1783, 2013. 
[48] 
C. E. Leiserson, "Fat-trees: universal networks for hardware-efficient supercomputing," IEEE Transactions on Computers, vol. 100, no. 10, pp. 892-901, 1985. 
[49] 
M. Al-Fares, A. Loukissas, and A. Vahdat, "A scalable, commodity data center network architecture," ACM SIGCOMM Computer Communication Review, vol. 38, no. 4, p. 63–74, 2008. 
[50] 
Greenberg, A., Hamilton, J. R., Jain, N., Kandula, S., Kim, C., Lahiri, P., Sengupta, S., "VL2: A Scalable and Flexible Data Center Network.," ACM SIGCOMM Computer Communication Review, vol. 39, no. 4, 2009. 
[51] 
Andreyev., A., "Introducing data center fabric, the next-generation facebook data center network.," Facebook, Nov 2014. [Online]. Available: https://code.facebook.com/posts/360346274145943/. [Accessed Jan 2017].
[52] 
A. Singh, J. Ong, A. Agarwal, G. Anderson, A. Armistead, R. Bannon, S. Boving, G. Desai, B. Felderman, P. Germano et al.,, "Jupiter rising: A decade of clos topologies and centralized control in google’s datacenter network," in ACM Conference on Special Interest Group on Data Communication, 2015. 
[53] 
C. Guo, H. Wu, K. Tan, L. Shi, Y. Zhang, and S. Lu, "Dcell: a scalable and fault-tolerant network structure for data centers," ACM SIGCOMM Computer Communication Review, vol. 38, no. 4, p. 75–86, 2008. 
[54] 
C. Guo, G. Lu, D. Li, H. Wu, X. Zhang, Y. Shi, C. Tian, Y. Zhang, and S. Lu, "Bcube: a high performance, server-centric network architecture for modular data centers," ACM SIGCOMM Computer Communication Review, vol. 39, no. 4, p. 63–74, 2009. 
[55] 
A. Singla, C.-Y. Hong, L. Popa, P. B. Godfrey, "Jellyfish: Networking data centers randomly," in The 9th USENIX Conference on Networked Systems Design and Implementation, CA, USA, 2012. 
[56] 
Trinh, L. Gyarmati and T. A., "Scafida: A scale-free network inspired data center architecture," ACM SIGCOMM Computer Communication Review, vol. 40, no. 5, pp. 4-12, 2010.. 
[57] 
Heller, B., Seetharaman, S., Mahadevan, P., Yiakoumis, Y., Sharma, P., Banerjee, S., & McKeown, N., "ElasticTree : Saving Energy in Data Center Networks," in NSDI'10 Proceedings of the 7th USENIX conference, CA, USA, 2010. 
[58] 
Pham Ngoc, N., Nguyen Huu, T., Vu Quang, T., Tran Hoang, V., Truong Thu, H., Tran-Gia, P., & Schwartz, C., "A new power profiling method and power scaling mechanism for energy-aware NetFPGA gigabit router.," Computer Networks, vol. 78, p. 4–25, 2015. 
[59] 
Mahadevan, P., Sharma, P., Banerjee, S., & Ranganathan, P., "Energy Aware Network Operations.," in IEEE INFOCOM Workshops, 2009. 
[60] 
"NetFPGA Gigabit Card," netFPGA, 2015. [Online]. Available:  [Accessed Dec 2016].
[61] 
Wang, X., Yao, Y., Wang, X., Lu, K., & Cao, Q., "CARPO: Correlation-aware power optimization in data center networks," in In 2012 Proceedings IEEE INFOCOM, 2012. 
[62] 
J. Case, M. Fedor, M. Schoffstall and D. J., "RFC 1157 - A Simple Network Management Protocol (SNMP)," 1990. [Online]. Available: https://tools.ietf.org/html/rfc1157.
[63] 
K. W. a. Y.-H. H. Yi-Chih Lei, "Multipath Routing in SDN-based Data Center Networks," in 2015 European Conference on Networks and Communications (EuCNC) , Paris, France, 2015. 
[64] 
D. P. Omair Fatmi, "Distributed multipath routing for data center networks based on stochastic traffic modeling," in Networking, Sensing and Control (ICNSC), Miami, FL, USA, 2014. 
[65] 
"D-ITG, Distributed Internet Traffic Generator," 2017. [Online]. Available: 
[66] 
Fabien Hermenier, Xavier Lorca, Jean-Marc Menaud, Gilles Muller, Julia Lawall, "Entropy: a Consolidation Manager for Clusters," ACM SIGPLAN/SIGOPS, 2009. 
[67] 
Hiroki Shirayanagi, Hiroshi Yamada, and Kenji Kono, "Honeyguide: A VM Migration-Aware Network Topology for Saving Energy Consumption in Data Center Networks," ISCC 2012 IEEE, Cappadocia, Turkey., July 2012.
[68] 
V. De Maio, R. Prodan, S. Benedict, G. Kecskemeti, "Modelling energy consumption of network transfers and virtual machine migration, Modelling energy consumption of network transfers and virtual machine migration," Future Generation Computer Systems, vol. 56, p. 388–406, 2016. 
[69] 
Chowdhury, N.K., & Boutaba, R. et al., "Network virtualization: state of the art and research challenges," Communications Magazine, IEEE, vol. 47, no. 7, p. 20–26, 2009. 
[70] 
Kleinberg, J., "Approximation algorithms for disjoint paths problems," MIT, Massachusetts Institute of Technology, 1996.
[71] 
R. Sherwood, G. Gibb, K. Yap, G. Appenzeller, N. McKeown, and G. Parulkar, "FlowVisor: A network virtualization layer," in OpenFlow Switch Consortium - OPENFLOW-TR-2009, 2009. 
[72] 
Rob Sherwood, Michael Chan,Glen Gibb, Nikhil Handigol,Te-Yuan Huang,Peyman Kazemian, Masayoshi Kobayashi, David Underhill, Kok-Kiong Yap, Guido Appenzeller, and Nick McKeown, "Carving Research Slices Out of Your Production Networks with OpenFlow," Newsletter ACM SIGCOMM Computer Communication, vol. 40, no. 1, 2010. 
[73] 
"Exploring networks of the future," GENI, [Online]. Available:  [Accessed Dec 2015].
[74] 
"Applications, Linking Infrastructure and Ofelia,," OpenFlow in Europe, [Online]. Available:  [Accessed dec 2016].
[75] 
"OF@TEIN - OpenFlow@Trans-Eurasian Information Network," [Online]. Available:  [Accessed Dec 2016].
[76] 
Botero, J. F., Hesselbach, X., Fischer, A., & de Meer, H., "Optimal mapping of virtual networks with hidden hops," Telecommunication Systems, vol. 51, no. 4, p. 273–282, 2013. 
[77] 
Andreas Fischer, Juan Felipe Botero, Michael Till Beck, Hermann de Meer, and Xavier Hesselbach, "Virtual Network Embedding: A Survey," IEEE COMMUNICATIONS SURVEYS & TUTORIALS, vol. 15, no. 4, 2013. 
[78] 
M. Y. Y. R. J. &. C. M. Yu, "Rethinking virtual network embedding," ACM SIGCOMM Computer Communication Review, vol. 38, no. 2, p. 17, 2008. 
[79] 
B. M. Waxman, "Routing of multipoint connections," IEEE J. Sel. Areas Communication, vol. 6, no. 9, pp. 1617-1622, , 1988. 
[80] 
Skiena, S. S., "The Algorithm Design Manual, 2nd Edition," Springer Pub- lishing Company, 2008.
[81] 
A. B. J. D. M. S. D. H. X. D. M. H. Fischer, "ALEVIN - A framework to develop, compare, and analyze virtual network embedding algorithms," Electronic Communications of the EASST, vol. 37, pp. 1-12, 2011. 
[82] 
P. Costa, M. Migliavacca, P. Pietzuch, A. L. Wolf, "NaaS: Network-as-aService in the cloud," in The 2nd USENIX Workshop on Hot Topics in Management of Internet, Cloud, and Enterprise Networks and Services, USENIX, San Jose, CA, USA, 2012. 
[83] 
Fischer, A., Botero, J. F., Beck, M. T., de Meer, H., Hesselbach, X., Meer, H. De,  Hesselbach, X., "Virtual network embedding: A survey," IEEE Communications Surveys and Tutorials, vol. 15, no. 4, p. 1888–1906, 2013. 
[84] 
Yoonseon Han, Jian Li, Jae-Yoon Chung, Jae-Hyoung Yoo, & Hong, J. W.-K., "SAVE: Energy-aware Virtual Data Center embedding and Traffic Engineering using SDN," in In Proceedings of the 2015 1st IEEE Conference on Network Softwarization (NetSoft), London, UK, 2015. 
[85] 
Guo, C., Lu, G., Wang, H. J. H. J., Yang, S., Kong, C., Sun, P.,  Zhang, Y., "SecondNet: a data center network virtualization architecture with bandwidth guarantees," in Proceedings of the 6th International COnference, (VDC, Pennsylvania, USA, 2010. 
[86] 
A. Amokrane, M. F. Zhani, R. Langar, R. Boutaba, G. Pujolle, "Greenhead: Virtual data center embedding across distributed infrastructures," IEEE Transactions on Cloud Computing, vol. 1, no. 1, p. 36–49, 2013. 
[87] 
H. Goudarzi, M. PedramH. Goudarzi, M. Pedram, "Energy-efficient virtual machine replication and placement in a cloud computing system," in IEEE Fifth International Conference on Cloud Computing, Honolulu, HI, USA, 2012. 
[88] 
F. Farahnakian, P. Liljeberg, J. Plosila, "Energy-efficient virtual machines consolidation in cloud data centers using reinforcement learning," in 22nd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing, Torino, Italy, 2014. 
[89] 
A. Beloglazov, R. Buyya, Y. C. Lee, A. Y. Zomaya, "A taxonomy and survey of energy-efficient data centers and cloud computing systems," Advances in Computers 82 (2011) 47 – 111, pp. 47-111, 2011. 
[90] 
"OpenStack," [Online]. Available: https://www.openstack.org. [Accessed Dec 2016].
[91] 
J. H. D. A. M. P. P. A. Greenberg, "The cost of a cloud: Research problems in data center networks," ACM SIGCOMM Computer Communication Review, vol. 39, no. 1, p. 68–73, 2009. 
[92] 
A. Desai, R. Oza, P. Sharma, and B. Patel, "Hypervisor : A Survey on Concepts and Taxonomy," no. 3, vol. 3, pp. 222 - 225, 2013. 
[93] 
"CT503-MIX High-Speed LANforge-FIRE Traffic Generator," Candela Technology, 2017. [Online]. Available:  [Accessed 2017].
[94] 
A. Al-Shabibi, M. De Leenheer, M. Gerola, A. Koshibe, G. Parulkar, E. Sal- vadori, B. Snow, "OpenVirteX: Make your virtual SDNs programmable," in Proceedings of the Third Workshop on Hot Topics in Software Defined Networking, HotSDN ’14, ACM, New York, NY, USA, 2014. 
[95] 
D. Schwerdel, D. Gunther, R. Henjes, B. Reuther, and P. Muller, "German-lab experimental facility in Future Internet," FIS 2010, ser.Lecture Notes in Computer Science - Springer-Verlag Belin Heidelberg, vol. 6369, p. 1, 2010. 
[96] 
H. T. Nguyen, A. V. Vu, D. L. Nguyen, V. H. Nguyen, M. N. Tran, Q. T. Ngo, T. H. Truong, T. H. Nguyen, T. Magedanz, "A generalized resource allocation framework in support of multi-layer virtual network embedding based on SDN, Computer Networks," Computer Networks, vol. 92, no. 2, pp. 251 - 269, 2015. 
[97] 
L. Ceuppens, A. Sardella, D. Kharitonov, "Power Saving Strategies and Technologies in Network Equipment Opportunities and Challenges, Risk and Rewards," in Proc. Internat. Symp. on Applications and the Internet (SAINT 2008), Turku, Finland, 2008. 
[98] 
Singh, M. Gupta and S., "Greening of the Internet," in ACM SIGCOMM Conf. (SIGCOMM 03, Karlshue, Germany, Aug-2003. 
[99] 
R. D. Corin, M. Gerola, R. Riggio, F. D. Pellegrini, E. Salvadori, "VeRTIGO: Network virtualization and beyond," in European Workshop on Software Defined Networking, Darmstadt, Germany, 2012. 
[100] 
Mohsen, Ehab, "Reducing System Power and Cost," XILINX White Paper: Artix-7 FPGAs, Sep-2016.