VNF Benchmarking Methodology

New paradigms of network services envisioned by NFV bring VNFs as software based entities, which can be deployed in virtualized environments . In order to be managed/orchestrated or compared with physical network functions, VNF Descriptors can specify performance profiles containing metrics (e.g., throughput) associated with allocated resources (e.g., vCPU). This document describes benchmarking methodologies to obtain VNF profiles (resource - performance figures).

The reader is assumed to be familiar with the terminology as defined in the European Telecommunications Standards Institute (ETSI) NFV document . Some of these terms, and others commonly used in this document, are defined below. Network Function Virtualization - The principle of separating network functions from the hardware they run on by using virtual hardware abstraction. NFV Infrastructure Point of Presence - Any combination of virtualized compute, storage and network resources. NFV Infrastructure - Collection of NFVI PoPs under one orchestrator. Virtualized Infrastructure Manager - functional block that is responsible for controlling and managing the NFVI compute, storage and network resources, usually within one operator's Infrastructure Domain (e.g. NFVI-PoP). NFV Orchestrator - functional block that manages the Network Service (NS) life-cycle and coordinates the management of NS life-cycle, VNF life-cycle (supported by the VNFM) and NFVI resources (supported by the VIM) to ensure an optimized allocation of the necessary resources and connectivity Virtualized Network Function - a software-based network function. Virtualised Network Function Descriptor - configuration template that describes a VNF in terms of its deployment and operational behaviour, and is used in the process of VNF on-boarding and managing the life cycle of a VNF instance. Virtualized Network Function Forwarding Graph - an ordered list of VNFs creating a service chain. Management and Orchestration - In the ETSI NFV framework , this is the global entity responsible for management and orchestration of NFV life-cycle. composition of Network Functions and defined by its functional and behavioural specification. Additional terminology not defined by ETSI NFV ISG. VNF Benchmarking Profile - the specification how to measure a VNF Profile. VNF-BP may be specific to a VNF or applicable to several VNF types. The specification includes structural and functional instructions, and variable parameters (metrics) at different abstractions (e.g., vCPU, memory, throughput, latency; session, transaction, tenants, etc.). is a mapping between virtualized resources (e.g., vCPU, memory) and VNF performance (e.g., throughput, latency between in/out ports) at a given NFVI PoP. An orchestration function can use the VNF Profile to select a host (NFVI PoP) for a VNF and to allocate necessary resources to deliver the required performance characteristics. A user/subscriber/consumer of ETSI's Network Service. Network Functions performing benchmarking tasks (e.g., synthetic traffic sources and sinks; measurement and observation functions, etc.). System Under Test comprises the VNF under test.

This document assumes VNFs as black boxes when defining VNF performance benchmarking methodologies. White box benchmarking of VNFs are left for further studies and may be added later.

We assume a VNF benchmarking set-up as shown in . Customers can request Network Services (NS) from an NFVO with associated service level specifications (e.g., throughput and delay). The NFVO, in turn, must select hosts and software resource allocations for the VNFs and build the necessary network overlay to meet the requirements. Therefore, the NFVO must know VNF Profiles per target hosts to perform location and resource assignments. In a highly dynamic environment, where both the VNF instances (e.g., revised VM image) and the NFVI resources (e.g., hw upgrades) are changing, the NFVO should be able to create VNF Profiles on-demand. We assume, that based on VNF Benchmarking Profile definitions NFVOs can run benchmarking evaluations to learn VNF Profiles per target hosts. In a virtualization environment, however, not only the SUT but all the other benchmarking agents may be software defined (physical or virtualized network functions). shows an example, where the NFVO can use PoPa and PoPb to set-up benchmarking functions to test VNFs hosted in PoP 1, 2, 3 domains corresponding to VIM 1, 2 and 3. The NFVO uses the VNF Benchmarking Profiles to deploy agents according to the SUT VNF. The VNF Benchmarking Profile is defined by the VNF Developer. The results of the VNF benchmarking is stored in a VNF Profile.

(VNF Benchmarking ) ( VNF-Profiles )<--->| NFVO / VNFM | \ Profiles / `---------------' | | `-------------' +-+----+----+-+ ____....----'/ | \---..__ ...----''' V V V ```--...__ +----+-+ +------+ +------+ +------+ +-+----+ | VIMa | | VIM1 | | VIM2 | | VIM3 | | VIMb | +-----++ +-+----+ +-+----+ +-+----+ +-----++ | | | | NFVI | +------+--+ *-------+--------+--------+--------* +------+--+ |PoPa | | | | | | |PoPb | |+------+ |SAP | +-----+-+ +---+---+ +-+-----+ | SAP| +------+| ||Agents|=|>O--+-| PoP 1 |--| PoP 2 |--| PoP 3 |--+--O>|=|Agents|| |+------+ | | +-------+ +-------+ +-------+ | | +------+| | | | PoP1 PoP2 PoP3 | | | | | | Container Enhanced Baremetal | | | | | | OS Hypervisor | | | +---------+ *----------------------------------* +---------+ ]]>

VNF benchmarking considerations are defined in . Additionally, VNF pre-deployment testing considerations are well explored in . This document list further considerations: In virtualization environments neither the VNF instance nor the underlying virtualization environment nor the agents specifics may be known by the entity managing abstract resources. This implies black box testing with black box functional components, which are configured by opaque configuration parameters defined by the VNF developers or alike for the benchmarking entity (e.g., NFVO).

Following the ETSI's model (), we distinguish three methods for VNF evaluation: Where resource {cpu, memory, storage} parameters are provided and the corresponding {latency, throughput} performance parameters are obtained. Note, such request might create multiple reports, for example, with minimal latency or maximum throughput results. Both resources {cpu, memory, storage} and performance {latency, throughput} parameters are provided and agents verifies if the given association is correct or not. Where performance parameters {latency, throughput} are provided and the corresponding {cpu, memory, storage} resource parameters obtained. Note, multiple deployment interactions may be required, or if possible, underlying allocated resources need to be dynamically altered. Note: Verification and Dimensioning can be reduced to Benchmarking. Therefore, we detail Benchmarking in what follows.

All benchmarking methodologies described in this section consider the definition of VNF-BPs for each testing procedure. Information about Benchmarking Methodology for Network Interconnect Devices, defined in , is considered in all subsections below. Besides, the tests are defined based on notions introduced and discussed in the IP Performance Metrics (IPPM) Framework document .

Provide, for a particular set of resources allocated, the throughput among two or more VNF ports, expressed in VNF-BP. VNF (SUT) must be deployed and stable and its allocated resources collected. VNF must be reachable by agents. The frame size to be used for agents must be defined in the VNF-BP. Establish connectivity between agents and VNF ports. Agents initiate source of traffic, specifically designed for VNF test, increasing rate periodically. Throughput is measured when traffic rate is achieved without frame losses. report must contain VNF allocated resources and throughput measured (aka throughput in ).

Provide, for a particular set of resources allocated, the latency among two or more VNF ports, expressed in VNF-BP. VNF (SUT) must be deployed and stable and its allocated resources collected. VNF must be reachable by agents. The frame size and respective throughput to be used for agents must be defined in the VNF-BP. Establish connectivity between agents and VNF ports. Agents initiate source of traffic, throughput and frame size specifically designed for VNF test. Latency is measured when throughput is achieved for the period of time specified in VNF-BP. report must contain VNF allocated resources, throughput used for stimulus and latency measurement (aka latency in ).

Provide, for a particular set of resources allocated, the frame loss rate among two or more VNF ports, expressed in VNF-BP. VNF (SUT) must be deployed and stable, its allocated resources collected specifying any particular feature of the underlying VNF virtualized environment, provided by NFVO/VIM or independently extracted. VNF must be reachable by agents. Rate of source traffic and frame type used for agents stimulus must be defined in VNF-BP. Establish connectivity between agents and VNF ports. Agents initiate source of traffic, specifically designed for VNF test, achieving rate of source traffic defined in VNF-BP. Frame loss rate is measured when pre-defined traffic rate is achieved for period of time established in VNF-BP. report must contain VNF allocated resources, rate of source traffic used as stimulus and frame loss rate measurement (aka frame loss rate in ).

This document describes black-box benchmarking methodologies for black-box VNFs in virtualization environments (e.g., ETSI NFV framework) to create VNF Profiles containing the association of resources and performance metrics of a given VNF at a given host (e.g., NFVI PoP). The authors see the following next steps: Two scaling options: single instance with more resources or multiple instances. Questions: What is the maximum performance of a single instance VNF at a given host with increasing resources? How many independent VNF instances (or components) can be run with maximum performance at a given host? On the other hand, what is the performance of the smallest resource footprint VNF allocation? this metric concerns at least three components: VNF bootstraping (SUT), execution environment and the orchestration process.

This memo includes no request to IANA.

TBD

The authors would like to thank the support of Ericsson Research, Brazil. This work is partially supported by FP7 UNIFY, a research project partially funded by the European Community under the Seventh Framework Program (grant agreement no. 619609). The views expressed here are those of the authors only. The European Commission is not liable for any use that may be made of the information in this document.