Interfaces

The BNG Blaster distinguishes between interface links and interface functions. An interface link can be considered as the actual interface with all the corresponding IO settings. This is similar but not the same as physical interfaces in typical router implementations. Interface functions are similar to logical interfaces and define the actual services. One or more interface functions can be attached to each interface link.

At least one interface function is required to start the BNG Blaster.

Operating System Settings

The BNG Blaster implements all protocols in user space. Therefore the used interfaces links must not have an IP address configured in the host operating system, to prevent that the received packets are handled or even responded to from the Linux kernel as well.

All used interface links must be in an operational state up.

sudo ip link set dev <interface> up

It is not possible to send packets larger than the configured interface MTU, which is 1500 bytes per default. For PPPoE with multiple VLAN headers, this might be not enough for large packets. Therefore the interface MTU should be increased using the following commands.

sudo ip link set mtu 9000 dev <interface>

All this can be also archived via netplan using the following configuration for each BNG Blaster interface link.

network:
version: 2
renderer: networkd
ethernets:
    eth1:
    dhcp4: no
    dhcp6: no
    link-local: []
    mtu: 9000
    eth2:
    dhcp4: no
    dhcp6: no
    link-local: []
    mtu: 9000

It might be also needed to increase the hardware and software queue size of your network interface links for higher throughput.

The command ethtool -g <interface> shows the currently applied and maximum hardware queue size.

$ sudo ethtool -g ens5f1
Ring parameters for ens5f1:
Pre-set maximums:
RX:             4096
RX Mini:        0
RX Jumbo:       0
TX:             4096
Current hardware settings:
RX:             512
RX Mini:        0
RX Jumbo:       0
TX:             512

The currently applied settings can be changed with the following command:

sudo ethtool -G ens5f1 tx 4096 rx 4096

You can even change the software queue size:

sudo ip link set txqueuelen 4096 dev ens5f1

Interface Settings

The interfaces section contains all configurations around interface links and options.

The following configuration allows to overwrite the global default interface link settings.

{ "interfaces": {} }

Attribute	Description	Default
io-mode	IO mode	packet_mmap_raw
io-slots	IO slots (ring size)	4096
qdisc-bypass	Bypass the kernel’s qdisc layer	true
tx-interval	TX polling interval in milliseconds	1.0
rx-interval	RX polling interval in milliseconds	1.0
tx-threads	Number of TX threads per interface link	0 (main thread)
rx-threads	Number of RX threads per interface link	0 (main thread)
capture-include-streams	Include traffic streams in the capture	true
mac-modifier	Third byte of access session MAC address (0-255)	0

The supported IO modes are listed with bngblaster -v but except packet_mmap_raw all other modes are currently considered experimental. In the default mode (packet_mmap_raw) all packets are received in a Packet MMAP ring buffer and sent directly through RAW packet sockets.

The default tx-interval and rx-interval of 1.0 (1ms) allows precise timestamps and high throughput. Those values can be further increased (e.g. 0.1) for higher throughput or decreased (e.g. 5.0) for lower system load.

It might be also needed to increase the io-slots from the default value of 4096 to reach the desired throughput. The actual meaning of IO slots depends on the selected IO mode. For Packet MMAP, it defines the maximum number of packets in the ring buffer.

{
    "interfaces": {
        "tx-interval": 0.1,
        "rx-interval": 0.1,
        "io-slots": 2048,
    }
}

Links

The link configuration is optional and allows to define per interface link configurations. An explicit link configuration with the global default settings is automatically generated if no link is defined for interface links referenced by interface functions.

{ "interfaces": { "links": [] } }

Attribute	Description	Default
interface	Interface name (e.g. eth0, …)
description	Interface description
mac	Overwrite the MAC address	Interface MAC address
io-mode	Overwrite the IO mode
io-slots	Overwrite the IO slots (ring size)
io-slots-tx	Overwrite the TX IO slots (ring size)
io-slots-rx	Overwrite the RX IO slots (ring size)
qdisc-bypass	Overwrite the kernel’s qdisc layer configuration
tx-interval	Overwrite the TX polling interval in milliseconds
rx-interval	Overwrite the RX polling interval in milliseconds
tx-threads	Overwrite the number of TX threads per interface link
rx-threads	Overwrite the number of RX threads per interface link
tx-cpuset	Optionally pin TX threads to CPU cores (cpuset)
rx-cpuset	Optionally pin RX threads to CPU cores (cpuset)
lag-interface	Add interface link to LAG group
lacp-priority	LACP interface priority	32768

{
    "interfaces": {
        "tx-interval": 0.1,
        "rx-interval": 0.1,
        "io-slots": 2048,
    }
}

Link Aggregation (LAG)

The BNG Blaster supports link aggregation (LAG) with and without LACP. The created LAG interface can be used as link for all kinds of interface functions.

{ "interfaces": { "lag": [] } }

Attribute	Description	Default
interface	Interface name (e.g. lag0, …)
lacp	De-/activate LACP	false
lacp-timeout-short	De-/activate LACP short timeout (3x1s)	false (3x30s)
lacp-system-priority	LACP system priority	32768
lacp-system-id	LACP system identifier	02:ff:ff:ff:ff:00
lacp-max-active-links	Limit the maximum number of active links	255
mac	LAG interface MAC address	02:ff:ff:ff:ff:<id>

Note

Multithreaded IO is not supported for LAG member interfaces!

{
    "interfaces": {
        "lag": [
            {
                "interface": "lag1",
                "lacp": true,
                "lacp-timeout-short": true
            }
        ],
        "links": [
            {
                "interface": "eth1",
                "lag-interface": "lag1"
            },
            {
                "interface": "eth1",
                "lag-interface": "lag1"
            }
        ],
        "network": [
            {
                "interface": "lag1",
                "address": "10.100.0.2/24",
                "gateway": "10.100.0.1"
            }
        ]
    }
}

Multithreaded Interfaces

The BNG Blaster handles all traffic sent and received (IO) in the main thread per default. With this default behavior, you can achieve between 100.000 and 200.000 PPS bidirectional traffic in most environments. Depending on the actual setup, this can be even less or much more, which is primarily driven by the single-thread performance of the given CPU.

Those numbers can be increased by splitting the workload over multiple IO worker threads. Every IO thread will handle only one interface and direction. It is also possible to start multiple threads for the same interface and direction.

The number of IO threads can be configured globally for all interfaces or per interface link.

{
    "interfaces": {
        "rx-threads": 2,
        "tx-threads": 1,
        "links": [
            {
                "interface": "eth1",
                "rx-threads": 4,
                "rx-cpuset": [4,5,6,7],
                "tx-threads": 3,
                "tx-cpuset": [1,2,3]
            }
        ]
    }
}

The configuration per interface link allows asymmetric thread pools. Assuming you would send massive unidirectional traffic from eth1 to eth2. In such a scenario, you would set up multiple TX threads and one RX thread on eth1. For eth2 you would do the opposite, meaning to set up multiple RX threads but only one TX thread.

It is also possible to start dedicated threads for TX but remain RX in the main thread or vice versa by setting the number of threads to zero (default).

With multithreading, you should be able to scale up to 1 million PPS bidirectional, depending on the actual configuration and setup. This allows starting 1 million flows with 1 PPS per flow over at least 4 TX threads to verify all prefixes of a BGP full table for example.

The configured traffic streams are automatically balanced over all TX threads of the corresponding interfaces but a single stream can’t be split over multiple threads to prevent re-ordering issues.

Enabling multithreaded IO causes some limitations. First of all, it works only on systems with CPU cache coherence, which should apply to all modern CPU architectures. It is also not possible to bundle (Link Aggregation) multithreaded interfaces. It is also not possible to capture traffic streams send or received on threaded interfaces. All other traffic is still captured even on threaded interfaces.

Note

The BNG Blaster is currently tested for up to 1 million PPS with 1 million flows, which is not a hard limitation but everything above should be considered experimental. It is also possible to scale beyond using DPDK-enabled interfaces.

Interface Functions

The BNG Blaster supports three types of interface functions, network, access, and a10nsp.

Network Interfaces

The network interfaces are used to emulate the core-facing side of the internet with optional routing protocols and traffic.

Those interfaces can communicate with the configured gateway only. Meaning that all traffic sent from the network interface will be sent to the learned MAC address of the configured gateway.

The network interface behaves like a router. It accepts all traffic sent to its own MAC address. This allows sending and receiving traffic for prefixes advertised via routing protocols or configured via static routes on the connected device under test.

The network interfaces are also used to inject downstream multicast test traffic for IPTV tests. It is also possible to send RAW traffic streams between network interfaces without any access interface defined for non-BNG testing.

The BNG Blaster responds to all ICMP echo requests sent to its own MAC address.

{ "interfaces": { "network": [] } }

Attribute	Description	Default
interface	Parent interface link name (e.g. eth0, …)
address	Local network interface IPv4 address
gateway	Gateway network interface IPv4 address
address-ipv6	Local network interface IPv6 address (implicitly /64)
gateway-ipv6	Gateway network interface IPv6 address (implicitly /64)
vlan	Network interface VLAN	0 (untagged)
gateway-mac	Optional set gateway MAC address manually
gateway-resolve-wait	Sessions will not start until gateways are resolved	true
isis-instance-id	Assign interface to ISIS instance
isis-level	ISIS interface level	3
isis-p2p	ISIS P2P interface	true
isis-l1-metric	ISIS level 1 interface metric	10
isis-l2-metric	ISIS level 2 interface metric	10

The BNG Blaster supports multiple network interfaces as shown in the example below.

{
    "interfaces": {
        "tx-interval": 1,
        "rx-interval": 1,
        "io-slots": 4096,
        "network": [
            {
                "interface": "eth2",
                "address": "10.0.0.1/24",
                "gateway": "10.0.0.2",
                "address-ipv6": "fc66:1337:7331::1/64",
                "gateway-ipv6": "fc66:1337:7331::2"
            },
            {
                "interface": "eth3",
                "address": "10.0.1.1/24",
                "gateway": "10.0.1.2",
                "address-ipv6": "fc66:1337:7331:1::1/64",
                "gateway-ipv6": "fc66:1337:7331:1::2"
            }
        ],
    }
}

Using multiple network interfaces requires selecting which network interface to be used. If not explicitly configured, one of the interfaces is selected automatically. Therefore, the configuration option network-interface is supported in different sections.

It is also supported to have multiple VLAN-tagged network interfaces on the same interface link.

VLAN-tagged network interfaces must be referenced by <interface>:<vlan> in the configuration to distinguish between them.

{
    "interfaces": {
        "network": [
            {
                "interface": "eth1",
                "address": "10.100.0.2/24",
                "gateway": "10.100.0.1",
                "vlan": 100
            },
            {
                "interface": "eth1",
                "address": "10.200.0.2/24",
                "gateway": "10.200.0.1",
                "vlan": 200
            }
        ]
    },
    "streams": [
        {
            "name": "S100",
            "type": "ipv4",
            "pps": 10,
            "network-interface": "eth1:100",
            "destination-ipv4-address": "10.200.0.2"
        },
        {
            "name": "S200",
            "type": "ipv4",
            "pps": 20,
            "network-interface": "eth1:200",
            "destination-ipv4-address": "10.100.0.2"
        }
    ]
}

Access Interfaces

The access interfaces are used to emulate PPPoE and IPoE clients.

{ "interfaces": { "access": [] } }

Attribute	Description	Default
interface	Parent interface link name (e.g. eth0, …)
network-interface	Select the corresponding network interface for this session
type	Set access type (pppoe or ipoe)	pppoe
vlan-mode	Set VLAN mode 1:1 or N:1	1:1
qinq	Set outer VLAN ethertype to QinQ (0x88a8)	false
outer-vlan-min	Outer VLAN minimum value	0 (untagged)
outer-vlan-max	Outer VLAN maximum value	0 (untagged)
outer-vlan	Set outer-vlan-min/max equally
inner-vlan-min	Inner VLAN minimum value	0 (untagged)
inner-vlan-max	Inner VLAN maximum value	0 (untagged)
inner-vlan	Set inner-vlan-min/max equally
third-vlan	Add a fixed third VLAN (most inner VLAN)	0 (untagged)
address	Static IPv4 base address (IPoE only)
ppp-mru	Overwrite PPP MRU (PPPoE only)
address-iter	Static IPv4 base address iterator (IPoE only)
gateway	Static IPv4 gateway address (IPoE only)
gateway-iter	Static IPv4 gateway address iterator (IPoE only)
username	Overwrite the username from the authentication section
password	Overwrite the password from the authentication section
authentication-protocol	Overwrite the username from the authentication section
agent-circuit-id	Overwrite the agent-circuit-id from the access-line section
agent-remote-id	Overwrite the agent-remote-id from the access-line section
rate-up	Overwrite the rate-up from the access-line section
rate-down	Overwrite the rate-down from the access-line section
dsl-type	Overwrite the dsl-type from the access-line section
ipcp	De-/activate PPP IPCP
ip6cp	De-/activate PPP IP6CP
ipv4	De-/activate IPv4
ipv6	De-/activate IPv6
dhcp	De-/activate DHCP
dhcpv6	De-/activate DHCPv6
igmp-autostart	Overwrite IGMP autostart
igmp-version	Overwrite IGMP protocol version (1, 2 or 3)
stream-group-id	Stream group identifier
access-line-profile-id	Access-line-profile identifier
cfm-cc	De-/activate EOAM CFM CC (IPoE only)	false
cfm-level	Set EOAM CFM maintenance domain level	0
cfm-ma-id	Set EOAM CFM maintenance association identifier	0
cfm-ma-name	Set EOAM CFM maintenance association short name
i1-start	Iterator for usage in strings {i1}	1
i1-step	Iterator step per session	1
i2-start	Iterator for usage in strings {i2}	1
i2-step	Iterator step per session	1
monkey	Enable monkey testing	false

For all modes, it is possible to configure between zero and three VLAN tags on the access interface. The VLAN identifier 0 disables the corresponding VLAN header.

[ethernet][outer-vlan][inner-vlan][third-vlan][pppoe]...

Untagged

{
    "access": {
        "interface": "eth1",
        "outer-vlan-min": 0,
        "outer-vlan-max": 0,
        "inner-vlan-min": 0,
        "inner-vlan-max": 0
    }
}

Single Tagged

{
    "access": {
        "interface": "eth1",
        "outer-vlan-min": 1,
        "outer-vlan-max": 4049,
        "inner-vlan-min": 0,
        "inner-vlan-max": 0
    }
}

Double Tagged

{
    "access": {
        "interface": "eth1",
        "outer-vlan-min": 1,
        "outer-vlan-max": 4049,
        "inner-vlan-min": 7,
        "inner-vlan-max": 7
    }
}

Triple Tagged

{
    "access": {
        "interface": "eth1",
        "outer-vlan-min": 10,
        "outer-vlan-max": 20,
        "inner-vlan-min": 128,
        "inner-vlan-max": 4000,
        "third-vlan": 7
    }
}

The BNG Blaster supports also multiple access interfaces or VLAN ranges as shown in the example below.

{
    "access": [
        {
            "interface": "eth1",
            "type": "pppoe",
            "username": "pta@rtbrick.com",
            "outer-vlan-min": 1000,
            "outer-vlan-max": 1999,
            "inner-vlan-min": 7,
            "inner-vlan-max": 7
        },
        {
            "interface": "eth1",
            "type": "pppoe",
            "username": "l2tp@rtbrick.com",
            "outer-vlan-min": 2000,
            "outer-vlan-max": 2999,
            "inner-vlan-min": 7,
            "inner-vlan-max": 7
        },
        {
            "interface": "eth3",
            "type": "pppoe",
            "username": "test@rtbrick.com",
            "outer-vlan-min": 128,
            "outer-vlan-max": 4000,
            "inner-vlan-min": 7,
            "inner-vlan-max": 7
        },
        {
            "interface": "eth4",
            "type": "ipoe",
            "outer-vlan-min": 8,
            "outer-vlan-max": 9,
            "address": "200.0.0.1",
            "address-iter": "0.0.0.4",
            "gateway": "200.0.0.2",
            "gateway-iter": "0.0.0.4"
        }
    ]
}

The configuration attributes for username, agent-remote-id and agent-circuit-id support also some variable substitution. The variable {session-global} will be replaced with a number starting from 1 and incremented for every new session. whereas the variable {session} is incremented per-interface section.

In VLAN mode N:1 only one VLAN combination is supported per access interface section. This means that only VLAN min or max is considered as a VLAN identifier.

{
    "access": [
        {
            "interface": "eth1",
            "type": "pppoe",
            "vlan-mode": "N:1",
            "username": "test@rtbrick.com",
            "outer-vlan-min": 7
        },
        {
            "interface": "eth2",
            "type": "pppoe",
            "vlan-mode": "N:1",
            "username": "test@rtbrick.com",
            "outer-vlan-min": 2000,
            "inner-vlan-min": 7,
        },
    ]
}

The BNG Blaster supports access and network interface functions on the same interface link if both are tagged with disjoint VLAN ranges.

A10NSP Interfaces

The A10NSP interface function is required for L2BSA tests and emulates a layer two provider interface. The term A10 refers to the end-to-end ADSL network reference model from TR-025.

{ "interfaces": { "a10nsp": [] } }

Attribute	Description	Default
interface	Parent interface link name (e.g. eth0, …)
qinq	Set outer VLAN ethertype to QinQ (0x88a8)	false
mac	Optional set gateway interface address manually

The BNG Blaster supports multiple A10NSP interfaces as shown in the example below.

{
    "interfaces": {
        "tx-interval": 1,
        "rx-interval": 1,
        "a10nsp": [
            {
                "interface": "eth4",
                "qinq": true,
                "mac": "02:00:00:ff:ff:01"
            },
            {
                "interface": "eth5",
                "qinq": false,
                "mac": "02:00:00:ff:ff:02"
            }
        ],
    }
}

Note

The A10NSP interface function can’t reside on the same link with with network or access interface functions!

I/O Modes

The BNG Blaster supports many configurable I/O modes listed with bngblaster -v. In the default mode packet_mmap_raw, all packets are received in a Packet MMAP ring buffer and sent through RAW packet sockets.

$ bngblaster -v
Version: 0.8.1
Compiler: GNU (7.5.0)
IO Modes: packet_mmap_raw (default), packet_mmap, raw

Packet MMAP

Packet MMAP is a so-called PACKET_RX_RING/PACKET_TX_RING abstraction where a user-space program gets a fast lane into reading and writing to kernel interfaces using a shared ring buffer. The shared ring buffer is a memory-mapped window shared between the kernel and the user space. This low overhead abstraction allows us to transmit and receive traffic without doing expensive system calls. Sending and transmitting traffic via Packet MMAP is as easy as copying a packet into a buffer and setting a flag.

RAW

RAW Packet Sockets. are used to receive or send raw packets at the device driver (OSI Layer 2) level.

DPDK

Using the experimental DPDK support requires building the BNG Blaster from sources with DPDK enabled as explained in the corresponding installation section.

Note

The official BNG Blaster Debian release packages do not support DPDK!

{
    "interfaces": {
        "tx-interval": 0.1,
        "links": [
            {
                "interface": "0000:23:00.0",
                "io-mode": "dpdk",
                "rx-threads": 4,
                "rx-cpuset": [4,5,6,7],
                "tx-threads": 3,
                "tx-cpuset": [1,2,3]
            },
            {
                "interface": "0000:23:00.2",
                "io-mode": "dpdk",
                "rx-threads": 4,
                "rx-cpuset": [12,13,14,15],
                "tx-threads": 3,
                "tx-cpuset": [9,10,11]
            }
        ],
        "a10nsp": [
            {
                "__comment__": "PPPoE Server",
                "interface": "0000:23:00.0"
            }
        ],
        "access": [
            {
                "__comment__": "PPPoE Client",
                "interface": "0000:23:00.2",
                "type": "pppoe",
                "outer-vlan-min": 1,
                "outer-vlan-max": 4000,
                "inner-vlan-min": 1,
                "inner-vlan-max": 4000,
                "stream-group-id": 1
            }
        ]
    },
    "pppoe": {
        "reconnect": true
    },
    "dhcpv6": {
        "enable": false
    },
    "streams": [
        {
            "stream-group-id": 1,
            "name": "S1",
            "type": "ipv4",
            "direction": "both",
            "pps": 1000,
            "a10nsp-interface": "0000:23:00.0"
        }
    ]
}