VXLAN
VXLAN is a network virtualization technology that addresses scalability challenges in large cloud computing environments. It encapsulates Ethernet frames (Layer 2) within UDP datagrams (Layer 4), which are then transmitted via UDP port 4789, as assigned by IANA. VXLAN endpoints, called VTEPs, terminate VXLAN tunnels and can be either virtual or physical switch ports.
VXLAN supports up to 16 million logical networks and enables Layer 2 adjacency across Layer 3 IP networks. It uses multicast or unicast with head-end replication (HER) to flood broadcast, unknown unicast, and multicast (BUM) traffic.
The VXLAN specification was initially developed by VMware, Arista Networks, and Cisco. Other supporters include Huawei, Broadcom, Citrix, Pica8, Big Switch Networks, Cumulus Networks, Dell EMC, Ericsson, Mellanox, FreeBSD, OpenBSD, Red Hat, Joyent, and Juniper Networks.
VXLAN is officially documented by the IETF in RFC 7348.
When configuring VXLAN in a VyOS virtual machine, ensure that MAC spoofing (Hyper-V) or Forged Transmits (ESX) are permitted. Otherwise, the hypervisor may block forwarded frames.
Note
Although the IANA-assigned VXLAN port is 4789, VyOS uses the Linux default UDP port 8472 for VXLAN interfaces. To ensure compatibility with other vendors, set the port to the IANA standard 4789.
Configuration
Common interface configuration
Configure one or more interface IP addresses.
<address> supports IPv4 and IPv6 in CIDR notation. For example, use
192.0.2.1/24 for IPv4 or 2001:db8::1/64 for IPv6.
Example:
set interfaces vxlan vxlan0 address 192.0.2.1/24
set interfaces vxlan vxlan0 address 2001:db8::1/64
Configure a clear, descriptive alias for the interface.
This alias appears in the show interfaces command and SNMP-based
monitoring tools.
Example:
set interfaces vxlan vxlan0 description 'This is an interface running on VyOS.'
Disable the interface.
The interface will be set to the administratively down
(A/D) state.
Example:
set interfaces vxlan vxlan0 disable
Disable Ethernet flow control (IEEE 802.3x pause frames) on the interface.
Ethernet flow control, defined by the IEEE 802.3x standard, temporarily stops data transmission to prevent packet loss during network congestion. For example, when a sender transmits data faster than the receiver can process it.
Disabling Ethernet flow control means the interface will not signal the connected device to pause transmission and will drop packets if overwhelmed.
Example:
set interfaces vxlan vxlan0 disable-flow-control
Disable physical link-state change detection on the interface, such as when a cable is unplugged.
By default, the interface detects physical link-state changes.
Example:
set interfaces vxlan vxlan0 disable-link-detect
Configure a custom MAC address on the interface.
Example:
set interfaces vxlan vxlan0 mac '00:53:01:02:03:04'
Configure the MTU on the interface.
This value defines the largest packet size, in bytes, that the interface transmits without fragmentation.
Example:
set interfaces vxlan vxlan0 mtu 1600
Configure the MSS advertised in outgoing TCP SYN packets on the specified interface.
By clamping the MSS value in TCP SYN packets, you explicitly inform the remote side not to send packets larger than that size. This prevents connection issues that occur when Path MTU Discovery (PMTUD) fails.
The following options are available:
mss: Sets the MSS to a specific value, in bytes. Use this option if you need to enforce a specific MSS, for example, to troubleshoot connectivity issues or accommodate specific network requirements.
clamp-mss-to-pmtu: The router automatically calculates the MSS to be the interface’s MTU minus 40 bytes for IPv4 traffic (20 bytes for the IPv4 header and 20 bytes for the TCP header). This option is recommended to automatically set the proper value.
Configure how long an ARP entry remains valid after learning an IP-to-MAC address mapping on this interface.
The default duration is 30 seconds.
An ARP entry remains valid if it receives positive feedback from higher-level protocols.
Example:
set interfaces vxlan vxlan0 ip arp-cache-timeout 180
Configure ARP filtering on this interface.
Default behavior: The kernel responds to ARP requests on this interface only if the traffic would be routed back to the ARP sender through that specific interface.
If configured: The kernel responds to ARP requests on this interface for any IP address configured on the local host, regardless of which specific interface that IP address is assigned to, and regardless of the routing table. This reflects the Linux concept that IP addresses belong to the host, not individual interfaces.
Example:
set interfaces vxlan vxlan0 ip disable-arp-filter
Configure the interface for host or router behavior.
If configured, the interface switches to host mode, and IPv4 forwarding is disabled on it.
Example:
set interfaces vxlan vxlan0 ip disable-forwarding
Configure whether to forward IP-directed broadcast packets received on this interface.
Default behavior: IP-directed broadcast packets are dropped.
If configured: IP-directed broadcast packets are forwarded to all hosts on the destination subnet, as defined in RFC 1812 and RFC 2644.
Example:
set interfaces vxlan vxlan0 ip enable-directed-broadcast
Configure how to process gratuitous ARPs on this interface.
If configured, an IP-to-MAC address mapping is added to the ARP table based on gratuitous ARP requests or replies.
Note
If the ARP table already contains the IP address from a gratuitous ARP, its entry is updated regardless of whether this setting is configured.
Example:
set interfaces vxlan vxlan0 ip enable-arp-accept
Configure the source IP selection for ARP requests on this interface.
Default behavior: The kernel can use any IP address the host owns as the source IP address in ARP requests on this interface.
If configured: The kernel first attempts to select a source IP address configured on the interface that shares a common subnet with the target IP address. If there is no such subnet, the kernel selects the IP address it would normally use (based on the routing table to reach the target destination).
Example:
set interfaces vxlan vxlan0 ip enable-arp-announce
Configure which ARP requests will be ignored on this interface.
Default behavior: The kernel responds to ARP requests for any local IP addresses, regardless of which interface they are assigned to.
If configured: The kernel responds to ARP requests only if the target IP address is assigned to this specific interface.
Example:
set interfaces vxlan vxlan0 ip enable-arp-ignore
Configure proxy ARP on this interface.
If configured, the router (kernel) intercepts ARP requests for non-local IP addresses and replies with the MAC address of the interface that received the request. Subsequent packets destined to these IP addresses are forwarded to their actual destinations on remote subnets.
Example:
set interfaces vxlan vxlan0 ip enable-proxy-arp
Configure local proxy ARP on the interface.
If configured, the router (kernel) responds to ARP requests on this VLAN interface even if the target IP address resides on the same subnet and interface.
This is used to support network isolation requirements (RFC 3069) for private VLANs (PVLANs). In PVLAN configurations, hosts on isolated ports are NOT allowed to communicate directly with each other at Layer 2, but they can communicate with the upstream router.
By replying to inter-host ARP requests with its own MAC address, the router (kernel) directs inter-host traffic through itself instead of directly between hosts.
Note
This command works independently and does not require enabling the standard proxy ARP on the interface.
Local proxy ARP is also known as:
VLAN aggregation (RFC 3069).
Private VLAN (Cisco, Allied Telesyn).
Source-port filtering or port isolation (Hewlett-Packard).
MAC-Forced Forwarding (Ericsson).
Configure source IP address validation using RPF on this interface, as specified in RFC 3704.
The following options are available:
strict: Each incoming packet’s source IP address is checked against the Forwarding Information Base (FIB). If the interface is not the best route back to that source, validation fails, and the packet is dropped.
loose: Each incoming packet’s source IP address is checked against the FIB. If the source IP address is unreachable through any interface, validation fails.
disable: No source IP address validation is performed. All incoming packets are accepted.
RFC 3704 recommends enabling strict mode to prevent IP spoofing, such as DDoS attacks. For asymmetric or other complex routing scenarios, use loose mode.
Configure the interface to automatically obtain an IPv6 address using SLAAC, as specified in RFC 4862.
IPv6 hosts can configure themselves automatically when connected to an IPv6 network using the Neighbor Discovery Protocol via ICMPv6 router discovery messages. When first connected to a network, a host sends a link-local router solicitation multicast request for its configuration parameters. The router responds with a router advertisement packet containing Internet Layer configuration parameters.
Note
This method automatically disables IPv6 traffic forwarding on the interface.
Example:
set interfaces vxlan vxlan0 ipv6 address autoconf
Configure the interface to assign itself an IPv6 address using the EUI-64 method, as specified in RFC 4291.
Example:
set interfaces vxlan vxlan0 ipv6 address eui64 2001:db8:beef::/64
Disable the automatic assignment of a link-local IPv6 address to this interface.
Example:
set interfaces vxlan vxlan0 ipv6 address no-default-link-local
Configure the interface for host or router behavior.
If configured, the interface switches to host mode, and IPv6 forwarding is disabled on it.
Example:
set interfaces vxlan vxlan0 ipv6 disable-forwarding
Configure the MSS advertised in outgoing TCP SYN packets on the specified interface.
By clamping the MSS value in TCP SYN packets, you explicitly inform the remote side not to send packets larger than that size. This prevents connection issues when Path MTU Discovery (PMTUD) fails.
The following options are available:
mss: Set the MSS to a specific value, in bytes. Use this option to enforce a specific MSS, for example, to troubleshoot connectivity issues or accommodate specific network requirements.
clamp-mss-to-pmtu: The router calculates the MSS to be the interface’s MTU minus 60 bytes for IPv6 traffic (40 bytes for the IPv6 header and 20 bytes for the TCP header). This option is recommended to automatically set the proper value.
Configure IPv6 DAD on the interface.
The following options are available:
0: Disables DAD. No duplicate address detection is performed.
1: Enables DAD (default). Duplicate addresses are detected. The interface’s IPv6 operation continues for valid IPv6 addresses.
2: Enables DAD and, if a MAC-based duplicate link-local address is found, disables IPv6 operation on this interface.
Example:
set interfaces vxlan vxlan0 ipv6 accept-dad 2
Configure the number of DAD messages that the router (kernel) sends during IPv6 address assignment on this interface.
The default value is 1.
Example:
set interfaces vxlan vxlan0 ipv6 dup-addr-detect-transmits 5
Assign the interface to a specific VRF instance.
See also
For information on configuring a VRF, refer to the VRF section.
Example:
set interfaces vxlan vxlan0 vrf red
VXLAN-specific options
Configure a VNI for the VXLAN interface.
Each VXLAN segment is identified by this 24-bit VNI, allowing up to 16 million segments to coexist within the same administrative domain.
Configure the UDP port of the remote VXLAN endpoint.
Note
Although the IANA-assigned VXLAN port is 4789, VyOS uses the Linux default UDP port 8472 for VXLAN interfaces.
Configure the source IP address for the VXLAN underlay.
Warning
This setting is mandatory when deploying VXLAN via L2VPN/EVPN.
Enable the GPE for the VXLAN interface.
To use this feature, you must configure the interface with the external
parameter.
Configure the VXLAN interface to use an external control plane, such as BGP L2VPN/EVPN, for remote endpoint discovery.
If not configured, the internal FDB is used.
Enable ARP and ND suppression on the VXLAN interface.
This reduces ARP and ND message flooding across the VXLAN network. As defined in RFC 7432#section-10, participating VTEPs use known MAC-to-IP bindings to reply to local requests on behalf of remote hosts.
Disable SLLA and IP address learning on the VXLAN interface.
Enable VNI filtering on the VXLAN interface.
When enabled, the interface only receives packets with VNIs configured in its VNI filtering table.
Note
VNI filtering works only if the interface is configured with the
external parameter.
Unicast
Multicast
Configure the source interface for the VXLAN underlay.
All VXLAN traffic is sent and received through the specified interface.
This setting is mandatory when deploying VXLAN over a multicast network.
Multicast VXLAN
Topology: PC4 - Leaf2 - Spine1 - Leaf3 - PC5
PC4 uses the IP address 10.0.0.4/24, and PC5 uses the IP address
10.0.0.5/24. Both devices assume they reside within the same broadcast
domain.
Assume PC4 on Leaf2 pings PC5 on Leaf3. Rather than manually specifying Leaf3 as the remote endpoint, Leaf2 encapsulates the packet into a UDP datagram and sends it to the designated multicast address via Spine1. Spine1 forwards the packet to all leaves in the same multicast group, including Leaf3. Upon receiving the datagram, Leaf3 forwards it to PC5 and learns that PC4 is reachable through Leaf2 by inspecting the source IP in the encapsulated datagram.
PC5 receives the ping and responds with an echo reply. Leaf3, now aware of PC4’s location, forwards the reply directly to Leaf2’s unicast address. Upon receiving the echo reply, Leaf2 learns that PC5 is reachable through Leaf3.
After this discovery, subsequent traffic between PC4 and PC5 will not use the multicast address between the leaves, as both leaves have learned the PCs’ locations. This reduces multicast traffic and network load, improving scalability as more leaves are added.
Single VXLAN device (SVD)
In VyOS, you can configure multiple VLAN-to-VNI mappings for EVPN-VXLAN on a single container interface, known as a single VXLAN device (SVD). This enables significant VNI scaling because a separate VXLAN interface is not required for each VNI.
Map a VLAN ID to a VNI on the specified VXLAN interface.
The VXLAN interface can be added to a bridge.
The following example shows an SVD configuration with multiple VLAN-to-VNI mappings.
set interfaces bridge br0 member interface vxlan0
set interfaces vxlan vxlan0 parameters external
set interfaces vxlan vxlan0 source-interface 'dum0'
set interfaces vxlan vxlan0 vlan-to-vni 10 vni '10010'
set interfaces vxlan vxlan0 vlan-to-vni 11 vni '10011'
set interfaces vxlan vxlan0 vlan-to-vni 30 vni '10030'
set interfaces vxlan vxlan0 vlan-to-vni 31 vni '10031'
Example
The following example demonstrates a multicast VXLAN deployment.
The setup includes three routers: Spine1, a Cisco IOS router, and Leaf2 and Leaf3, which are VyOS routers.
Topology: Leaf2 - Spine1 - Leaf3.
The topology is built using GNS3.
Spine1:
fa0/2 towards Leaf2, IP-address: 10.1.2.1/24
fa0/3 towards Leaf3, IP-address: 10.1.3.1/24
Leaf2:
Eth0 towards Spine1, IP-address: 10.1.2.2/24
Eth1 towards a VLAN-aware switch
Leaf3:
Eth0 towards Spine1, IP-address 10.1.3.3/24
Eth1 towards a VLAN-aware switch
Spine1 configuration:
conf t
ip multicast-routing
!
interface fastethernet0/2
ip address 10.1.2.1 255.255.255.0
ip pim sparse-dense-mode
!
interface fastethernet0/3
ip address 10.1.3.1 255.255.255.0
ip pim sparse-dense-mode
!
router ospf 1
network 10.0.0.0 0.255.255.255 area 0
Multicast routing is required for scalable traffic forwarding between leaves. PIM must be enabled towards the leaves so the spine can learn from which multicast groups each leaf expects traffic.
Leaf2 configuration:
set interfaces ethernet eth0 address '10.1.2.2/24'
set protocols ospf area 0 network '10.0.0.0/8'
! First VXLAN interface
set interfaces bridge br241 address '172.16.241.1/24'
set interfaces bridge br241 member interface 'eth1.241'
set interfaces bridge br241 member interface 'vxlan241'
set interfaces vxlan vxlan241 group '239.0.0.241'
set interfaces vxlan vxlan241 source-interface 'eth0'
set interfaces vxlan vxlan241 vni '241'
! Second VXLAN interface
set interfaces bridge br242 address '172.16.242.1/24'
set interfaces bridge br242 member interface 'eth1.242'
set interfaces bridge br242 member interface 'vxlan242'
set interfaces vxlan vxlan242 group '239.0.0.242'
set interfaces vxlan vxlan242 source-interface 'eth0'
set interfaces vxlan vxlan242 vni '242'
Leaf3 configuration:
set interfaces ethernet eth0 address '10.1.3.3/24'
set protocols ospf area 0 network '10.0.0.0/8'
! First VXLAN interface
set interfaces bridge br241 address '172.16.241.1/24'
set interfaces bridge br241 member interface 'eth1.241'
set interfaces bridge br241 member interface 'vxlan241'
set interfaces vxlan vxlan241 group '239.0.0.241'
set interfaces vxlan vxlan241 source-interface 'eth0'
set interfaces vxlan vxlan241 vni '241'
! Second VXLAN interface
set interfaces bridge br242 address '172.16.242.1/24'
set interfaces bridge br242 member interface 'eth1.242'
set interfaces bridge br242 member interface 'vxlan242'
set interfaces vxlan vxlan242 group '239.0.0.242'
set interfaces vxlan vxlan242 source-interface 'eth0'
set interfaces vxlan vxlan242 vni '242'
The configurations for Leaf2 and Leaf3 are nearly identical. Detailed explanations for each command are provided below.
set interfaces bridge br241 address '172.16.241.1/24'
This command creates a bridge to bind traffic on eth1 VLAN 241 with the
vxlan241 interface. The IP address is optional. If configured, it can serve
as the default gateway for each leaf, allowing devices on the VLAN to reach
other subnets. Subnets must be redistributed by OSPF so the spine can learn how to reach them. To advertise 172.16/12
networks, change the OSPF network from
10.0.0.0/8 to 0.0.0.0/0.
set interfaces bridge br241 member interface 'eth1.241'
set interfaces bridge br241 member interface 'vxlan241'
These commands bind eth1.241 and vxlan241 as member interfaces of the
same bridge.
set interfaces vxlan vxlan241 group '239.0.0.241'
This command configures the multicast group used by all leaves for this VLAN extension. It must be the same on all leaves that have this interface.
set interfaces vxlan vxlan241 source-interface 'eth0'
This command configures the interface that listens for multicast packets. It can also be a loopback interface.
set interfaces vxlan vxlan241 vni '241'
This command configures the unique ID for the VXLAN interface.
set interfaces vxlan vxlan241 port 12345
VyOS uses the Linux default UDP port 8472 for VXLAN interfaces. This command allows you to configure a different UDP port.
Unicast VXLAN
As an alternative to multicast, you can configure the VXLAN tunnel by specifying the remote IPv4 address directly. The following updates the previous multicast example:
# leaf2 and leaf3
delete interfaces vxlan vxlan241 group '239.0.0.241'
delete interfaces vxlan vxlan241 source-interface 'eth0'
# leaf2
set interfaces vxlan vxlan241 remote 10.1.3.3
# leaf3
set interfaces vxlan vxlan241 remote 10.1.2.2
The default UDP port is 8472. To configure a different port, use set
interfaces vxlan <vxlanN> port <port>.