Understanding Underlay and Overlay Networks: Complete Guide

Introduction to Network Layers

In modern networking, the concepts of underlay and overlay networks have become fundamental to understanding how data centers, cloud platforms, and enterprise networks operate. Whether you're working with Cisco ACI, VMware NSX, SD-WAN solutions, or Kubernetes networking, grasping these concepts is essential for any network engineer in 2025.

The traditional approach to networking—where physical infrastructure directly determines network topology—has evolved dramatically. Today's networks operate on two distinct planes: the physical underlay that provides basic connectivity and the virtual overlay that creates flexible, programmable network topologies on top of it. This separation has revolutionized how we design, deploy, and manage networks at scale.

Think of the underlay as the highway system—the physical roads and infrastructure. The overlay is like the GPS routing layer that determines how traffic flows, completely independent of the physical road structure.

Understanding Underlay Networks

The underlay network is the physical foundation of your network infrastructure. It consists of the actual hardware—routers, switches, cables, and fiber connections—that moves packets from one point to another. The underlay provides the basic IP connectivity that everything else depends on.

Physical Infrastructure Components

An underlay network typically includes:

• Spine-Leaf Architecture: Modern data centers use spine-leaf topologies where leaf switches connect to servers and spine switches provide interconnectivity between leaves. This creates predictable latency and high bandwidth.
• Core Routers: High-performance routers that handle large volumes of traffic between different network segments or sites.
• Physical Links: Ethernet cables, fiber optic connections, or wireless links that physically connect network devices.
• Layer 3 Routing: IP routing protocols that determine how packets traverse the physical network.

The underlay's primary responsibility is to provide reachability between all endpoints. It doesn't care about virtual networks, tenant separation, or application-level policies—it simply moves IP packets efficiently from source to destination.

┌─────────────────────────────────────────────┐
│           SPINE LAYER (L3)                  │
│  ┌─────────┐  ┌─────────┐  ┌─────────┐    │
│  │ Spine 1 │  │ Spine 2 │  │ Spine 3 │    │
│  └────┬────┘  └────┬────┘  └────┬────┘    │
│       │            │            │          │
│  ┌────┴────────────┴────────────┴────┐    │
│  │    Equal-Cost Multi-Path (ECMP)   │    │
│  └────┬────────────┬────────────┬────┘    │
│       │            │            │          │
│  ┌────┴────┐  ┌────┴────┐  ┌────┴────┐   │
│  │ Leaf 1  │  │ Leaf 2  │  │ Leaf 3  │   │
│  └────┬────┘  └────┬────┘  └────┬────┘   │
│       │            │            │          │
│    Servers      Servers      Servers       │
└─────────────────────────────────────────────┘

Routing Protocols in Underlay

The underlay network relies on traditional routing protocols to establish and maintain connectivity:

• BGP (Border Gateway Protocol): Commonly used in data center fabrics for its scalability and policy control. BGP enables large-scale network designs with thousands of endpoints.
• OSPF (Open Shortest Path First): A link-state protocol often used in enterprise networks for its fast convergence and hierarchical design capabilities.
• IS-IS (Intermediate System to Intermediate System): Another link-state protocol favored by service providers for its scalability and simplicity.
• Static Routing: Sometimes used in simple or highly controlled environments, though less common in modern designs.

These protocols ensure that every device in the underlay can reach every other device via IP. The key characteristic of underlay routing is that it operates at Layer 3 and focuses purely on IP reachability, not on application-level concerns or multi-tenancy.

Understanding Overlay Networks

An overlay network is a virtual network topology built on top of the underlay. It uses tunneling technologies to encapsulate traffic, allowing you to create completely different network topologies that are independent of the physical infrastructure below. This is where the magic of network virtualization happens.

Overlay networks solve several critical challenges:

• Multi-tenancy: Create isolated networks for different applications or customers on shared infrastructure
• Scalability: Break free from VLAN limitations (4,096 VLANs) by supporting millions of virtual networks
• Mobility: Move workloads across data centers without changing IP addresses
• Flexibility: Define network topologies in software without physical rewiring
• Security: Implement micro-segmentation and granular security policies

Tunneling Technologies

Several tunneling protocols power overlay networks:

• VXLAN (Virtual Extensible LAN): The most popular overlay protocol, VXLAN extends Layer 2 segments over Layer 3 networks. It uses UDP port 4789 and provides 24-bit segment IDs, supporting over 16 million logical networks.
• GENEVE (Generic Network Virtualization Encapsulation): A newer, more flexible protocol designed to address VXLAN's limitations. It's used in modern cloud platforms like OpenStack and some SD-WAN solutions.
• GRE (Generic Routing Encapsulation): A simple tunneling protocol that encapsulates a wide variety of network layer protocols. Often used in site-to-site VPNs and older overlay implementations.
• NVGRE (Network Virtualization using GRE): Microsoft's overlay protocol, primarily used in Hyper-V environments.
• STT (Stateless Transport Tunneling): Designed for high-performance environments, uses TCP-like headers but is stateless.

VXLAN Deep Dive

VXLAN has become the de facto standard for overlay networking. Let's understand how it works in detail.

VXLAN Encapsulation Process:

1. A virtual machine sends an Ethernet frame to its virtual switch
2. The VXLAN Tunnel Endpoint (VTEP) receives the frame
3. The VTEP adds a VXLAN header with a 24-bit VNI (VXLAN Network Identifier)
4. The VXLAN packet is encapsulated in UDP with destination port 4789
5. An outer IP header is added with the source and destination VTEP IP addresses
6. An outer Ethernet header is added for the next-hop router
7. The packet is sent through the underlay network
8. The destination VTEP decapsulates the packet and delivers the original frame

┌─────────────────────────────────────────────┐
│    Original Ethernet Frame (VM to VM)       │
├─────────────────────────────────────────────┤
│  Original MAC Header | Original IP | Data  │
└─────────────────────────────────────────────┘
                    ↓ VXLAN Encapsulation
┌─────────────────────────────────────────────┐
│         VXLAN Encapsulated Packet           │
├─────────────────────────────────────────────┤
│ Outer Ethernet Header (Physical Network)    │
│ ┌─────────┬──────────┬──────────┐          │
│ │Dest MAC │ Src MAC  │ EtherType│          │
│ └─────────┴──────────┴──────────┘          │
├─────────────────────────────────────────────┤
│ Outer IP Header (Underlay Network)          │
│ ┌────────────┬───────────────┐             │
│ │ Source IP  │ Destination IP│ (VTEPs)    │
│ │ (VTEP A)   │   (VTEP B)    │             │
│ └────────────┴───────────────┘             │
├─────────────────────────────────────────────┤
│ UDP Header                                  │
│ ┌─────────┬─────────┬────────┬──────────┐ │
│ │Src Port │Dst Port │ Length │ Checksum │ │
│ │ Random  │  4789   │        │          │ │
│ └─────────┴─────────┴────────┴──────────┘ │
├─────────────────────────────────────────────┤
│ VXLAN Header (8 bytes)                      │
│ ┌──────┬─────────┬───────────────────┐    │
│ │Flags │Reserved │  VNI (24-bit)     │    │
│ └──────┴─────────┴───────────────────┘    │
├─────────────────────────────────────────────┤
│ Original Ethernet Frame                     │
│ ┌──────────────────────────────────────┐   │
│ │ Inner MAC | Inner IP | Inner Data   │   │
│ └──────────────────────────────────────┘   │
└─────────────────────────────────────────────┘

The VNI (VXLAN Network Identifier) is crucial—it's like a VLAN tag but with 24 bits instead of 12, allowing 16,777,216 unique segments versus 4,096 VLANs. This massive scale enables true multi-tenancy in cloud environments.

How Underlay and Overlay Work Together

The relationship between underlay and overlay is hierarchical and interdependent. The overlay completely relies on the underlay for packet delivery, but the underlay has no knowledge of the overlay's virtual topologies.

Key Dependencies:

• IP Reachability: The underlay must provide IP connectivity between all VTEPs (or tunnel endpoints)
• MTU Considerations: Because overlay encapsulation adds headers (50 bytes for VXLAN), the underlay must support larger MTU sizes—typically 1600 bytes minimum, ideally 9000 bytes (jumbo frames)
• Bandwidth: The underlay must have sufficient bandwidth to carry both the original traffic and the encapsulation overhead
• QoS Mapping: Quality of Service markings from overlay networks should be preserved or mapped to underlay QoS policies
• Multicast Support: Some overlay implementations use multicast in the underlay for broadcast, unknown unicast, and multicast (BUM) traffic

Traffic Flow Analysis

Let's walk through a complete traffic flow from VM to VM across data centers:

1. VM-A in Data Center 1 wants to communicate with VM-B in Data Center 2
2. VM-A sends an Ethernet frame with VM-B's MAC address as the destination
3. The virtual switch (OVS, vSwitch, or similar) on the hypervisor receives the frame
4. The virtual switch identifies that VM-B is in the same VXLAN segment (same VNI) but on a remote host
5. The local VTEP encapsulates the frame in VXLAN, adding the appropriate VNI
6. The VTEP adds a UDP header (port 4789) and an outer IP header with the remote VTEP's IP address
7. The packet enters the underlay network
8. Underlay routers use BGP/OSPF to route the packet to Data Center 2 based on the outer IP header
9. The packet arrives at the remote VTEP in Data Center 2
10. The remote VTEP decapsulates the VXLAN packet, checks the VNI
11. The original Ethernet frame is delivered to VM-B
12. From VM-A and VM-B's perspective, they're on the same Layer 2 network, even though they're in different data centers!

┌───────────────────────────────────────────────────┐
│              OVERLAY LAYER (Virtual)              │
│  ┌────────────┐         ┌────────────┐           │
│  │  VNI 5000  │         │  VNI 6000  │           │
│  │ (Tenant A) │         │ (Tenant B) │           │
│  │  ┌──┐ ┌──┐ │         │  ┌──┐ ┌──┐ │           │
│  │  │VM│ │VM│ │         │  │VM│ │VM│ │           │
│  │  └──┘ └──┘ │         │  └──┘ └──┘ │           │
│  └──────┬─────┘         └──────┬─────┘           │
│         │ VXLAN Tunnels        │                  │
│         └──────┬───────────────┘                  │
├────────────────┼──────────────────────────────────┤
│                │                                   │
│         VTEP Layer (Encapsulation)                │
│    ┌───────────┴───────────┐                     │
│    │  VTEP 1      VTEP 2   │                     │
│    └───────────┬───────────┘                     │
├────────────────┼──────────────────────────────────┤
│                │                                   │
│          UNDERLAY LAYER (Physical)                │
│    ┌───────────┴───────────┐                     │
│    │   IP Network (BGP)     │                     │
│    │  Spine-Leaf Fabric     │                     │
│    │  ┌──────┐  ┌──────┐   │                     │
│    │  │Spine │  │Spine │   │                     │
│    │  └──┬───┘  └───┬──┘   │                     │
│    │     │          │       │                     │
│    │  ┌──┴──┐   ┌──┴──┐    │                     │
│    │  │Leaf │   │Leaf │    │                     │
│    │  └─────┘   └─────┘    │                     │
│    └────────────────────────┘                     │
└───────────────────────────────────────────────────┘

Real-World Implementations

Understanding theory is important, but let's look at how major vendors and platforms implement underlay and overlay networks:

Cisco ACI (Application Centric Infrastructure)

Cisco ACI uses a VXLAN overlay on top of a spine-leaf underlay. The underlay runs IS-IS routing protocol, while the overlay uses VXLAN for tenant separation. Cisco's APIC (Application Policy Infrastructure Controller) centrally manages policies and configurations.

• Underlay: IS-IS for optimal path calculation in the spine-leaf fabric
• Overlay: VXLAN with 15-bit segment IDs (slightly different from standard VXLAN)
• Control Plane: MP-BGP EVPN for distributing endpoint information
• Use Cases: Large enterprise data centers, multi-tenant environments

VMware NSX

NSX provides network virtualization and security for VMware vSphere environments. It creates logical networks completely independent of physical topology.

• Underlay: Any IP network (can be existing infrastructure)
• Overlay: GENEVE (NSX-T) or VXLAN (NSX-V)
• Control Plane: NSX Manager and Controllers for distributed state
• Features: Distributed firewalling, micro-segmentation, load balancing
• Use Cases: VMware-based data centers, software-defined data centers

SD-WAN Solutions

SD-WAN platforms use overlay networks to create intelligent WAN connectivity over multiple underlay networks (MPLS, Internet, LTE).

• Underlay: Multiple transport options (Internet, MPLS, 4G/5G)
• Overlay: IPsec or proprietary tunneling protocols
• Intelligence: Application-aware routing, automatic failover
• Vendors: Cisco Viptela, VMware VeloCloud, Fortinet SD-WAN

Kubernetes Networking

Container orchestration platforms like Kubernetes use overlay networks for pod-to-pod communication.

• Underlay: Physical or virtual network between Kubernetes nodes
• Overlay: Various CNI plugins (Calico, Flannel, Weave, Cilium)
• Encapsulation: VXLAN, IPIP, or WireGuard depending on the CNI

Cloud Provider Virtual Networks

AWS VPC, Azure VNet, and Google Cloud VPC all use overlay networks internally:

• AWS VPC: Mapping service translates virtual IPs to physical hosts, proprietary encapsulation
• Azure VNet: Uses NVGRE and VXLAN for virtual network isolation
• Google Cloud: Andromeda SDN stack with custom encapsulation

Design Best Practices

Designing robust underlay and overlay networks requires careful planning:

Underlay Design Principles

• Keep it Simple: The underlay should be as simple and stable as possible. Avoid complex configurations that could impact overlay traffic.
• Use Consistent MTU: Configure jumbo frames (9000 bytes) throughout the underlay to accommodate overlay encapsulation overhead.
• Enable ECMP: Use Equal-Cost Multi-Path routing to fully utilize all available links in spine-leaf topologies.
• Plan IP Addressing: Use a clean, hierarchical IP addressing scheme for underlay networks. /31 point-to-point links are common.
• Minimize Hop Count: Design topologies that minimize the number of hops between endpoints.
• Avoid Spanning Tree: In modern data centers, rely on Layer 3 routing rather than Layer 2 spanning tree protocols.

Overlay Design Principles

• Plan VNI Assignment: Create a logical VNI numbering scheme that maps to tenants, applications, or environments.
• Use Automation: Leverage controllers and orchestration tools to manage overlay configurations at scale.
• Implement Micro-segmentation: Take advantage of overlay's flexibility to create granular security zones.
• Monitor Encapsulation Overhead: Account for the 15-20% bandwidth overhead from encapsulation headers.
• Test Failover Scenarios: Verify that overlay networks properly failover when underlay paths change.

Troubleshooting Tips

Common issues and how to resolve them:

Essential Troubleshooting Commands:

• show ip bgp summary - Check underlay BGP peering status
• show nve peers - Verify VXLAN tunnel endpoint status (Cisco)
• show interface nve1 - Check VXLAN interface statistics
• show mac address-table - Verify MAC learning in overlay
• show vxlan tunnel - Display active VXLAN tunnels

Career Opportunities

Mastering underlay and overlay networking opens doors to high-paying, in-demand roles in the networking industry. As enterprises migrate to cloud-native architectures and software-defined infrastructure, professionals with these skills are increasingly valuable.

Key Skills to Develop

• Routing Protocols: Deep understanding of BGP, OSPF, and IS-IS
• VXLAN and Tunneling: Hands-on experience with VXLAN configuration and troubleshooting
• Network Automation: Python scripting, Ansible, Terraform for network provisioning
• SDN Controllers: Experience with Cisco APIC, VMware NSX, or OpenDaylight
• Cloud Networking: Understanding of AWS VPC, Azure VNet, Google Cloud networking
• Container Networking: Kubernetes CNI plugins and service mesh concepts

Relevant Certifications

• Cisco CCNP Enterprise: Covers SD-Access, VXLAN EVPN, and automation
• Cisco CCIE Enterprise Infrastructure: Advanced routing, switching, and overlay technologies
• VMware NSX-T Data Center: NSX overlay networking and security
• AWS Certified Advanced Networking: VPC design, Transit Gateway, Direct Connect
• Juniper JNCIP-ENT: Advanced enterprise networking including VXLAN

Salary Expectations (India, 2025)

• Network Engineer (3-5 years): ₹8-15 LPA
• Senior Network Engineer (5-8 years): ₹15-25 LPA
• Network Architect (8+ years): ₹25-45 LPA
• SDN/Cloud Network Engineer: ₹18-35 LPA

Industry Demand

The shift to cloud and virtualization has created massive demand for engineers who understand overlay networking:

• Data Center Transformation: Companies migrating to spine-leaf architectures with VXLAN overlays
• Multi-Cloud Connectivity: Organizations need experts to connect AWS, Azure, and on-premises networks
• SD-WAN Deployments: Enterprises replacing MPLS with SD-WAN solutions
• Container Platforms: Kubernetes adoption driving demand for container networking expertise
• 5G and Edge Computing: New architectures requiring innovative overlay solutions

According to industry reports, network engineers with SDN and overlay networking skills earn 30-40% more than those with only traditional networking knowledge.

Learning Path

Here's a structured approach to mastering these technologies:

1. Foundation (3-6 months): CCNA-level networking, TCP/IP, routing fundamentals
2. Intermediate (6-12 months): CCNP Enterprise, VXLAN basics, Python for network automation
3. Advanced (12-18 months): Hands-on with Cisco ACI or VMware NSX, cloud networking certifications
4. Specialization (18+ months): CCIE or equivalent, contribute to open-source projects, build production experience

Conclusion

Understanding the relationship between underlay and overlay networks is fundamental to modern networking. The underlay provides the stable, physical foundation while the overlay delivers flexibility, scalability, and multi-tenancy. Together, they power today's cloud-native applications, containerized workloads, and distributed systems.

As you advance in your networking career, you'll encounter these concepts in virtually every modern infrastructure—from enterprise data centers to public cloud platforms to SD-WAN deployments. Investing time to master both the theoretical concepts and practical implementations will pay enormous dividends in your professional growth and earning potential.

The networking industry is evolving rapidly, but the fundamental principles of underlay and overlay networking will remain relevant for years to come. Start building your expertise today, and you'll be well-positioned for the high-demand, high-paying roles of tomorrow.