★★★Advanced🏢 Data Center / Cloud

Cisco ACIData CenterSDNAPICSpine-LeafContracts

Cisco ACI Fundamentals: Tenants, EPGs, Contracts, and Day-2 Operations

March 13, 2026·23 min read

Overview

Cisco ACI (Application Centric Infrastructure) is Cisco's policy-based SDN solution for data center fabrics. The fundamental departure from traditional networking is this: you never log into a spine or leaf switch and type forwarding policy. Instead, every forwarding decision — which VLANs exist, which endpoints can talk to which, what happens to traffic crossing a zone boundary — is defined as objects in the APIC (Application Policy Infrastructure Controller) and pushed down to the hardware on-demand.

Traditional networks are configured imperatively: you SSH to each switch, create VLANs, configure trunk ports, add ACLs. ACI is declarative: you define the desired state in the APIC object store, and the system figures out which ASICs need which TCAM entries. A leaf switch that has no endpoints belonging to an EPG literally has no VLAN deployed on it — the hardware stays clean until traffic needs to flow.

This has real operational consequences. Day-2 troubleshooting requires understanding the policy model, not just reading interface counters. A packet drop might be a contract miss (no policy permitting the flow), a missing static binding (endpoint not assigned to an EPG on that port), or an endpoint not learned (leaf hasn't seen the MAC/IP yet). The APIC gives you visibility into all of these through atomic counters, fault objects, and the endpoint tracker.

Key ACI differentiators vs. traditional Nexus:

Policy is centrally defined — no per-switch VLAN/ACL config
Whitelist security model — EPG-to-EPG traffic is denied by default without a contract
iVXLAN/VXLAN underlay with MP-BGP EVPN control plane (automatic, no manual config)
IS-IS runs as the underlay routing protocol between spine and leaf (not OSPF)
APICs are not in the data path — fabric forwards at line rate even if APIC is down

ACI Physical Fabric

The physical fabric has three types of nodes: APIC controllers, spine switches, and leaf switches. Understanding what each does — and critically, what each does not do — prevents hours of troubleshooting confusion.

APIC Cluster

Deploy a minimum of 3 APIC nodes for production HA. APICs run a distributed object store using a Raft-based consensus protocol. The cluster elects a leader; all nodes hold full copies of the policy database. Critically: APICs are not in the data path. If all three APICs go offline, the fabric continues forwarding based on the last-pushed policy. No traffic interruption occurs from APIC failure — you just lose the ability to make changes.

APIC nodes connect to leaf switches (not spines) via a dedicated infra VLAN (VLAN 4093 by default). The APIC-to-leaf link is a standard 10G/25G port, usually eth1/1.

Spine Switches (Nexus 9500 series)

Spines run only two protocols: IS-IS for the underlay (TEP/VTEP reachability) and MP-BGP EVPN for the overlay control plane. Spines never connect to servers or external networks directly — they connect only to leaf switches. Every spine connects to every leaf (full mesh). Spine count is typically 2 for small fabrics, up to 4–6 for large deployments.

Leaf Switches (Nexus 9300 series)

Leaves connect to everything: servers, storage, external routers (for L3Out), APICs, and other leaf switches in a vPC pair. Every leaf connects to every spine. Leaves learn endpoint MAC/IP addresses and report them to the APIC via OpFlex (a policy protocol over HTTPS). The leaf is where EPG policy is enforced — ingress and egress TCAM entries from contracts live on the leaf ASIC.

Underlay: IS-IS + iVXLAN

ACI uses IS-IS (not OSPF) to distribute TEP (Tunnel Endpoint) addresses — each leaf and spine gets a unique TEP IP from the infra address pool (typically 10.0.0.0/16). iVXLAN adds an extra shim header to standard VXLAN carrying ACI-specific policy metadata (src-EPG class ID). MP-BGP EVPN distributes endpoint (MAC/IP) information across the fabric so leaves can build forwarding tables without flooding.

ACI Logical Object Model

The logical hierarchy is what separates ACI from every other network OS. You are not configuring interfaces and VLANs — you are building a policy model that describes who can talk to whom and what network services they share. The hardware configuration is derived automatically.

Tenant

A Tenant is an administrative domain — a container for all the other objects. ACI ships with three built-in tenants: common (shared services: DNS, NTP, shared L3Outs), mgmt (APIC management), and infra (fabric underlay). You create one tenant per application, business unit, or customer depending on your operational model. Tenants provide fault isolation — objects in one tenant cannot accidentally leak into another.

VRF (Context)

Inside a Tenant, a VRF defines one L3 routing domain. Think of it as a routing table — all Bridge Domains in the same VRF share the same IP space and can route to each other (subject to contracts). A tenant can have multiple VRFs for complete L3 separation (e.g., Production VRF and Dev VRF that must never communicate).

Bridge Domain (BD)

A BD is ACI's replacement for a VLAN. It is the L2 forwarding domain and always belongs to exactly one VRF. A BD can have multiple subnets — you configure the gateway IP on the BD, and ACI's distributed anycast gateway responds on every leaf simultaneously (same MAC, same IP, no STP). A BD maps to a VXLAN VNI in the underlay. Key BD settings: unicast routing (enable/disable), ARP flooding (flood vs. proxy), L3 subnet with scope (private/public/shared).

EPG (Endpoint Group)

An EPG is a collection of endpoints that share the same forwarding and security policy. It always belongs to exactly one BD. The EPG is what gets a VLAN ID on the wire — you create a static port binding that says "port eth1/5 on leaf-101, VLAN 100 = this EPG." When an endpoint sends traffic on that port/VLAN, it is classified into the EPG and the ASIC applies EPG-based TCAM rules.

EPGs can also be associated with VMM domains (VMware vCenter integration), which automates port group creation in vSphere — the leaf trunk port gets the VLAN automatically when a VM is attached in vCenter.

Endpoint

The actual host or VM. ACI learns endpoints via ARP/RARP/DHCP snooping on the leaf. When learned, the leaf reports the endpoint (MAC + IP + location: leaf node + port + VLAN) to the APIC via OpFlex. The APIC stores this in the endpoint tracker, giving you a full history of where every IP address has been — invaluable during VM live migration troubleshooting.

The key insight is that two EPGs sitting in the same Bridge Domain are on the same L2 segment — but they still cannot communicate at L3 without a contract. This is the whitelist model: topology determines reachability, but policy determines whether traffic is actually forwarded.

Contracts and Filters

Contracts are the heart of ACI's security model. The default behavior between any two EPGs is deny all — there is no implicit permit-same-subnet rule, no VRF-level allow, nothing. If you don't configure a contract, traffic doesn't flow. Period.

Filter

A Filter defines what traffic to match: protocol (TCP/UDP/ICMP) + destination port + optionally source port and DSCP. You create filters as reusable objects at the Tenant level. A single filter entry might say "TCP, dst-port 8080" — that's it. Filters are intentionally simple so they can be reused across many contracts.

Subject

A Subject groups one or more filters and specifies directionality: both (bidirectional — preferred for most use cases), Provider-to-Consumer only, or Consumer-to-Provider only. Subjects also control whether QoS policy is applied and whether reverse port filtering is needed. One contract can have multiple subjects with different filter sets and different directionalities.

Contract

A Contract contains one or more subjects. The contract is the policy object you export (Provider EPG) and import (Consumer EPG). Contracts live at the Tenant level and can be shared across application profiles within the same tenant.

Provider and Consumer

When you bind a contract to EPGs, you designate one EPG as the provider (the service being accessed) and one as the consumer (the client making requests). The convention is: the EPG offering a service provides the contract; the EPG consuming the service imports it. Traffic from Consumer to Provider is permitted by the filter definition. With a bidirectional subject, return traffic is also permitted. If you get the direction wrong — consumer set as provider or vice versa — traffic drops silently with no obvious error message. This is the single most common ACI misconfiguration in production environments.

vzAny

vzAny is a special object that represents all EPGs within a VRF simultaneously. Attaching a contract to vzAny applies it to every EPG in that VRF — useful for shared services like DNS (UDP/53) and NTP (UDP/123) that every EPG needs to reach. Be careful: vzAny contracts can unintentionally open large amounts of traffic if the filter is too broad.

L3Out — External Network Connectivity

L3Out is how ACI connects to the outside world — WAN edge routers, internet, corporate network, or anything external to the ACI fabric. Without an L3Out, traffic cannot leave the fabric.

Border Leaf

L3Out is configured on a specific leaf or pair of leaves designated as border leaves. The border leaf runs a routing protocol (OSPF, BGP, EIGRP, or static) toward the external router. From the fabric's perspective, the external router is just another endpoint — but one that happens to have routes behind it.

External EPG (l3extInstP)

The external EPG defines what subnets are "outside." You list prefixes here (0.0.0.0/0 for default route, or specific subnets). The external EPG participates in contracts just like an internal EPG — if you want web servers to reach the internet, you create a contract between Web-EPG (consumer) and the external EPG (provider). Without the contract, even if routing works, ACI drops the traffic at the ingress leaf TCAM.

L3Out Configuration Object Hierarchy

The L3Out hierarchy is deeper than internal EPGs: L3Out → Logical Node Profile (selects which leaves) → Logical Interface Profile (selects which interfaces) → Routing protocol configuration → External EPG with subnet definitions. Each layer has specific attributes, and missing any one causes the routing adjacency or route import to fail, often silently.

Transit Routing

ACI supports transit routing — traffic entering on one L3Out (e.g., from WAN) and exiting another L3Out (e.g., to internet DMZ). This requires the fabric VRF to be shared between both L3Outs and the route reflector (spine) to propagate the routes. Transit routing has scale limits depending on the ACI version and hardware generation.

Preferred Group

By default, every EPG-to-EPG communication (including external EPG) requires a contract. The preferred group feature allows you to mark certain EPGs as trusted — traffic within the preferred group is permitted without contracts. Useful for migration scenarios where you want to bring ACI in without immediately building all contracts. Use sparingly in production — it removes the whitelist enforcement that ACI is designed to provide.

APIC Configuration Workflow

Building a new application tenant in ACI follows a consistent sequence. Skipping steps or building out of order causes objects to fail validation or deploy incorrectly.

Step 1: Create Tenant

Navigate to Tenants → Add Tenant. Give it a meaningful name matching the application or business unit. ACI creates a default VRF automatically — you can keep or rename it.

Step 2: Create VRF

Tenant → Networking → VRFs → Create VRF. Set policy enforcement to "Enforced" (contracts required) — never use unenforced in production. Set preferred group to disabled unless you have a specific migration reason.

Step 3: Create Bridge Domain

Tenant → Networking → Bridge Domains → Create BD. Associate with your VRF. Add a subnet (the gateway IP that ACI will answer ARP for). Enable unicast routing. ARP mode: if you have unknown unicast flooding needs, set to flood — otherwise ARP proxy (default) is preferred for scale.

Step 4: Create Application Profile and EPGs

Tenant → Application Profiles → Create Application Profile (logical grouping of EPGs for a single application). Within the AP, create EPGs. Associate each EPG to a BD. Add static port bindings: select the leaf node, interface, and VLAN. Add a VMM domain if deploying VMs via vCenter.

Step 5: Create Filter and Contract

Tenant → Contracts → Filters: create a filter entry specifying EtherType (IP), protocol (TCP/UDP), and destination ports. Tenant → Contracts: create a Contract, add a Subject, associate the filter. Set filter direction on the subject (both for bidirectional).

Step 6: Bind Contract to EPGs

On the provider EPG: Add Contract → Provider → select your contract. On the consumer EPG: Add Contract → Consumer → select the same contract. The APIC validates and pushes TCAM entries to all leaves where these EPGs have endpoints.

REST API Equivalents

# Create Tenant via APIC REST API (JSON)
POST https://apic/api/mo/uni.json
{
  "fvTenant": {
    "attributes": {
      "name": "Prod-App",
      "descr": "Production Application Tenant"
    }
  }
}
# Create VRF inside Tenant
POST https://apic/api/mo/uni/tn-Prod-App.json
{
  "fvCtx": {
    "attributes": {
      "name": "Prod-VRF",
      "pcEnfPref": "enforced"
    }
  }
}
# Create Bridge Domain with subnet and VRF association
POST https://apic/api/mo/uni/tn-Prod-App.json
{
  "fvBD": {
    "attributes": {
      "name": "Prod-BD",
      "arpFlood": "no",
      "unicastRoute": "yes"
    },
    "children": [
      {"fvRsCtx": {"attributes": {"tnFvCtxName": "Prod-VRF"}}},
      {"fvSubnet": {"attributes": {"ip": "10.1.1.1/24", "scope": "public"}}}
    ]
  }
}
# Create EPG with static port binding (leaf 101, eth1/5, vlan-100)
POST https://apic/api/mo/uni/tn-Prod-App/ap-App-Profile.json
{
  "fvAEPg": {
    "attributes": {"name": "Web-EPG"},
    "children": [
      {"fvRsBd": {"attributes": {"tnFvBDName": "Prod-BD"}}},
      {"fvRsPathAtt": {
        "attributes": {
          "tDn": "topology/pod-1/paths-101/pathep-[eth1/5]",
          "mode": "regular",
          "encap": "vlan-100"
        }
      }}
    ]
  }
}
# Create Filter entry: TCP dst 8080
POST https://apic/api/mo/uni/tn-Prod-App.json
{
  "vzFilter": {
    "attributes": {"name": "flt-tcp8080"},
    "children": [{
      "vzEntry": {
        "attributes": {
          "name": "tcp8080",
          "etherT": "ip",
          "prot": "tcp",
          "dToPort": "8080",
          "dFromPort": "8080"
        }
      }
    }]
  }
}
# Create Contract with Subject referencing filter
POST https://apic/api/mo/uni/tn-Prod-App.json
{
  "vzBrCP": {
    "attributes": {"name": "web-to-app", "scope": "tenant"},
    "children": [{
      "vzSubj": {
        "attributes": {"name": "permit-app", "revFltPorts": "yes"},
        "children": [
          {"vzRsSubjFiltAtt": {"attributes": {"tnVzFilterName": "flt-tcp8080"}}}
        ]
      }
    }]
  }
}

Day-2 Verification Commands

Once the fabric is running, you live in the NX-OS CLI on leaves and the APIC moquery tool. These are the commands used daily for operational verification.

Endpoint Learning Verification

# Show all locally learned endpoints on a leaf
Leaf-101# show endpoint
Legend: L - local   O - peer-attached   H - vtep   S - static
        V - vpc-attached   a - locally-aged   B - bounce
+-----------------------------------+------------------+-----------+--------+
      VLAN/Domain           MAC Address          Node/Module  Interface
+-----------------------------------+------------------+-----------+--------+
Prod-App:Web-EPG   vlan-100  a0b1.c2d3.e4f5  10.1.1.10   local eth1/5
Prod-App:App-EPG   vlan-101  b1c2.d3e4.f5a6  10.1.2.10   local eth1/7
# Find endpoint by IP address
Leaf-101# show endpoint ip 10.1.1.10
# Find endpoint by MAC address
Leaf-101# show endpoint mac a0b1.c2d3.e4f5
# Show endpoints on a specific interface
Leaf-101# show endpoint interface ethernet 1/5

EVPN Control Plane Verification

# Check BGP EVPN neighbor state (spines are route reflectors)
Leaf-101# show bgp l2vpn evpn summary
BGP summary information for VRF overlay-1, address family L2VPN EVPN
BGP router identifier 10.0.32.96, local AS number 65001
Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State
10.0.32.64      4 65001    2341    2100      890    0    0  5d04h    48
10.0.32.65      4 65001    2338    2097      890    0    0  5d04h    48
# Show all EVPN routes for a VNI (L2 VNI for VLAN 100 = VNI 10100)
Leaf-101# show bgp l2vpn evpn vni-id 10100
# Show Type-2 MAC/IP route for a specific endpoint
Leaf-101# show bgp l2vpn evpn route-type 2 mac a0b1.c2d3.e4f5
# Show Type-5 IP prefix routes (L3 inter-subnet routing)
Leaf-101# show bgp l2vpn evpn route-type 5

VXLAN and VNI Mapping

# Show VXLAN VNI to VLAN mapping on leaf
Leaf-101# show vxlan
VLAN      VN-Segment     Type       Intf
100       10100          L2         nve1
101       10101          L2         nve1
3000      3000100        L3 (VRF)   nve1
# Show NVE interface and remote VTEP peers
Leaf-101# show nve peers
Interface  Peer-IP          State  LearnType  Uptime    Router-MAC
nve1       10.0.32.97       Up     CP         5d04h     001b.54a1.bd08
nve1       10.0.32.98       Up     CP         5d04h     001b.54a2.bd09
# Show NVE interface counters
Leaf-101# show interface nve 1 counters

Zoning Rule and Contract Verification

# Show TCAM zoning rules for a VRF scope (get scope from show vrf)
Leaf-101# show vrf
VRF-Name                           VRF-ID  State  Reason
Prod-App:Prod-VRF                  10      Up     --
Leaf-101# show zoning-rule scope 2621440
Rule ID   SrcEPG  DstEPG  Prio              Action  Label
4099      16386   16387   src_dst_any       permit  web-to-app
4100      16387   16386   src_dst_any       permit  web-to-app (reverse)
4101      0       0       any               deny    implicit-deny
# Show contract via APIC moquery
apic# moquery -c vzBrCP -f 'vzBrCP.name=="web-to-app"'
# Run atomic counters between two EPGs to verify traffic
apic# acidiag run atomicCounters epg Prod-App/App-Profile/Web-EPG Prod-App/App-Profile/App-EPG

APIC moquery Reference

# Find endpoint by IP across entire fabric
apic# moquery -c fvCEp -f 'fvCEp.ip=="10.1.1.10"'
# Show endpoint move history
apic# moquery -c fvIpAttr -f 'fvIpAttr.ip=="10.1.1.10"'
# Show all active faults in a tenant
apic# moquery -c faultInst -f 'faultInst.dn~"tn-Prod-App"' -p severity
# Show all leaf nodes in the fabric
apic# moquery -c topSystem -f 'topSystem.role=="leaf"'
# Show deployed EPG port bindings on a specific node
apic# moquery -c fvIfConn -f 'fvIfConn.dn~"node-101"'
# Show L3Out configuration
apic# moquery -c l3extOut -f 'l3extOut.dn~"tn-Prod-App"'

Troubleshooting

ACI troubleshooting is a structured process of eliminating layers: physical fabric up? Endpoint learned? EPG deployed on correct leaf and port? Contract exists? Contract in the right direction? Work through these systematically — jumping to contract issues before verifying endpoint learning wastes time.

Endpoint Not Learned

Symptom: show endpoint ip 10.1.1.10 returns nothing on any leaf.

Static port binding missing or wrong: Verify the EPG has a static port binding for the correct node, interface, and VLAN. Use moquery -c fvRsPathAtt -f 'fvRsPathAtt.dn~"Web-EPG"' to check.
VLAN not deployed: The VLAN will not appear on the leaf trunk if the EPG has no static binding. Check show vlan id 100 on the leaf.
AEP (Attachable Entity Profile) missing: The physical interface policy group must reference an AEP that links to the EPG's domain. Missing AEP means the VLAN is not programmed in the TCAM even if the static binding exists.
VM not connected: If using VMM integration, check that the VM's port group in vCenter matches the EPG. An unconnected VM generates no ARP and cannot be learned.

Contract Not Enforcing

Symptom: endpoints are learned, show endpoint shows both sides, but ping and application traffic fail.

Provider/consumer reversed: The most common error in production. Verify which EPG is provider and which is consumer. The server offering the service should be provider.
Filter direction wrong: If the Subject filter is "Consumer-to-Provider only" and you need bidirectional, change to "both" and add revFltPorts: yes on the subject.
Wrong filter entry: TCP vs UDP mismatch, wrong destination port, or EtherType set to "unspecified" instead of "IP" for L4 matches.
Contract scope wrong: Inter-tenant contracts require scope "global" — tenant-scope contracts do not cross tenant boundaries.

L3Out Route Not Imported

Symptom: external destinations unreachable despite L3Out configured and BGP neighbor established.

External EPG subnet mismatch: The subnet defined in the external EPG must match or be a superset of the route being received. Using 0.0.0.0/0 with scope "import-security" catches all external routes.
Contract missing on external EPG: A routing adjacency and route import do not automatically permit traffic. The external EPG still requires a contract binding with the internal EPG.
Route-map filtering: ACI auto-generates route-maps based on external EPG subnet definitions. Routes not matching the external EPG are silently dropped at import.

APIC Disconnected from Fabric

Symptom: APIC GUI shows a leaf as "unavailable" or "disconnected" under Fabric → Inventory.

Infra VLAN 4093: The APIC connects to the leaf on VLAN 4093. If this VLAN is pruned or the port is misconfigured, APIC loses connectivity. Check show interface eth1/1 trunk on the leaf.
LLDP blocked: ACI uses LLDP for initial node discovery. If LLDP is disabled or blocked by upstream switches between the APIC and leaf, discovery fails.
Cluster quorum: Check acidiag avread on any APIC to see cluster leader, quorum state, and which nodes are reachable.

Symptom	First Check	Command	Likely Cause
Traffic dropped between EPGs	Contract exists and direction correct?	show zoning-rule scope <vnid>	Missing or reversed contract
Endpoint not in show endpoint	Static port binding configured?	moquery -c fvRsPathAtt	Missing EPG-to-port binding or AEP
External route missing from VRF	BGP neighbor state up?	show bgp l2vpn evpn summary	L3Out routing not established
VLAN not present on leaf trunk	AEP → Domain → EPG chain intact?	show vlan id <id>	AEP or VMM domain not linked to EPG
APIC shows leaf disconnected	Infra VLAN 4093 on uplink?	show interface eth1/1 trunk	Infra VLAN missing or LLDP blocked
Atomic counter shows drops	Filter entry protocol and port correct?	acidiag run atomicCounters	Wrong L4 protocol or port in filter

ACI Object Model Quick Reference

ACI Object	MO Class	APIC DN Example	Traditional Equivalent
Tenant	fvTenant	uni/tn-Prod-App	Administrative boundary / VRF instance
VRF (Context)	fvCtx	uni/tn-Prod-App/ctx-Prod-VRF	VRF / routing table
Bridge Domain	fvBD	uni/tn-Prod-App/BD-Prod-BD	VLAN + SVI (anycast GW)
EPG	fvAEPg	uni/tn-Prod-App/ap-AP/epg-Web-EPG	VLAN access group + ACL zone
Endpoint	fvCEp	uni/epp/fv-[epg-dn]/node-101/stpathatt-[eth1/5]/conndef/ip-[10.1.1.10]	ARP / MAC table entry
Contract	vzBrCP	uni/tn-Prod-App/brc-web-to-app	ACL / Security Policy
Filter	vzFilter	uni/tn-Prod-App/flt-tcp8080	ACL entry (protocol + port match)
L3Out	l3extOut	uni/tn-Prod-App/out-WAN-L3Out	BGP/OSPF neighbor + redistribution
External EPG	l3extInstP	uni/tn-Prod-App/out-WAN-L3Out/instP-ext-epg	Prefix-list / route source definition

ACI rewards engineers who understand the object model. Every GUI action, every REST API call, every moquery maps to these managed objects (MOs). When something breaks, trace the DN (distinguished name) of the failing object and query it directly — the fault object attached to it will tell you exactly what the APIC attempted to push and why it failed. The fabric is transparent if you know where to look.