Troubleshooting ACI Fabric Discovery Issues

Introduction:

One of the first steps in building your ACI Fabric is to go through Fabric Discovery. While Fabric Discovery is usually a straightforward process, there are various issues that may prevent you from discovering an ACI switch. This article will cover possible reasons for not seeing the first leaf switch entry under Fabric > Inventory > Fabric Membership

Prerequisites:

Go through the initial setup script of APIC1
APIC1, Spines, and Leaves are all running the same ACI version
- The version can vary a little; however, its best to have the same exact version to be 100% sure.
Switch version meets minimum supported version for the hardware (check Hardware to Version support matrix here)
Ensure that transceivers used are supported for the switch hardware and current version (check Transceiver Matrix here)
Configured CIMC IP address for the APIC Controllers (Guide)
APIC1 connected to at least 1 leaf switch

Solution:

Part I: Checking CIMC Configurations

Some CIMC settings are crucial for fabric discovery to function properly. While these settings are usually set to the proper values from the factory, sometimes this is not the case. Follow these steps to confirm proper CIMC settings.

1. Login to the CIMC GUI by typing its ip address on an internet browser

2. Navigate to Admin > Network > Network Settings and confirm that the "NIC Mode" is set to "Dedicated"

3. Navigate to Server > Inventory > Cisco VIC Adapters. Under Adapter Card Properties, confirm that the LLDP is listed as disabled. If this is not the case, please follow this guide to disable the VIC LLDP: https://supportforums.cisco.com/legacyfs/online/attachments/document/files/apic-vic-lldp-fn.pdf

5M69N-2CL01-58193-0H8UK-29E4J

Part II: Checking LLDP Entry Information

Fabric Discovery relies heavily on the exchange and comparison of LLDP information. Go through the below steps to make sure that LLDP is working correctly and each device has the correct information.

1. An APIC may be connected up to two different leaves; however, they run in an active/standby mode. The APIC will only attempt to discover the leaf that is connected to its active interface. The troubleshooting effort should therefore be focused on the leaf connected to the active interface. The below command can be run in order to find out which leaf is active.

apic1# cat /proc/net/bonding/bond0 | grep Active

Currently Active Slave: eth2-2

From the above output, we are able to tell that the active interface is the eth2-2. Therefore, we should focus on the leaf that is connected to eth2-2.

2. Now, we are ready to start checking the lldp information. On the active leaf, copy/paste the following command:

moquery -c lldpIf | grep "wiringIssues : \S\|dn" | grep -B 1 wiring

If you do not see any output at all OR if you do not see any output for the leaf interface connected to APIC1, then you are good to move on to Part III.

Below are the possible wiringIssues output and a brief description on what it signifies:

wiringIssues	Description
fabric-domain-mismatch	Adjacent node belongs to a different fabric
ctrlr-uuid-mismatch	APIC UUID mismatch (duplicate APIC ID)
wiring-mismatch	Invalid connection - Leaf to Leaf, Spine to non-leaf, Leaf fabric port to non-spine etc.
unknown-neighbor	Adjacent node is not part of fabric
adjacency-not-detected	No LLDP adjacency on fabric port (normal if not connected)
infra-vlan-mismatch	Mismatch of infra-vlan value

Most of the above possible output for wiringIssues are self explanatory. The most commonly hit output from above is "infra-vlan-mismatch." See Resolving Infra-Vlan Mismatch section to correct this issue.

Resolving Infra-VLAN Mismatch

=========================

Infra-VLAN mismatch indicates that the two devices are advertising a different infra-vlan between each other. This is often caused by a non properly wiped switch. In order to remedy an infra-vlan-mismatch issue, perform the following:

Physically disconnect all undiscovered leaf or spine switches from the APIC and any other leaf or spine switches
Login to the CLI of the undiscovered leaf switch and clean the switch by running the command "setup-clean-config.sh"
After the command completes, run vsh -c "reload"
Repeat steps #2 and #3 for other undiscovered leaf or spine switch
After all the undiscovered switches have been cleaned and reloaded, physically connect back the leaf/spine switches to the fabric

Part III: Check for issues with APIC DHCP

In order for a entry for the leaf switch to show up under the Fabric Membership folder, the APIC needs to properly process the DHCP Discover message that the leaf switch sends. There is a commonly hit bug (CSCvf12024) for the APIC DHCP that will prevent the above behavior. In order to check if you are running into this issue, run this command on APIC1 CLI:

grep "No subnet declaration for bond0" /var/log/dme/log/dhcpd.bin.log

5402||18-01-22 23:06:12.890+00:00||dhcp||ERROR||||ISC dhcpd: No subnet declaration for bond0. (no IPv4 addresses).||../svc/dhcpd/src/gen/ifc/beh/imp/./DhcpdSvc.cc||53

If you see an output similar to above, you are most likely hitting this bug. Apply the workaround by running this command on APIC1 CLI:

acidiag restart dhcpd

Documentation Credit: [GUIDE_BY_TAC] Fabric Discovery Troubleshooting: Discovering the First Leaf Switch in ACI