So, in this this stage I wanted to test the Distributed Logical Router (DLR) setup. The DLR is one of two types of routers available in NSX Platform and is mainly used in East-West traffic routing to avoid the traffic hair-pinning which can occur in normal centralized routing. For more info, check Roie ben Haim‘s marvelous article about the DLR here.
The DLR consists mainly of two components:
- The Control VM (CVM) which is an Edge Services Gateway (ESG) built specifically to handle routing. Like any normal physical router, this is the SUP module that does all routing stuff (calculating routes and neighboring) and owns the routing tables. tThe CVM isn’t a part of the Data path.
- The Kernel modules in the ESXi Hosts: This module does all data forwarding and contains the forwarding tables.
One of the things I wanted to test in my lab, is the CLI interface of the DLR CVM. You can enable SSH by selecting Enable SSH while deploying DLR using GUI. The problem I faced was how to access it using SSH client from my machine, and to be more clear and concise, I didn’t know which IP I needed to enter in my SSH client to access the DLR CVM.
First thing I tried is the Uplink Interface IP of the DLR (A single DLR can have up to 8 Uplinks) . This is the interface which allow DLR to communicate with the northbound world. I failed to have SSH access on this interface although I disabled the built-in Firewall of the DLR.
I looked through the VMware Documentation, KB articles and experts blogs, until I found this KB article written by my expert friend Dmitri Kalintsev. It states the following:
The DLR Control VM can be reached from the network using one of these options:
- Through the Management Interface, but only from an IP address on the same subnet.
- Through the DLR’s Protocol address, set as part of OSPF or BGP configuration, from any IP address that DLR knows how to get back to.
Option 1 wasn’t useful for me as the Management Interfaces, or HA Interfaces as per NSX 6.2, will not establish any communication with any different subnet. So, I had to use option 2.
For Option 2, let us first differentiate between Protocol Address and Forwarding Address. The Protocol Address is the address used by the DLR CVM for control-plane communication with other physical or virtual routers. The DLR CVM uses the Protocol Address to establish routing adjacency and exchange routing information. It is set on a real vNic that can transmit and receive communication.
The Forwarding Address is the IP address of the Uplink Interface used for communication with the northbound world and it’s owned by the DLR LIF on ESXi Hosts which are connected to the DLR instance. Note also that all DLR LIFs has same vMAC for communication including the Uplink DLR LIF.
Side Note:
When you first create your Uplink interface, you will see that a vNIC is connected on the DLR CVM and it has an Uplink Interface IP (which is the Forwarding Address). You can show this using show interface on DLR CVM console.
After setting the Protocol and Forwarding IPs you can enter the same command. You’ll notice that same vNIC will change its own IP to the Protocol Address.
To show the vMAC Address of the DLR LIFs you have to SSH into the controller and then get the DLR instance ID using show control-cluster logical-routers instance all. After that you can obtain the LIFs’ IDs using show control-cluster logical-routers interface-summary DLR_Instance_ID. Finally you can get the MAC Addresses using show control-cluster logical-routers interface DLR_Instance_ID LIF_ID.
Knowing that, this is what I did to enable SSH Access on my DLR CVM:
- I set up my Uplink interface to be used with required subnet (195.168.1.x).
- I set up my default gateway configuration to allow traffic from and to the DLR Uplink Interface.
- Under OSPF settings, I set the required Protocol and Forwarding addresses. Don’t forget to enable OSPF and Publish Settings.
- On Firewall tab, I allowed SSH communication from any IP to the Uplink IP I used. Don’t forget to Publish Changes after you finish adding your rules.
- I connected a VM to same portgroup or Logical Switch to which Uplink Interface is connected.
When I tested SSH access to my DLR CVM after configuring some static routes, it went successfully!
What I tested in my lab:
- Deploying the DLR CVM.
- Configuring the Interfaces and the required routing configurations.
- Testing the routing between different Logical Switches.
- Testing SSH Access to the DLR CVM.
- Testing how Firewall settings may affect my access and routing.
- Testing come Controller, Central NSX Manager and ESXi hosts CLI commands related to DLR.
The following diagram is showing how my lab looks like at this stage (click here to view in full size or download)
The following diagram explains how my networks (Logical Switches, Portgroups, etc.) are connected (click here to view in full size or download)
References:
** Distributed Logical Router (DLR) by Giuliano Bertello.
** NSX CLI Reference.
Leave a Reply