vCloud Director Service Down ... /opt Directory Full

Recently, I had my vCD service down suddenly. After some troubleshooting identified that its due to /opt directory full. Here I am briefing the concept behind it.

Consider an example when you upload media to your cloud organization using vCD. This media will be located physically in the datastore of the OvDC which is providing resources to this organization and will be counted from the storage quota of this organization. To be more detailed, during the upload process, the media will written to the following path in vCD Server:

/$VCLOUD_HOME/data/transfer

From this path it will be rewritten to OvDC datastore.

Example,

[root@EHDF-VCLOUD-01 transfer]# du -sh /opt/vmware/vcloud-director/data/transfer/*

64M     /opt/vmware/vcloud-director/data/transfer/a73ea460-6be7-4c22-a4dd-1520c687da64    !!!... The cookie representing media name

4.0K    /opt/vmware/vcloud-director/data/transfer/cells

16K     /opt/vmware/vcloud-director/data/transfer/lost+found

Typically, $VCLOUD_HOME corresponds to opt/vmware/vcloud-director. This can be verified by browsing the file /etc/profile.d/vcloud.sh

[root@EHDF-VCLOUD-01 transfer]# cat  /etc/profile.d/vcloud.sh

export VCLOUD_HOME=/opt/vmware/vcloud-director

export VCLOUD_MAX_FD=65535

The problem is that vCD will keep the uploaded media in this path for 24hrs before quarantining it. It won't delete it immediately after writing it to datastore. This is mentioned in vCloud Director Installation and Configuration Guide

"Uploads and downloads occupy this storage for a few hours to a day. Transferred images can be large, so allocate at least several hundred gigabytes to this volume"

Now, assume that your opt directory is having 12GB size and you are hosting 200 clouds (organizations) where 50 of them tried to upload 4GB media simultaneously. We are talking about 200GB which is much more that opt size.

This will cause vCD service to fail, i.e. the whole cloud will be down (definitely it won't impact running VMs).

In fact this vCD Transfer Storage is used for the following purposes:

1. Media Upload/Download 

2. Import/Upload vApp Templates

The proper solution for this is to mount vCD transfer storage to an external NFS or other shared storage to provide much more space. Here are the steps.

1. Add new vDisk to vCD server (say 100 GB) 
2. Login to vCD server using SSH 
3. Stop vCD Service


4. Verify the name of your new vDisk (the system already has sda/sdb. Therefore, sdc is new)
 
5. Format the new vDisk.
 
6. Create File System in the new partition
 
7. Mount the new partition to vCD transfer storage
 
You can verify the mounting as follow:
 
8. Edit /etc/fstab directory and add the red line to make sure that vCD server mount the new partition to transfer storage at each boot.

9. Modify the permissions to allow vCD Service to write to the new location.
   
10. Start vCD Service

vCloud Director (5.1.1) Networking - PART 3

How to Configure vCD Networking?

1. Create External Network

2. Create Network Resource Pool

Note that the above pool is vCDNI Backed. This means that we need to adjust the MTU value on vDS as well as physical switches to accommodate VLM header.

3. Configure OvDC Network as part of OvDC configuration to allocate resources to organization. You should have system admin or Org admin privileges to create OvDC Networks. 

In case you want to connect OvDC Network to External Network through vSE, you should have vSE pre-created.

Coming back to OvDC Network creation. Notice below the three types we covered (Isolated, Direct, Routed).

Note: If vSE isn't created prior to OvDC Network creation, Create a routed network by connecting to an existing edge gateway will be greyed-out, i.e. you can't created Routed network.

Since we selected OvDC Network to be connected through vSE, a Portgroup is created in vCenter Infrastructure.

Note: While configuring OvDC, you need to select the type of Network Resource Pool. When creating OvDC Networks in OvDC, the type of portgroup created in vCenter Infrastructure will be based on the type of Network Resource Pool assigned to OvDC.

In case Network Resource Pool is selected as None, you won't be able to create OvDC Network with vSE. The steps will be accepted but the creation will fail.

As you know, we can have different OvDCs within one Organization. This means that we can have multiple OvDC Networks within Organization. vApps in the Organization can connect to different OvDC Networks either directly or through vSE.

4. Configure vApp Networks within organizations

vApp Network Fencing option can be used only for Direct vApp Networks. For Routed vApp Networks, this option will be greyed-out.

To configure vSE used by vApp Networks (Routed or Fenced).

Note: vApp Network vSE and Portgroup won't be deployed in vCenter Infrastructure unless a VM connected to this vApp Network is powered on. This isn't not similar to OvDC Network.

Also, in case you stopped the vApp, all VMs in vApp are stopped, vSE VA and Portgroup will be deleted.

TIP: To get the mapping of vApp Network to Portgroup, Right-Click vApp Network > Properties.

vCloud Director (5.1.1) Networking - PART 2

Back to OvDC Network Resource Pools

We said that OvDC Network can have different types of Portgroups in vCenter Infrastructure environment based on OvDC Network Resource Pool type used. What does this mean?

While creating OvDC network, you need to select whether its directly connected to External network or connected through vSE. In case vSE is selected, you need to select the type of network resource pool to be used. The type of network resource pool will impact the way how vDS Portgroups operate but from vCenter Infrastructure point of view, it will be always a typical vDS Portgroup.

There are four types of network resource pools:

1. Portgroup Backed: In this type, you need to have a pre-created pool of Portgroups in vcenter prior to configuring OvDC Network. While configuring OvDC Network, assuming that this type is selected, the first available Portgroup will be mapped to the OvDC Network. This is one-to-one mapping, i.e. each OvDC Net can be mapped to one Portgroup only and vice-versa. 

2. VLAN Backed: In this type, you need to define a range of VLANs in network resource pool. During OvDC Network configuration, assuming this type is selected, the next available VLAN will be selected and new Portgroup will be created in vSphere using this VLAN. This is one-to-one mapping, i.e. each OvDC Net can be mapped to one Portgroup only and vice-versa. 

3. vCDNI Backed

I would like to brief on what is vCDNI. vCDNI is new protocol developed by VMware (not standardized by IEEE/ITU)  which is used to encapsulate Ethernet frames on VMware Lab Manager (VLM) frames. VLM frames are having Ether-Type as '88de'. The header size of VLM Frames is 24-bytes, i.e. the overall frame size will be 1524 bytes.

Note: This is different than MAC-in-MAC protocol (802.1ah)

By what is the purpose of this encapsulation?

In multi-tenant cloud environment, you need to have some sort of isolation between tenants. This is originally implemented using VLANs. However, we will be limited to 4094 tenants which is the max number of VLANs.

vCDNI basically is similar to private VLANs concept where you will have one primary VLAN but instead of secondary VLANs, you will use vCDNI IDs. In this case multiple tenants will be having same VLAN but different vCDNIs. Tenants with same vCDNI IDs will be able to speak while different vCDNI IDs will require a device to inter-vCDNI routing (this device is vSE).

Using this technology, the max number of tenants will be 4M (1024 IDs per VLAN) theoretically.

How is it implemented?

VMware vDS switch is having code component called PGI which is responsible for generating vCDNI IDs, calculating new MAC addresses for VLM header based on vCDNI IDs, and encapsulating Ethernet Frames in VLM Frames. When a frame reaches vDS, it will check ether-type to verify whether its normal Ethernet frame of VLM frame. In case its VLM Frame, vDS will do reverse calculation to get vCDNI ID from MAC addresses. In case vCDNI IDs are matching, vDS will forward the frame to destination VM based on original MAC, else frame is dropped.

In case VMs are sharing vCDNI IDs across hosts, the frame will be forwarded across physical switches to destination host. Therefore, physical switches should have MTU value increased to 1524-bytes.

Typical vCD Traffic Flow

Coming back to OvDC Networks, while configuring them, assuming vCDNI Backed Pool is selected, a Portgroup will be created having the shared primary VLAN and using the next available vCDNI ID.

Disadvantages of vCDNI (Thanks to Ivan Pepelnjak for highlighting this):

While it sounds good feature, here are the challenges I saw during testing:

a. VLM frame MAC is calculate based in vCDNI ID, VLAN ID, source and destination MAC addresses. The calculation algorith isn't documented anywhere by VMware. Therefore, looking from uplink switch, there is no way to track the VM based on its MAC address. This is sometimes required for network troubleshooting when you have a faulty VM.

b. Its not at all secured when coming to broadcast and multicast traffic.

- When a broadcast message is sent, the destination MAC is FFFF.FFFF.FFFF. When vDS is doing the calculation to generate VLM MAC addresses, VLM destination MAC will remain FFFF.FFFF.FFFF.  This is very dangerous because the destination vDS switches won't be able to extract vCDNI ID from VLM MACs and accordingly will flood the frame to all tenants in all vCDNI IDs sharing same VLAN.

Although have separate VLAN for each tenant isn't scalable but such kind of broadcast storms is limited per VLAN, i.e. per tenant. However, with vCDNI since tenants are sharing same VLAN but different vCDNI IDs, we have 1024 tenants impacted by broadcast storm generated by one infected VM.

Below example for simple ARP broadcast request.

- In case of multicast, vCDNI technology bypass IGMP snooping functionality. IGMP messages are encapsulated in VLM frames and switches can't read them to implement IGMP snooping table. In addition, multicast destination MAC will remain same in VLM frame destination MAC (i.e. destination vDS won't be able to extract vCDNI ID). Combing both points, each multicast frame will be flooded to all tenants in all vCDNIs within same VLAN.

 

4. VXLAN Backed (will be covered in separate post)

What are Isolated Networks?

We mentioned in previous post that during OvDC network & vApp network creation, you can have it directly connected to upper network or through vSE. There is a third type listed which is called Isolated Network. In this case, you will create OvDC/vApp Network which isn't connected to Upper Network. This can be sometimes useful when you create Organizations for testing and development.

From vCenter Infrastructure side, new Portgroup will be created as well as new vSE VA. However, this vSE isn't connected to any external Portgroup.

vApp Network Fencing?

This is a combination of Direct and Routed vApp Networks. When this option is enabled, vApp Network will be connected to OvDC Network using Transparent Firewall (Routed). This transparent firewall is vSE. From the other side, the IP Pool defined in OvDC Network will be extended to vApp Network to be used by VMs (Direct). Also, the gateway of the VMs will be OvDC Network.

Fencing allow the usage of overlapping IP addresses for different VMs in different vApp Networks. Once vSE detects overlapping IPs, it will NAT one of them to different IP in the same pool to avoid conflict at OvDC Network level.

To configure Fencing, vApp Network should be Direct. If vApp Network is Routed, Fencing will be greyed-out.

From vCenter Infrastructure, Portgroups will be created for each vApp Fenced Network and vSE VA will be deployed for each fenced vApp Network.

Notes:

- You can overwrite fencing setting at VM level while deploying it in the vApp. 

- You can't overlap IPs within same Fenced vApp Network. 

- When you create Fenced vApp Network, you can't create any other Direct vApp Network

vCloud Director (5.1.1) Networking - PART 1

This part is covering the basics of vCD Networking.

vCD networking is used to provide network services to vCD elements which are:

1. Organizations

2. vApps 

3. vCD VMs

Those network services include:

1. Internet Connection 

2. Firewall Services 

3. NAT Services 

4. Load Balancer Services 

5. VPN Services

The elements which provide vCD network services are:

1. vShield Edge (vSE) - The firewall component provided by VMware is now upgraded to be present in vCD environment in addition to vCenter environment. 
2. vApp Networks - Each vApp network represents a single wire where VM(s) are connected to. vNIC(s) can have IP addresses assigned manually or obtain from vApp Network IP Pool. 
3. Org vDC Networks - Those represents the next hop where vApp Networks will connect to. vApp network can connect to Org vDC network directly or through vSE component. Single Org vDC network can have one or more vApp networks connected to it. 
4. External Networks - This is the entry/exist point to/from your cloud which connect Org(s) to external physical network. Its always preferred to have one dedicated External Network per Org instead of sharing it across Orgs. Again Org vDC networks can be connected to external networks directly or through vSE.

Those elements can be used in different cloud deployments, below are some examples.

How vCD Network Elements Reflect in vCenter Infrastructure ?

An External Network is always backed by a Portgroup, meaning that a portgroup needs to exist within vSphere before you can create this vCD network object. This portgroup can be on a regular vSwitch, a dvSwitch or you could use Nexus 1KV.

Note: An external network can be a port group shared with non-vCD VMs.

Org vDC Network can be backed by a Portgroup or not. This depends on the type of OvDC Network selected during creation. In case OvDC Network is directly connected to External network without vSE, Portgroup won't be required or created as it will be logical link. Physically, vApp Networks will be directly connected to External Network Portgroup.

However, if OvDC Network is connected to External network using vSE, in this case Portgroup will be created . This Portgroup will include one vNIC from vSE. There are different types of Portgroups created for OvDC networks based on the type of OvDC network resource pool. We will be covering this later.

Similar to OvDC network, vApp network can be backed by Portgroup or not. If vApp network is directly connected to OvDC network, no Portgroup is created as this will be a logical link. Physically, vNICs for VMs are directly connected to OvDC network Portgroup.

However, if vApp network is connected to OvDC network through vSE, Portgroup is created and will have one vNIC from vSE as well as VM's vNIC(s).

Important note: For OvDC network and vApp network, vDS is only supported for their Portgroups. In case PvDC is using N1KV or vSS, OvDC networks and vApp networks can't be created.

Updating / Modifying DVS Switch Configuration from vCenter DB

Yesterday, one of my ESXi hosts crashed (showing purple screen and not booting) and I was forced to rebuild it. This host was already added and used in DVS switch. After rebuilding it and adding it back to vCenter, I wasn't able to add the ESXi host back to DVS. The following was noticed:

1. From vSphere Client, navigate to Inventory > Hosts and Clusters > #Select ESXi host# > Configuration Tab > Networking > VSphere Distributed Switch. You won't see any DVS switch present. 
2. From vSphere Client, navigate to Inventory > Networking > #Select DVS Switch# > Hosts Tab. You will see the ESXi host is already attached to DVS switch.

I tried to remove the host from DVS but wasn't going through and throwing exception error.

This is due to the fact that vCenter DB is still having this host attached to DVS with uplinks attached to dvUplingPG. Therefore, the fix for this is to clear the tables in vCenter DB and re-add ESXi host to DVS switch.

1. Stop VMware VirtualCenter Service in vCenter Server
2. Connect to vCenter DB (I am using MSSQL DB)

3. Identify the IDs of the DVS Switch and ESXi Host in the DB using the following SQL queries.

select * from VPX_ENTITY where name='DVS_NAME';
select * from VPX_ENTITY where name='HOST_IP';

 

4. Run the following queries to clear related DB tables.

delete from vcenter.dbo.VPX_DVHOST  where DVS_ID=#DVS_ID# and HOST_ID=#HOST_ID#

delete from vcenter.dbo.VPX_DVPORT_MEMBERSHIP where HOST_ID=#HOST_ID# and DVS_ID=#DVS_ID#

5. Start VMware VirtualCenter Service.
6. Go back to ESXi Host > Configuration > Networking. Proxy error will popup. 

To understand this error, the HostProxySwitch is a software entity which represents the component of a DistributedVirtualSwitch on a particular host. What is happening is that after removing DVS and its components from vCenter DB, HostProxySwitch component for this particular host isn't able to find records in vCenter DB. You need to uninstall this component to completely remove DVS.

You need to select Remove hyperlink to uninstall it.

7. Re-Add ESXi Host back to DVS Switch