vCloud Director Service Down ... /opt Directory Full

Recently, I had my vCD service down suddenly. After some troubleshooting identified that its due to /opt directory full. Here I am briefing the concept behind it.

Consider an example when you upload media to your cloud organization using vCD. This media will be located physically in the datastore of the OvDC which is providing resources to this organization and will be counted from the storage quota of this organization. To be more detailed, during the upload process, the media will written to the following path in vCD Server:

/$VCLOUD_HOME/data/transfer

From this path it will be rewritten to OvDC datastore.

Example,

[root@EHDF-VCLOUD-01 transfer]# du -sh /opt/vmware/vcloud-director/data/transfer/*

64M     /opt/vmware/vcloud-director/data/transfer/a73ea460-6be7-4c22-a4dd-1520c687da64    !!!... The cookie representing media name

4.0K    /opt/vmware/vcloud-director/data/transfer/cells

16K     /opt/vmware/vcloud-director/data/transfer/lost+found

Typically, $VCLOUD_HOME corresponds to opt/vmware/vcloud-director. This can be verified by browsing the file /etc/profile.d/vcloud.sh

[root@EHDF-VCLOUD-01 transfer]# cat  /etc/profile.d/vcloud.sh

export VCLOUD_HOME=/opt/vmware/vcloud-director

export VCLOUD_MAX_FD=65535

The problem is that vCD will keep the uploaded media in this path for 24hrs before quarantining it. It won't delete it immediately after writing it to datastore. This is mentioned in vCloud Director Installation and Configuration Guide

"Uploads and downloads occupy this storage for a few hours to a day. Transferred images can be large, so allocate at least several hundred gigabytes to this volume"

Now, assume that your opt directory is having 12GB size and you are hosting 200 clouds (organizations) where 50 of them tried to upload 4GB media simultaneously. We are talking about 200GB which is much more that opt size.

This will cause vCD service to fail, i.e. the whole cloud will be down (definitely it won't impact running VMs).

In fact this vCD Transfer Storage is used for the following purposes:

1. Media Upload/Download 

2. Import/Upload vApp Templates

The proper solution for this is to mount vCD transfer storage to an external NFS or other shared storage to provide much more space. Here are the steps.

1. Add new vDisk to vCD server (say 100 GB) 
2. Login to vCD server using SSH 
3. Stop vCD Service


4. Verify the name of your new vDisk (the system already has sda/sdb. Therefore, sdc is new)
 
5. Format the new vDisk.
 
6. Create File System in the new partition
 
7. Mount the new partition to vCD transfer storage
 
You can verify the mounting as follow:
 
8. Edit /etc/fstab directory and add the red line to make sure that vCD server mount the new partition to transfer storage at each boot.

9. Modify the permissions to allow vCD Service to write to the new location.
   
10. Start vCD Service

vCloud Director (5.1.1) Networking - PART 3

How to Configure vCD Networking?

1. Create External Network

2. Create Network Resource Pool

Note that the above pool is vCDNI Backed. This means that we need to adjust the MTU value on vDS as well as physical switches to accommodate VLM header.

3. Configure OvDC Network as part of OvDC configuration to allocate resources to organization. You should have system admin or Org admin privileges to create OvDC Networks. 

In case you want to connect OvDC Network to External Network through vSE, you should have vSE pre-created.

Coming back to OvDC Network creation. Notice below the three types we covered (Isolated, Direct, Routed).

Note: If vSE isn't created prior to OvDC Network creation, Create a routed network by connecting to an existing edge gateway will be greyed-out, i.e. you can't created Routed network.

Since we selected OvDC Network to be connected through vSE, a Portgroup is created in vCenter Infrastructure.

Note: While configuring OvDC, you need to select the type of Network Resource Pool. When creating OvDC Networks in OvDC, the type of portgroup created in vCenter Infrastructure will be based on the type of Network Resource Pool assigned to OvDC.

In case Network Resource Pool is selected as None, you won't be able to create OvDC Network with vSE. The steps will be accepted but the creation will fail.

As you know, we can have different OvDCs within one Organization. This means that we can have multiple OvDC Networks within Organization. vApps in the Organization can connect to different OvDC Networks either directly or through vSE.

4. Configure vApp Networks within organizations

vApp Network Fencing option can be used only for Direct vApp Networks. For Routed vApp Networks, this option will be greyed-out.

To configure vSE used by vApp Networks (Routed or Fenced).

Note: vApp Network vSE and Portgroup won't be deployed in vCenter Infrastructure unless a VM connected to this vApp Network is powered on. This isn't not similar to OvDC Network.

Also, in case you stopped the vApp, all VMs in vApp are stopped, vSE VA and Portgroup will be deleted.

TIP: To get the mapping of vApp Network to Portgroup, Right-Click vApp Network > Properties.