vCloud Director Service Down ... /opt Directory Full

Recently, I had my vCD service down suddenly. After some troubleshooting identified that its due to /opt directory full. Here I am briefing the concept behind it.

Consider an example when you upload media to your cloud organization using vCD. This media will be located physically in the datastore of the OvDC which is providing resources to this organization and will be counted from the storage quota of this organization. To be more detailed, during the upload process, the media will written to the following path in vCD Server:

/$VCLOUD_HOME/data/transfer

From this path it will be rewritten to OvDC datastore.

Example,

[root@EHDF-VCLOUD-01 transfer]# du -sh /opt/vmware/vcloud-director/data/transfer/*

64M     /opt/vmware/vcloud-director/data/transfer/a73ea460-6be7-4c22-a4dd-1520c687da64    !!!... The cookie representing media name

4.0K    /opt/vmware/vcloud-director/data/transfer/cells

16K     /opt/vmware/vcloud-director/data/transfer/lost+found

Typically, $VCLOUD_HOME corresponds to opt/vmware/vcloud-director. This can be verified by browsing the file /etc/profile.d/vcloud.sh

[root@EHDF-VCLOUD-01 transfer]# cat  /etc/profile.d/vcloud.sh

export VCLOUD_HOME=/opt/vmware/vcloud-director

export VCLOUD_MAX_FD=65535

The problem is that vCD will keep the uploaded media in this path for 24hrs before quarantining it. It won't delete it immediately after writing it to datastore. This is mentioned in vCloud Director Installation and Configuration Guide

"Uploads and downloads occupy this storage for a few hours to a day. Transferred images can be large, so allocate at least several hundred gigabytes to this volume"

Now, assume that your opt directory is having 12GB size and you are hosting 200 clouds (organizations) where 50 of them tried to upload 4GB media simultaneously. We are talking about 200GB which is much more that opt size.

This will cause vCD service to fail, i.e. the whole cloud will be down (definitely it won't impact running VMs).

In fact this vCD Transfer Storage is used for the following purposes:

1. Media Upload/Download 

2. Import/Upload vApp Templates

The proper solution for this is to mount vCD transfer storage to an external NFS or other shared storage to provide much more space. Here are the steps.

1. Add new vDisk to vCD server (say 100 GB) 
2. Login to vCD server using SSH 
3. Stop vCD Service


4. Verify the name of your new vDisk (the system already has sda/sdb. Therefore, sdc is new)
 
5. Format the new vDisk.
 
6. Create File System in the new partition
 
7. Mount the new partition to vCD transfer storage
 
You can verify the mounting as follow:
 
8. Edit /etc/fstab directory and add the red line to make sure that vCD server mount the new partition to transfer storage at each boot.

9. Modify the permissions to allow vCD Service to write to the new location.
   
10. Start vCD Service