Saturday, 16 March 2013

vCloud Connector 2.0 - Data Flow


After having your vCC environment ready with your clouds added, let's see how data flow works. In this example we will copy a VM from vSphere Cloud to vCD Cloud.

First we will list the steps to copy VM using vCC UI.
Note: The target vCD cloud must have at least one Catalog (published or unpublished). Without catalog, copy operation won't take place. We will see in details later.
Note: You need to select the target vApp Network for the VM to be connected. This vApp network will be created always in Fenced mode.

Now let's see what's happening in the background from both vCC UI and vSphere Client.
Now let's explain the output of vSphere Client using blocks.
1. Assuming vCC Server isn't running SSL, vCC UI (vSphere Client or vcloud.vmware.com) will send copy request to vCC Server on TCP port 80. 
2. vCC Server will forward the request on TCP port 443 to vCC Node (since SSL is enabled by default in vCC Node). 
3. vCC Node will send request to vCenter Server (using REST API on TCP port 443) to initiate "Export OVF Template" operation for the source VM. 
4. Accordingly, vCenter Server will locate the VM, start export task, and direct ESXi Server (hosting the VM) that template should be exported in vCC-Node (the path will be vCC-Node transfer storage). 

Here is a small hint about OVF templates.

OVF Template can be exported in two formats:

  1. .ovf: The template is made of multiple files which are the components of OVF.
  1. .ova: The template is made of one file which combines all components.

OVF Template is made of three components:

  1. Manifest file (.mf): This file contains SHA-1 digests of the other files and used by vCenter to verify the integrity of other files. This file is protected by certificate.
  2. OVF Descriptor XML Document (.ovf): This file contains information about the template such as virtual hardware used, requirements, product details, licensing, etc.
  1. Virtual Hard Disk (.vmdk): This represents that virtual hard disk which carries the actual data of the VM. OVF templates support different virtual hard disk formats other than VMDK which can be used by VMware vSphere.

Now let's see this operation from vCC-Node debugs (tail -f /opt/vmware/hcagent/logs/hca.log).

!!! … vCC-Node received request from vCC-Server to start copy operation. Although no connection request is listed, but we can see that request received on tomcat-http service which is the way of communication between vCC-Node and vCC-Server as we mentioned in point 2. The connection between vCC-Node and vCC-Server is pre-established the moment vCC-Node registers with vCC-Server.

[tomcat-http--22] INFO  c.v.hc.agent.service.TaskScheduler - Enter scheduleJob()
[tomcat-http--22] INFO  c.v.hc.agent.service.TaskScheduler - Enter scheduleJob()
[tomcat-http--22] DEBUG c.v.hc.agent.service.TaskScheduler - TaskType copy
[tomcat-http--22] INFO  c.v.hc.agent.service.TaskScheduler - Exit scheduleJob()
[tomcat-http--22] INFO  c.v.hc.agent.service.TaskScheduler - Exit scheduleJob()
[tomcat-http--22] DEBUG c.vmware.hc.agent.business.task.Task - isCompleted = false

!!! ... vCC-Node sent request to vCenter Server (vCenter Server IP is 80.227.54.182) to start Export OVF Template operation as a reaction for copy request.

[taskExecutor-4] DEBUG c.v.h.a.service.TaskProgressUpdater - job name: AgentGroup.copy.copy.4132570d-cb5e-4dcd-8aa9-812898536fcf
[taskExecutor-4] DEBUG c.vmware.hc.agent.business.task.Task - taskName copy, progressLabel Exporting to source node.
[taskExecutor-4] DEBUG c.vmware.hc.agent.business.task.Task - taskName copy, progressLabel Exporting to source node.
[taskExecutor-4] DEBUG c.v.hc.cloud.vsphere.VsphereUtils - ServiceInstance ServiceInstance:ServiceInstance @ https://80.227.54.182/sdk login with key 522661ed-fb14-e4c5-47ba-6be61ec99e27 to https://80.227.54.182/sdk.

!!! ... vCenter Server responds to the request with OVF Template components. The first component is VMDK file which vCC-Node will start downloading. Important to note that vCC-Node will connect directly to ESXi host to download VMDK file from datastore. Therefore, vCC-Node should be able to connect to ESXi Host using https protocol. In our example ESXi host IP is 10.157.102.10.The details of ESXi Host IP and VMDK file name/hash will be provided to vCC-Node by vCenter Server.

[taskExecutor-4] DEBUG c.v.h.a.s.v.i.VsphereStorageAdaptorImpl - Total vmdk bytes to be written: 42949672960.
[taskExecutor-4] DEBUG c.v.h.a.s.v.i.VsphereStorageAdaptorImpl - Downloading https://10.157.102.10/nfc/5207b89e-13d3-d51a-9116-d7c19e76a242/disk-0.vmdk to disk-0.vmdk.
[taskExecutor-4] INFO  c.v.h.a.s.v.i.VsphereStorageAdaptorImpl - SHA1 digest for disk-0.vmdk : 96d36dba9aa974a83dc0eacf6a8bb3730af6029a
[taskExecutor-4] DEBUG c.v.h.a.s.v.i.VsphereStorageAdaptorImpl - Total vmdk bytes written: 72704.

!!! … The second component is OVF descriptor which will be copied from vCenter Server

[taskExecutor-4] DEBUG c.v.h.a.s.v.i.VsphereStorageAdaptorImpl - Writing ovf Descriptor: <?xml version="1.0" encoding="UTF-8"?>
<!--Generated by VMware VirtualCenter Server, User: EHDFCLOUD\baqari, UTC time: 2013-03-15T17:37:10.37785Z-->

… output omitted ...

[taskExecutor-4] DEBUG c.v.h.a.s.v.i.VsphereStorageAdaptorImpl - Adding descriptor.ovf to tar file.
[taskExecutor-4] DEBUG c.v.h.a.s.v.i.VsphereStorageAdaptorImpl - Finished adding descriptor.ovf to tar file.
[taskExecutor-4] DEBUG c.v.h.a.s.v.i.VsphereStorageAdaptorImpl - SHA1 digest for descriptor.ovf: f78f35757d37ba8caa973e70a500addb79aca7a7

!!! … The last component is manifest which is used for integrity check for both VMDK and Descriptor files. In earlier debugs, we noticed that after each task SHA1 value is calculated (its bolded for VMDK file and descriptor file). Each SHA1 value is added now to manifest file.

[taskExecutor-4] DEBUG c.v.h.a.storage.StorageAdaptorUtil - Figuring out manifest file size...
[taskExecutor-4] DEBUG c.v.h.a.storage.StorageAdaptorUtil - Manifest file size: 121
[taskExecutor-4] DEBUG c.v.h.a.storage.StorageAdaptorUtil - Adding manifest to tar file: [ descriptor.mf ]...
[taskExecutor-4] DEBUG c.v.h.a.storage.StorageAdaptorUtil - SHA1(disk-0.vmdk)=96d36dba9aa974a83dc0eacf6a8bb3730af6029a
[taskExecutor-4] DEBUG c.v.h.a.storage.StorageAdaptorUtil - added hash for: disk-0.vmdk
[taskExecutor-4] DEBUG c.v.h.a.storage.StorageAdaptorUtil - SHA1(descriptor.ovf)=f78f35757d37ba8caa973e70a500addb79aca7a7
[taskExecutor-4] DEBUG c.v.h.a.storage.StorageAdaptorUtil - added hash for: descriptor.ovf
[taskExecutor-4] DEBUG c.v.h.a.storage.StorageAdaptorUtil - finished writing manifest to tar.
[taskExecutor-4] DEBUG c.v.h.a.storage.StorageAdaptorUtil - Finished adding manifest to tar file: [ descriptor.mf ].

!!! … We noticed also that manifest and descriptor files were compressed in tar file. This is due to the fact that OVA template is generated for the VM which is one file including VMDK, descriptor.mf, manifest. This is the nature of OVA templates as mentioned earlier.

!!! … Now vCC-Node will logout from vCenter Server as export operation is completed.

[taskExecutor-4] DEBUG c.v.hc.cloud.vsphere.VsphereUtils - ServiceInstance ServiceInstance:ServiceInstance @ https://80.227.54.182/sdk logged out from https://80.227.54.182/sdk.
[taskExecutor-4] INFO  c.v.h.a.s.v.i.VsphereStorageAdaptorImpl - Done exporting OVA archive.
[taskExecutor-4] INFO  com.vmware.hc.agent.jobs.CopyJob - Exported from source to archive file: 4132570d-cb5e-4dcd-8aa9-812898536fcf/TEST-VM.ova

5. The initial copy request sent from vCC-Server to vCC-Node includes copying the OVA template after export to destination vCC-Node transfer storage. Copying between vCC-Nodes uses checkpoint-restart. This is good to resume copying in case transfer fails instead of zero-restart.


!!! … In our example the destenation vCC-Node IP is 213.132.34.10

[taskExecutor-9] DEBUG c.v.hc.agent.impl.AgentAdaptorImpl - Connection to the agent: https://213.132.34.10/agent/api/v2/org/ehdf-private-cloud is expired. Creating new connection
[taskExecutor-9] DEBUG c.v.hc.agent.impl.AgentAdaptorImpl - Renewed RestTemplate for agent https://213.132.34.10/agent/api/v2/org/ehdf-private-cloud , the new expiry time is 1363430391942 Date: Sat Mar 16 14:39:51 GST 2013
[taskExecutor-9] DEBUG c.vmware.hc.agent.business.task.Task - taskName copy, progressLabel Transferring to destination node.
[taskExecutor-9] DEBUG c.vmware.hc.agent.business.task.Task - taskName copy, progressLabel Transferring to destination node.
[tomcat-http--9] DEBUG c.vmware.hc.agent.business.task.Task - isCompleted = false
[taskExecutor-9] DEBUG c.v.h.a.i.RemoteAgentStorageAdaptorImpl - Archive file transferred 1.0. Total bytes transferred: [81708, 81708]
[taskExecutor-9] DEBUG c.v.h.a.i.RemoteAgentStorageAdaptorImpl - Archive file transferred 1.0. Total bytes transferred: [81708, 81708]
[taskExecutor-9] INFO  com.vmware.hc.agent.jobs.CopyJob - Copied to destination node archive file: /817479d6-7094-446e-aceb-847924ad205b/Copy_of_TEST-VM

6. Once copy is completed, the destination vCC-Node will connect to vCD Server using REST API to start importing OVA template as vApp Template in Org Catalog. Next will be importing vApp from vApp Template. Both actions will require communication in the background between vCD Server and associated vCenter Server which we have seen in vSphere Client Snapshot above. 

 Note: This is the reason for saying earlier that we need to have at least one Org Catalog for copy to be completed successfully.

[tomcat-http--26] DEBUG com.vmware.hc.vcd.VcdUtils - Updating vcloudclient version to V5_1.
[tomcat-http--26] DEBUG com.vmware.hc.vcd.VcdUtils - Created VcloudClient com.vmware.vcloud.sdk.VcloudClient@3ca1b989.
[tomcat-http--26] DEBUG com.vmware.hc.vcd.VcdUtils - VcloudClient com.vmware.vcloud.sdk.VcloudClient@3ca1b989 login with vcloudToken ZxbbBTI/A5GBWGv71Kh+529aWqMCnBpvRnQC13oHkYk= to https://vcd.ehdf.com.
[tomcat-http--26] DEBUG c.v.h.c.impl.CloudConnectionFactory - Created VcloudClient$$EnhancerByCGLIB$$841d9ec {}.
[tomcat-http--26] DEBUG com.vmware.hc.vcd.VcdUtils - VcloudClient com.vmware.vcloud.sdk.VcloudClient@3ca1b989 logged out from https://vcd.ehdf.com/api.
[taskExecutor-4] DEBUG c.v.h.a.service.TaskProgressUpdater - job name: AgentGroup.import.import.0668f520-3e3a-4502-a750-04810e09acf9
[taskExecutor-4] DEBUG c.v.h.a.service.TaskProgressUpdater - jobName import
[taskExecutor-4] DEBUG c.vmware.hc.agent.business.task.Task - taskName import, progressLabel The task has completed
[taskExecutor-4] DEBUG c.vmware.hc.agent.business.task.Task - taskName import, percent complete 100
[taskExecutor-4] DEBUG c.vmware.hc.agent.business.task.Task - taskName import, state COMPLETED
[taskExecutor-4] DEBUG c.vmware.hc.agent.business.task.Task - taskName import, completed true

Troubleshooting vCloud Connector

1. Test the vCC Server connections by logging on to the vCC Server. 
2. Test the connection between the vCC Server and a vCloud Director cloud:
3. Test the connection between the vCC Server and a vCenter Server: 4. Test the connection between the vCC Server and a vCC Node: 5. If you are surfacing the vCloud Connector UI at vcloud.vmware.com, test the connection between vCC Server and the website: 6. Test the vCC Node connections used in the copy path by first logging on to the vCC Node located in the vSphere internal cloud. 
7. Test the connection between the vCC Node and the vCenter Server:
8. Test the connection between the vCC Node and the ESX host: 9. Test the connection between the vSphere vCC Node and a vCloud Director vCC Node outside the firewall: 10. Log on to the vCloud Director vCC Node. 
11. Test the connection between the vCloud Director vCC Node and the vCloud Director cloud:

You can check the logs of vCC-Server and Node using GUI or Console. CLI Logs path is /opt/vmware/hcserver_or_hcagent/logs/hcs.log_or_hca.log. To restart vCC-Node or vCC-Server service, use the command service tomcat-hcagent restart or service tomcat-hcserver restart.
Posted by at

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)