Velostrata v2.5 Release Notes

New Functionality

Public REST API for Automation

In addition to a GUI, Velostrata can connect with third-party management tools for automation, monitoring and creation of catalogs. In previous versions, a PowerShell module was provided. From version 2.5, a new REST API is provided too. The API is currently at preview stage and may still change in future versions.

The API enables third-party systems to manage:

  1. Cloud Extensions
  2. VMs
  3. Tasks

Functions includes getting inventories, status information and applying various operations such as create Cloud Extension, move VM to cloud and VM migration.

The API is served by a REST Web service, protected with basic HTTP authentication. API input and output objects are serialized in JSON format.

You may find complete documentation for the API in the Velostrata documentation site here, or you may access it from the Velostrata Manager landing page.

Reference implementations for the REST API are also available in the Velostrata github community:

Azure: Using Internal Azure DNS Server

In some cases, network access for a public DNS server is not available. In such cases, Cloud Extension nodes may be impacted. The system can now overcome this by working with the internal Azure DNS server.

Azure: Encryption of Cache and Object Store

All data in Azure cloud extension, including cache and blob object store is now encrypted using Azure SSE (Storage Service Encryption):  256-bit AES encryption.

AWS: Auto-Move-Back of Spot Instances

When a spot in AWS is terminated (e.g. due to marketplace price threshold or time limited), it is now automatically moved back on-prem in order to avoid needless human intervention.

AWS: Page File on EBS Drive

Velostrata configures VMs to keep page files on ephemeral storage, which is provided for some of the instances. In this release, Velostrata optimizes performance for the instances that do not have ephemeral storage, by creating a small dedicated volume and configuring VMs to keep page file on it. This improves VM performance, reduces load on Velostrata Edge and reduce traffic over VPN to the on-prem system.

AWS: Windows 2016

Velostrata operations for Windows 2016 VMs are now supported in AWS as beta.

New: Security and Compatibility Improvements

Support for HTTP(s) Proxy in the Cloud Extension

IT managers may choose to divert HTTP(S) traffic through an HTTP(S) proxy for reasons of security and network monitoring.

Today, communication from the Velostrata manager and edge node on-prem supports a proxy configuration. From version 2.5, several changes were made:

  • There is no more direct access to AWS S3 for telemetry and support services. Rather, this is implemented as an API gateway with communication over HTTPS.
  • Velostrata CloudExtension also supports the option to communication with the Velostrata proactive support service through an HTTPS Proxy.

Note that you can define communication through the proxy to include either control only, or also include data-plane communication. The latter option may require significant bandwidth and have proxy scale impactions.  

When implementing such configuration, the following steps may be required

  • Define the proxy during Velostrata Cloud Extension creation process.
  • If you would like to inspect the SSL traffic by the proxy, the proxy’s SSL certificate needs to be configured in the Velostrata Cloud Extension.
  • If you apply URL whitelisting, you will need to whitelist the following URLs:
    • AWS:

windows.core.net

qt1.velostrata.com

.amazon.com

.cloudfront.net

telemetry1.prod.velostrata.com

  • Azure:

windows.core.net

login.microsoftonline.com

management.azure.com

qt1.velostrata.com

telemetry1.prod.velostrata.com

AWS: Using VPC Endpoints

Previously access to S3 required public internet access. With the introduction of AWS VPC endpoints http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-endpoints.html, Velostrata Cloud Extension nodes now communicate with S3 without leaving the AWS network. VPC endpoints are virtual devices, they are horizontally scaled, redundant, and highly available VPC components. This direct communication reduces risks and bandwidth constraints on your network traffic.

Cloud Credential Management

In CloudExtension creation process, you no longer need to provide the actual credentials (password). Instead the system can save credentials and allow using them in later stage by reference only.

See operation instructions here: Adding a Cloud Extension.

Management Server Dual Networks Connectivity

Separate networks for management and internet/VPN access are now supported. This can be accomplished by defining an additional NIC from the Velostrata Manager landing page.

New: Operation and Maintenance Improvements

System Health Indicators are Now Available to System Admin

The Velostrata system keeps track of the health of system components at all times (at one minute intervals). This information is now available for system admins directly from the vCenter UI.

Healthchecks can also be retrieved from our APIs using commands:

  • REST:  GET / cloudextensions
  • Powershell:  Get-VelosCe

The information is available per system component and allows quick and simple view of major faults to enable fast issue resolution.

The complete list of health-checks can be found here Understanding Velostrata Health Checks.

Integration with vCenter Events and Alarms

Velostrata is integrated with VMware vCenter in order to allow system administrators ease of system operation using a single system.

From version 2.5, Velostrata integrates with vCenter alarms and events to enable notifications, custom actions, and event propagation in vCenter and related products. This allows for easier monitoring of the system using the vCenter tools already deployed and in use.

The following events were added in version 2.5:

  • Data loss (VM level) - this will only be sent in the event of dual failure in the CE
  • FrontendStarted
  • BackendStarted

The following alarms were added in version 2.5:

  • VM moved to degraded mode
  • CE moved to impaired mode
    • This alarm is set when any CE enters an impaired state
    • This alarm is cleared once all associated CEs are no longer Impaired (either Active/Stopped)

Some alarms are configured by Velostrata to be included by default. Additional alarms can be created by a system administrator.

Easy View of Cloud VM Console Log and Screenshot

To view the cloud VM status, administrators no longer need to open a separate window and access Azure or AWS portals. Instead, you can now view the console log and a screenshot of the VM from vCenter, as if it was running locally.

The console log and screenshot are available from the VM Velostrata portlet in vCenter with a single click. .

Temporary Credentials for Cloud Access

For support purposes, there may be a need for Velostrata or Velostrata partners support engineers to access the CloudExtension components using SSH. In order to do so in a secure way, and without compromising security considerations, the following mechanism has been developed.

The Velostrata management server now enables the creation of credentials for SSH access to CloudExtension components. The credentials have private and public keys which can function as CA certificates. Credentials are passphrase protected and are valid for eight hours only in order to avoid abuse.

Other O&M Improvements

  • Telemetry now supports multiple deployments under the same subscription ID.
  • The number of logs generated in the Velostrata support bundles have been reduced in order to make it easier to upload for support.
  • There is an option to opt-out of support bundle uploads if you do not want to enable them.
  • Status of connectivity to vCenter plugin is now visible from the manager UI.
  • Easy download of support bundle is available in the Velostrata manager UI (under configuration).
  • It is now possible to repair a CE (Velostrata repair operation) in cloud that has attached VMs. Note that the VMs need to be stopped first.
  • When a VM fails to boot in cloud it performs auto-healing, and system reboot if required.
  • The following AWS packages installed on workloads were updated:
    • EC2ConfigServer  upgraded to v4.5.1534.0
    • SSM agent  upgraded to v2.0.645.1

New: Optimizations

AWS: New Instance Types in Use

  • Large CE: m4.2xlarge (first choice), r4.2xlarge (fallback)
  • Small CE: r4.large (first choice), m4.large (fallback)

The new instance types provide higher performance at a lower cost.

Other Performance Improvements

  • Cache read latency optimizations, specifically read-after-write. This is specifically important for CI/CD cases and DevTest cases.
  • Migration throughput optimizations  storage migration is now faster.

New: Compatibility and Installation

ESXi, vCenter Server, and vSphere Web Client Compatibility

This Velostrata release is compatible with vSphere 5.5 U1 or higher and vSphere 6.0 U1 or higher. In certain configurations, ESXi version 5.0 and 5.1 may be used, please contact support for details.

vCenter is a required component in a deployment. At minimum, version 5.5 or higher of vCenter is required even in cases where the ESXi host version is lower.

Web Browser Compatibility

The latest versions of the following web browsers are known to be compatible with the Velostrata vCenter Web Client plugin: Chrome, Internet Explorer, Firefox, Safari.

The vCenter Web Client leverages the Adobe Flash extension in your browser. Velostrata recommends that the Flash plug-in for your browser is updated to the latest version. Use Flash player version 19.0.0.245 or later.

To check your current Flash player version, go to:http://www.adobe.com/software/flash/about/.

Virtual Machine Compatibility

Only virtual machines configured with BIOS virtual firmware are supported. UEFI virtual firmware is not supported.

Velostrata makes use of Virtual Machine snapshots. Virtual Machine disks in Dependent mode (default) and Virtual RDM mode are supported with full functionality.

Virtual Machine disks in Independent mode (persistent and non-persistent) and Physical RDM mode are supported with limited functionality  contact support for details if used.

IDE virtual disks are currentlynot supported. Change the virtual disk type to SCSI to enable move to cloud.

Guest Operating System Compatibility

Windows

Supported Windows versions:

  • Windows Server 2008 R2 SP1 or higher
  • Windows Server 2012 / Windows Server 2012 R2.
  • Windows 2016  supported only for AWS (beta)
  • No pre-installation is required for Windows Virtual Machines.

Linux

Supported Linux distributions and versions:

  • RHEL 6.x, RHEL 7.x, CentOS 6.x, CentOS 7.x, SUSE Linux Enterprise Server 11 SP2 or higher.
  • SUSE Linux Enterprise Server 12 is currently in BETA level support.

You will need to install the pre-requisite Velostrata prep RPM package and its dependencies on the virtual machine you intend to run in cloud. The Velostrata package can remain installed when the virtual machine is running on-premises to allow for a future impromptu run-in-cloud.

Note: Block device naming considerations differ across target cloud provider and depend on underlying virtualization platforms. Velostrata recommends you use either LVM or GUID based device names in configuration files such as those used by the boot loader, file system and mount tables. The Velostrata RPM will automatically check for direct device mapping in typical configuration files and indicate in a warning during installation if any found. A script is provided in the RPM to change such configurations to use GUIDs as device names.

General

Only MBR disks are supported for Guest OS boot disks. Both MBR and GPT disks are supported for data disks.

Cloud Instance Type Compatibility

The following AWS instance types may experience missing driver issues. Refer to the AWS driver lists for specific kernel and driver information.

For C4 types, see:

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/c4-instances.html

For G2 and P2 types, see:

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using_cluster_computing.html

For X1 types, see:

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/x1-instances.html

Known Compatibility Issues

  • c4.8xlarge- Verified to work with CentOS 6.8, 7.2, RHEL 7.2, SUSE 11 SP4 and Windows Server 2008 R2 SP1, Windows Server 2012 R2. Not supported with RHEL 6.x.
  • g2.2xlarge- Verified to work with CentOS 6.8, 7.2, SUSE 11 SP4 and Windows Server 2008 R2 SP1, Windows Server 2012 R2. Not supported with RHEL 6.x, 7.2.
  • x1.32xlarge- Verified to work with CentOS 6.8 and RHEL 7.2, SUSE Linux Enterprise for SAP Applications 11 SP4.Not supported with RHEL 6.x, CentOS 7.2, SUSE 11 and Windows Server.

VPN Connection Bandwidth

A minimum of 20 Mbit/sec symmetric or equivalent VPN connection bandwidth and 0.5-1 Mbit/sec upload BW to cloud is required for acceptable access times when moving a VM to the cloud.

For example, Velostrata recommends a minimum of 50-100 Mbit/sec for production use of up to 100 VMs running in the cloud.

Solved Bugs

Item#
Description
Comment
3502
For workloads running RHEL 6.7 - after Detach, missing default gateway configuration. 
Fixed, added default Gateway. 
3501
RHEL 6.7 workload with multiple volume groups failed to load in cloud.
Fixed, refreshing volume groups.
3556
Telemetry is not supported for multiple deployments with the same subscription ID.
Fixed, now supported. 
3882
Problem in creating CE in Azure in regions where Standard_DS1 instance is not available.
Fixed. By using Standard_DS2_v2.
2811
Azure – in rare cases, a “run-in-cloud” operation may fail. 
Fixed.
3466
Velostrata Manager fails to start an Azure VM that is stopped externally (not using the Velostrata manager UI, PowerShell Module or API), if it is left in Stopped but not Deallocated state.
Fixed.
3472
Cancel detach might fail due to network disruptions
Fixed. Velostrata manager will retry. 
3512
Azure Marketplace error- "500, Code: InternalBillingError when trying to purchase too close to previous cancel" - a "Cancel Detach" operation might fail if initiated less than 15 minutes following a previous detach operation
Fixed. Cancelling of a VM in Azure results in lock on resources for a certain period of time. Velostrata manager will retry longer in order to avoid error due to resources lock. Note that the operation may be prolonged as a result.
2808
Workloads with VMDK size larger than 2TB fail to boot in the cloud
Fixed. 

Known Limitations

Velostrata vCenter UI Plugin does not work properly for vCenter in linked-mode.

Workaround: Use PowerShell or the REST API

When write-back consistency checkpoints are persisted in the vSphere Datastore on-premises, a snapshot rotation and virtual disk consolidation is performed. During consolidation, the virtual machine will have an event showing that its disks require consolidation. This status is temporary and will automatically get resolved when consolidation triggered by Velostrata has completed. A similar behavior occurs when moving a virtual machine to run back on-premises.

Due to vSphere Storage API limitations, throughput achieved on-premises per VMDK is observed to be limited to about 20Mbytes/sec. When workloads are highly concentrated into a single virtual disk, the initial performance seen in the cloud may be limited due to an increase in storage access latency back on-premises. This situation typically resolves itself within minutes, as soon as an active working set is established in the Velostrata cache, in the cloud.

Due to vSphere Storage API limitations, the number of storage access sessions per ESX host is constrained. Velostrata enforces a limit of up to 60 VMDKs per ESX host, for virtual machines that run in the cloud (active). Multiple ESX hosts may be used. Given that these virtual machines are not actually executing on the ESX hosts, these hosts can be of minimal specs.

A system event is not yet implemented for write back that is put on-hold. The write back for a VM running in the cloud is put on hold when the backing vSphere Datastore’s free capacity is lower that 10%. Write back will resume automatically as space is freed up on the datastore.

Workaround: Monitor the Uncommitted Writes counter and correlate it to low-disk space events from the vSphere datastore. When write back is put on-hold, the Uncommitted Writes graph indicates a growing count for longer periods.

#3647: Note that the "collect console log" action does not work in Google Chrome incognito mode.

Workaround: Work in Chrome regular mode.

#3265: The system supports up to 10 SetPowerState parallel requests.

Known Issues

#1257: When Velostrata Prep RPM is installed on a SUSE Linux Enterprise Server 11, the VM obtains a DHCP IP address in addition to an existing static IP configuration. This issue occurs when the VM is started on-premises in a subnet that is enabled with DHCP services.

Workaround: The issue does not occur when the subnet has no DHCP services. There is no connectivity impact for communications with the original static IP address.

#1027: In some cases, when a VM is moved to run-in-cloud while a 3rd party VM-level backup solution holds a temporary snapshot, the Velostrata periodic write-back operations will not complete even after the backup solution deletes the temporary snapshot. The uncommitted writes counter on the VM will show an increasing size and no consistency checkpoint will be created on-premises.

Workaround: Select the Run On-Premises action for the VM and wait for the task to complete, which will commit all pending writes. Then select the Run-in-Cloud action again. Note that committing many pending writes may take a while. Do not use the Force option as this will result in the loss of the uncommitted writes.

#1745: After registering the Velostrata plug-in, creating a Cloud Extension might fail.

Workaround: Un-register the Velostrata plug-in and restart the vSphere web client service, then re-register the plug-in. Contact support if issue persists.

#1744: In rare cases, creating a Cloud Extension in Microsoft Azure fails due to the Azure service reporting it is busy.

Workaround: If the Cloud Extension appears in “impaired” state, retry using the “Repair Cloud Extension” operation. If no Cloud Extension entry is shown, retry the create operation.

#1667: vCenter reboot causes Velostrata tasks in vCenter to disappear from UI. This is a vCenter limitation.

Workaround: Use the Velostrata PowerShell module to monitor Velostrata managed VMs or Cloud Extensions tasks that are currently running.

#1951: With an ESXi host in maintenance mode, if a VM is moved to cloud, the operation will fail but will get stuck in rollback phase.

Workaround: Manually cancel the Run-in-Cloud task, migrate the VM to another ESXi host in the cluster and retry the Run-in-cloud operation.

#1917: A Run-in-cloud operation fails with the error - "Failed to create virtual machine snapshot. The attempted operation cannot be performed in the current state (Powered off)".

Workaround: The VMware VM snapshot file may be pointing to a non-existent snapshot. Contact support for assistance in correcting the issue.

#1808: After running a Workload back on premises - Workload VMDK's are locked. In certain cases, this is due to network disruptions between the Velostrata management appliance and the ESXi host on which the workload is running.

Workaround: The issue will resolve itself after 1-2 hours.

#2551: Workload whose disks were shrunk using VMware-Converter is unable to boot in cloud.

Workaround: VMware-Converter may have replaced the MBR to a non-supported version. Contact support for assistance in correcting the issue.

#2736: During Detach, operation might fail with the following error message "Operation was canceled".

Workaround: Retry the Detach operation.

# 3775: When performing a detach after a cancel detach operation, the action may fail.

Workaround: retry

#3682: In the event of certain system failures, Velostrata components disconnect from vCenter. In this case, an event may not be sent, resulting in the alarm either not being set properly or not being cleared properly.

Workaround: Clear the alarm manually in vCenter.

#3412: In some cases, creating a Cloud Extension fails with the following message: "Failed to Create Cloud Extension: Your previous request to create the named bucket succeeded and you already own it…."

Workaround: Use the Repair CE Velostrata operation.

#3320: For workloads with Windows machine using a retail license, when returning from the cloud, the license is not present.

Workaround: Reinstall the license.

#3281: Export OVA is an available option when VM in cloud is running in cache mode, however, this operation results in a corrupted OVA.

Workaround: Only create OVA after the detach.

#3142: In rare cases, when a cloud component instance is created and system fails before it is tagged, the instance will remain untagged. This will not allow full clean-up or repair of the CE.

Workaround: manually tag the instance, and then run "Repair"

#2596: In Azure, in some cases when canceling the detach operation in the middle of the operation - after the instance is terminated, the VM rolls back on-prem.

Workaround: Instead of cancelling the detach task in process, wait for the task to complete and then run the “Cancel Detach” action.

#3681: Workloads running RHEL7.2 fail in AWS instance type g2.8xlarge.

Workaround: Use a different instance type.

#3779: Workloads running Windows OS fail to boot in Azure using A instance types.

Workaround: Use AV2 instance types.