SparkleFormation 2.0: now with more clouds
Back to the Blog Page

SparkleFormation 2.0: now with more clouds

by Chris Roberts | Wednesday, January 20, 2016

SparkleFormation originally centered strongly around the AWS CloudFormation template structure. This was mainly due to CloudFormation being the most mature and useful orchestration API when the SparkleFormation library started. As more orchestration APIs are reaching maturity, we’ve refactored the internals of the SparkleFormation library to add native support for providers other than AWS. The 2.0 release of SparkleFormation introduces native support for Azure, Heat, and Rackspace.

SparkleFormation has always been able to generate templates targeted at non-AWS orchestration APIs. What hasn’t been available are builtin helpers for common data structure generation (intrinsic functions and pseudo-parameters) or support for template nesting. These features have always directly targeted AWS CloudFormation templates. The 2.0 release features a refactored internal implementation allowing templates to individually request the target provider.

SparkleFormation.new(:my_stack, :provider => :azure)

By default, templates will automatically be set to target AWS to ensure expected compatibility.

Heat and Rackspace

OpenStack’s Heat template format very closely resembles that of AWS CloudFormation. This allows for a quick and rather painless provider integration. Rackspace runs OpenStack, and in turn provides the Heat API for orchestration. The only difference being that Rackspace only offers a subset of resources supported by OpenStack, and also provides Rackspace specific resources.

For Heat:

SparkleFormation.new(:my_stack, :provider => :heat) do
  dynamic!(:nova_server, :app)
end

or for Rackspace:

SparkleFormation.new(:my_stack, :provider => :rackspace) do
  dynamic!(:cloud_big_data, :backend)
end

Azure

The Microsoft Azure Cloud has recently transitioned to a new portal with a strong focus on the Azure Resource Manager. All services launched using the new portal are created by the Resource Manager orchestration service. Azure also recommends that any new resources be provisioned using the Resource Manager, and that existing resources be migrated. Like other orchestration APIs, Azure’s Resource Manager supports JSON serialized templates.

Now, SparkleFormation can generate Azure format templates:

SparkleFormation.new(:my_stack, :provider => :azure) do
  dynamic!(:compute_virtual_machines, :app)
end

Azure Templates

The Azure Resource Manager template schema is slightly different than that of other orchestration API templates. Most notably, the resources are defined within an Array structure instead of a Hash. While this schema structure can be used within SparkleFormation, helper methods like dynamic! still assume a Hash based data structure. This adds a requirement that all resource names be unique within the template, regardless of the resource type, but it makes authoring the templates much easier. When the SparkleFormation template is compiled and dumped, it will automatically convert the resources Hash data structure to an Array data structure. For example:

SparkleFormation.new(:my_stack, :provider => :azure) do
  dynamic!(:compute_virtual_machines, :app)
end

The internally held data structure can be thought of as:

{
  "resources" => {
    "appComputeVirtualMachines" => {
      "type" => "Microsoft.Compute/VirtualMachines"
    }
  }
}

and when the template is compiled and dumped, the resulting structure will be:

{
  "resources" => [
    {
      "type" => "Microsoft.Compute/VirtualMachines",
      "name" => "appComputeVirtualMachines"
    }
  ]
}

Azure Intrinsic Functions

Azure’s Resource Manager also has a different idea about intrinsic functions. Instead of intrinsic functions being a proper data structure that can be defined and serialized, Azure’s intrinsic functions are specially formatted strings. This requires some new functionality to allow SparkleFormation to build these function calls programmatically, while still producing the expected string style output. Here are some examples ranging from simple to complex:

SparkleFormation.new(:azure, :provider => :azure) do
  simple add!(2, 2)
  nested add!(2, add!(1, 1))
  chained deployment!.properties
  chained_indexed deployment!.properties[0].apiVersion
  chained_params list_keys!(resource_id!('Resource.Type', 'Resource.Name'), '2015-05-01-preview')
  complex reference!(
    concat!(
      'Microsoft.Storage/storageAccounts/',
      parameters!('storageAccountName')
    ),
    providers!(
      'Microsoft.Storage',
      'storageAccounts'
    ).apiVersions[0]
  ).primaryEndpoints.blob
end

which will produce the following JSON (NOTE: Not a valid template)

{
  "simple": "[add(2, 2)]",
  "nested": "[add(2, add(1, 1))]",
  "chained": "[deployment.properties]",
  "chainedIndexed": "[deployment.properties[0].apiVersion]",
  "chainedParams": "[listKeys(resourceId('Resource.Type', 'Resource.Name'), '2015-05-01-preview')]",
  "complex": "[reference(concat('Microsoft.Storage/storageAccounts/', parameters('storageAccountName')), providers('Microsoft.Storage', 'storageAccounts').apiVersions[0]).primaryEndpoints.blob]"
}

Plans

Currently sfn (the SparkleFormation CLI) only supports stack update planning on AWS CloudFormation templates. Heat and Rackspace templates will be the next providers with planning support implemented. There is currently no information available on the behavior of existing resources when properties are updated by the Azure Resource Manager. Once that information is available, we’ll begin to add planner support for Azure.

The Hybrid Cloud

The 2.0 release of SparkleFormation marks the beginning of full hybrid cloud support. SparkleFormation, in combination with sfn and other OSS tooling, can now support deployment of infrastructure to multiple regions, and multiple providers, from a single command.