SparkleFormation 3.0 - Google Cloud
Back to the Blog Page

SparkleFormation 3.0 - Google Cloud

by Chris Roberts | Monday, April 25, 2016

The 3.0 release of SparkleFormation and the SparkleFormation CLI brings a host of new and exciting features. A highlight of this release is the addition Google Cloud Deployment Manager of support.

Google Cloud provides an orchestration API in the form of the Google Cloud Deployment Manager. The 3.0 releases of SparkleFormation and the SparkleFormation CLI (sfn) introduce support for this provider API.

Templates

The Google Cloud Deployment Manager supports 3 formats for defining infrastructure resources templates:

  1. YAML
  2. Jinja
  3. Python

SparkleFormation is based on a declarative approach to infrastructure resources. The focus on declarative templates provides a more stable and reproducible means of creating or re-creating infrastructure resources. It also allows for easier changeset tracking and planning as the entirety of the desired resources are fully defined within the generated template. Due to there being no concept of intrinsic functions represented by data structures like we find in CloudFormation or HEAT, YAML isn’t a valid format due to the requirement of runtime processing (like joining strings or extracting an item from a list). For these reasons SparkleFormation only supports one format of templates for Google Cloud Deployment Manager: jinja

Jinja Templates

What is jinja?

Jinja2 is a full featured template engine for Python.

Jinja allows users to embed Python into files providing support for dynamic generation. In the Ruby world a comparable library would be ERB. This style of template, where strings of evaluated code are included, is very similar to the Azure Resource Manager templates. Strings of evaluated code can be a difficulty in serialized templates, from simple authoring issues like mixing and escaping of quoting styles to the processing of template information and the customization requirement to evaluate code strings instead of proper data structures.

SparkleFormation and Jinja

SparkleFormation already has helpers implemented for building evaluated code strings using Ruby within templates for the Azure Resource Manager. This made implementing the same authoring functionality for Google Cloud Deployment Manager much more straight forward. The helpers provided by SparkleFormation allow authors to represent evaluated code strings as regular Ruby within templates. This ability provides two features: ease of use and expanded flexibility when processing template data.

Using SparkleFormation with GCDM

The Google Cloud Deployment Manager structures templates differently than other providers. Template parameters are not defined inline, rather they are defined within a .schema file that is included with the template. The API calls do not accept parameters for templates on create or update. Parameters are provided directly within the resource declaration which represents the template. Though the structure required by the API is slightly different than other providers, the core concepts are still the same. A template is composed of three important sections:

  1. Parameters
  2. Resources
  3. Outputs

SparkleFormation’s implementation for the Google Cloud Deployment Manager retains the common structure familiar to all SparkleFormation templates being composed of these three sections. When the template is compiled and sent to the remote API, it automatically reformats the template into the structure expected by the API.

Templates in Action

To quickly showcase the features provided by SparkleFormation for Google Cloud Deployment Manager a simple template can be defined for creating a bucket in the object store:

SparkleFormation.new(:example_bucket, :provider => :google) do
  parameters.index_page.type 'string'
  dynamic!(:v1_bucket, :example_bucket) do
    properties.website.mainPageSuffix property!(:index_page)
  end
end

Using the SparkleFormation CLI the logical representation of the template can be displayed via the print command using the --sparkle-dump flag:

$ sfn print --file example_bucket --sparkle-dump
{
  "parameters": {
    "indexPage": {
      "type": "string"
    }
  },
  "resources": {
    "exampleBucketV1Bucket": {
      "type": "storage.v1.bucket",
      "properties": {
        "website": {
          "mainPageSuffix": "{{ properties[\"indexPage\"] }}"
        }
      }
    }
  }
}

The output of this template shows the usage of the property! helper which injects a jinja expression into the template accessing the properties variable automatically defined within the runtime. The output shown is not the valid structure accepted by the API. To view the output sent to the API the --sparkle-dump flag can be removed allowing SparkleFormation to restructure the output making it compatible for the remote API:

$ sfn print --file example_bucket
{
  "imports": [
    {
      "name": "example_bucket.jinja",
      "content": {
        "resources": [
          {
            "type": "storage.v1.bucket",
            "properties": {
              "website": {
                "mainPageSuffix": "{{ properties[\"indexPage\"] }}"
              }
            },
            "name": "exampleBucketV1Bucket"
          }
        ],
        "outputs": [

        ]
      }
    },
    {
      "name": "example_bucket.jinja.schema",
      "content": {
        "info": {
          "title": "example_bucket.jinja template",
          "description": "example_bucket.jinja template schema"
        },
        "properties": {
          "indexPage": {
            "type": "string"
          }
        }
      }
    }
  ],
  "config": {
    "content": {
      "resources": [
        {
          "name": "example_bucket",
          "type": "example_bucket.jinja"
        }
      ],
      "imports": [
        "example_bucket.jinja",
        "example_bucket.jinja.schema"
      ]
    }
  }
}

When SparkleFormation processes the template, it will extract parameters automatically into .schema files, extract templates into .jinja files, and create a root template which consists of the core jinja template.

Resource Naming

Many orchestration APIs provide a concept of Logical Resource IDs and Physical Resource IDs. A Logical Resource ID is the identifier of the resource within the context of the template. A Physical Resource ID is the actual identifer of the generated resource within the target cloud. This is not an enforced concept within Google Cloud Deployment Manager and can be immediately shown by the example template above.

Inspecting the generated templates above it can be seen that the only name defined for the bucket is the name of the resource. This same name will also be used as the name for the actual bucket created, resulting in the logical resource ID being equivalent to the physical resource ID. The impact of this equivalence is two fold:

  1. The generated name is not a valid bucket name
  2. This template can only be used to create a single stack

A manual fix for this would be to explicitly create unique names for the resources as the template is generated. Instead of a customized solution, SparkleFormation provides a simple and consistent helper for generating stack unique names automatically:

SparkleFormation.new(:example_bucket, :provider => :google) do
  parameters.index_page.type 'string'
  dynamic!(:v1_bucket, :example_bucket, :sparkle_unique) do
    properties.website.mainPageSuffix property!(:index_page)
  end
end

which shows a new resource name when the logical representation is generated:

$ sfn print --file example_bucket --sparkle-dump
{
  "parameters": {
    "indexPage": {
      "type": "string"
    }
  },
  "resources": {
    "example-bucket-enlsvijhbw": {
      "type": "storage.v1.bucket",
      "properties": {
        "website": {
          "mainPageSuffix": "{{ properties[\"indexPage\"] }}"
        }
      }
    }
  }
}

And when regenerated with the expected API structure the name is persisted:

$ sfn print --file example_bucket
{
  "imports": [
    {
      "name": "example_bucket.jinja",
      "content": {
        "resources": [
          {
            "type": "storage.v1.bucket",
            "properties": {
              "website": {
                "mainPageSuffix": "{{ properties[\"indexPage\"] }}"
              }
            },
            "name": "example-bucket-enlsvijhbw"
          }
        ],
        "outputs": [

        ]
      }
    },
    {
      "name": "example_bucket.jinja.schema",
      "content": {
        "info": {
          "title": "example_bucket.jinja template",
          "description": "example_bucket.jinja template schema"
        },
        "properties": {
          "indexPage": {
            "type": "string"
          }
        }
      }
    }
  ],
  "config": {
    "content": {
      "resources": [
        {
          "name": "example_bucket",
          "type": "example_bucket.jinja"
        }
      ],
      "imports": [
        "example_bucket.jinja",
        "example_bucket.jinja.schema"
      ]
    }
  }
}

Using the uniqueness helper now allows this template to be used for generating multiple stacks without causing name collisions on resources.

Target SparkleFormation APIs

The implementation of support for the Google Cloud Deployment Manager now rounds out template support for all the major orchestration APIs: