Callbacks in Action: sfn & Serverspec
Back to the Blog Page

Callbacks in Action: sfn & Serverspec

by Cameron Johnston | Wednesday, May 4, 2016

We recently wrote about the features in the 3.0 releases of SparkleFormation and the sfn command line interface. Now we want to bring to your attention a somewhat overlooked sfn capability: customizable callbacks.

In this post we’ll show how one can use sfn callbacks to validate provisioned compute resources with Serverspec via callbacks provided by the sfn-serverspec project.

Introduced in version 1.0 of the command line tool, sfn callbacks provide a way to inject functionality into sfn command execution. This allows operators to execute custom code at various stages, e.g. before or after executing a create, update or destroy action.

An example callback

Callbacks are implemented as Ruby classes within the Sfn::Callback namespace. This simple example illustrates a class which provides callback methods which will print messages before and after execution of an sfn command:

module Sfn
  class Callback
    class MyCallback < Callback

      def before(*args)
        puts "this message will appear before every command"
      end

      def after(*args)
        puts "this message will appear after every command"
      end

    end
  end
end

Configuring Callbacks

Callbacks are attached to execution hooks which generally run either prior to, or following a given sfn command’s actual action. By specifying callbacks configuration in your project’s .sfn file, you can specify gems to require, the callbacks that should run on particular hooks, and provide per-callback configuration. For example, imagine gem sfn-custom-callback provides the above MyCallback class which we want to run after each command:

Configuration.new do
  callbacks do
    # specify an array of callbacks to be executed after each sfn command
    after ['my_callback']
  end
end

Now whenever we run a command like sfn create, sfn print or sfn validate, our my_callback callback will execute after the respective create, print or validate command has run:

$ bundle exec sfn list
[Sfn]: Callback after_list my_callback: starting
this message will appear after every command
[Sfn]: Callback after_list my_callback: complete
Name                                                        Created                   Updated              Status               Template Description
ecr-test-blue                                               2016-05-01 08:17:41 UTC                        CREATE_COMPLETE

For executing callbacks across every command, global hooks include:

  • before - callbacks run prior to every command’s exection.
  • after - callbacks run following every command’s execution.
  • default - callbacks run before and after every command execution
  • template - callbacks run after a template has been loaded but before sfn command is executed

Configuring a callback for execution on the default hook effectively allows a callback register all the hooks it provides automatically.

Lets attach our MyCallback example to the default hook:

Configuration.new do
  callbacks do
    # specify an array of callbacks to be executed after each sfn command
    default ['my_callback']
  end
end

… and add a method which will run on the template hook:

module Sfn
  class Callback
    class MyCallback < Callback
      # before and after methods ommitted for clarity
      def template(info)
        puts "template callback ran for template #{info[:sparkle_stack].name}"
      end
    end
  end
end

Now when we validate a template, we see various methods of our MyCallback class firing at different points in the command execution:

$ bundle exec sfn validate --file sensu_enterprise
[Sfn]: Callback template my_callback: starting
template callback ran for template sensu_enterprise
[Sfn]: Callback template my_callback: complete
[Sfn]: Template Validation (aws):  sparkleformation/sensu_enterprise.rb
[Sfn]: Validating: sensu_enterprise
[Sfn]: Callback before_validate my_callback: starting
this message will appear before every command
[Sfn]: Callback before_validate my_callback: complete
[Sfn]: Callback after_validate my_callback: starting
this message will appear after every command
[Sfn]: Callback after_validate my_callback: complete
[Sfn]:   -> VALID

Furthermore, the before and after keywords can be combined with specific sfn commands to attach callbacks to command-specific hooks:

Configuration.new do
  callbacks do
    # only run my_callback callback before executing create command
    before_create ['my_callback']
  end
end

With the above configuration, the before method of our MyCallback class would be executed after every sfn create command, but not after any other command, e.g. sfn update. For more detail on implementing and configuring custom callbacks, see the sfn Callback documentation.

Proof of Concept: sfn-serverspec

Inspired by tools like Test Kitchen, we decided to put sfn callbacks to work running validation tests against provisioned stacks of compute resources. Choosing Serverspec for a proof-of-concept implementation was pretty easy, as it provides a means for us to assert the expected state of the system (e.g. port 80 should be listening, service apache should be running) and receive feedback on the success or failure of those assertions. Serverspec is also integrated with other tools we already use (e.g. Test Kitchen, rspec).

The result is the sfn-serverspec gem, which we add to the Gemfile’s :sfn group when using Bundler:

group :sfn do
  gem 'sfn-serverspec'
end

After running bundle you should now have a sfn serverspec command:

$ bundle exec sfn serverspec --help

NOTE: adding a sub-command is not a built-in feature of sfn callbacks, but is made possible by sfn’s bundler loading behavior

Later we’ll discuss how this command can be used for on-demand validation of compute resources. For now, let’s edit our .sfn configuration file to run our validator on every supported callback:

Configuration.new
  callbacks do
    default ['serverspec_validator']
  end
  ...
end

At this point sfn is configured to load our serverspec_validator callback and run it at every hook, but we haven’t done anything to indicate which resources we wish to test, nor the source of the tests to run. This is accomplished by adding serverspec configuration to template resources. Similar to the in-line policy processing of the built-in Stack Policy callback, our Serverspec callback will cache any serverspec configuration provided on compute resources and remove it from the final rendered template.

NOTE: As of this writing, sfn-serverspec supports only AWS::EC2::Instance or AWS::AutoScaling::AutoScalingGroup resources, but we expect to remove this limitation in the future so as to support other compute providers.

Callbacks in Action: Validating Stacks

For your consideration, a simple set of Serverspec examples for an all-in-one Sensu Enterprise server, written to spec/sensu_enterprise/sensu_enterprise_spec.rb, relative to our SparkleFormation project root:

require 'serverspec'

describe 'sensu enterprise' do

  %w( sensu-enterprise sensu-enterprise-dashboard sensu-client redis-server ).each do |svc|
    describe service(svc) do
      it { should be_running }
    end
  end

  describe file('/etc/sensu/conf.d/client.json') do
    it { should exist }
  end

end

In the relevant SparkleFormation template, we specify the file globbing pattern for loading the Serverspec assertions that apply to this resource:

resources(:sensu_enterprise_ec2_instance) do
  type 'AWS::EC2::Instance'
  properties do
    ...
  end
  serverspec.spec_patterns [File.join(Dir.pwd, 'spec/sensu_enterprise/*_spec.rb')]
end

With this Serverspec configuration in place on compute resources, our assertions will be tested when we create or update a stack.

Shown here, an infrastructure stack which contains nested stacks with compute resources:

$ bundle exec sfn create sensu-puppet-test-green --file infrastructure --defaults
[Sfn]: SparkleFormation: create
[Sfn]:   -> Name: sensu-puppet-test-green
[Sfn]: Events for Stack: sensu-puppet-test-green
# EVENT OUTPUT REDACTED FOR BREVITY
Stack create complete: SUCCESS
Stack description of sensu-puppet-test-green
Outputs for stack: sensu-puppet-test-green
# OUTPUTS REDACTED FOR BREVITY
Outputs for stack: sensu-puppet-test-green-LazyVpcNatSubnetVpc-1J05O5BEA7E0K
# OUTPUTS REDACTED FOR BREVITY
Outputs for stack: sensu-puppet-test-green-SensuEnterprise-1UZP26F1XXAQL
# OUTPUTS REDACTED FOR BREVITY

[Sfn]: Serverspec validating i-692b54f4 (52.207.255.245)

base
  Port "22"
    should be listening
  Process "cfn-hup"
    should be running
  Process "puppet agent"
    should be running
  Command "hostname -f"
    stdout
      should match /^.*.compute-1.amazonaws.com$/

sensu enterprise
  Service "sensu-enterprise"
    should be running
  Service "sensu-enterprise-dashboard"
    should be running
  Service "sensu-client"
    should be running
  Service "redis-server"
    should be running
  File "/etc/sensu/conf.d/integrations/puppet.json"
    should exist

Finished in 2.52 seconds (files took 36 minutes 45 seconds to load)
9 examples, 0 failures

NOTE: The above output indicates our tests took a very long time to load. In reality, this how long it took to provision the entire infrastructure stack (Multi-AZ VPC resources, Sensu Enterprise, etc). We will address this in a future release.

We can also run these assertions against an existing stack on an ad-hoc basis:

$ bundle exec sfn serverspec sensu-puppet-test-green-SensuEnterprise-1UZP26F1XXAQL --file sensu_enterprise
[Sfn]: Serverspec validating stack sensu-puppet-test-green-SensuEnterprise-1UZP26F1XXAQL with template sparkleformation/sensu_enterprise.rb:
[Sfn]: Serverspec validating i-30fa8cad (54.86.197.74)

sensu enterprise
  Service "sensu-enterprise"
    should be running
  Service "sensu-enterprise-dashboard"
    should be running
  Service "sensu-client"
    should be running
  Service "redis-server"
    should be running
  File "/etc/sensu/conf.d/client.json"
    should exist

Finished in 3.22 seconds (files took 2.88 seconds to load)
5 examples, 0 failures

But wait, there’s more! In addition to specifying per-resource specs, we can also modify our .sfn configuration file to specify one or more ‘global’ file globbing patterns, specifying Serverspec assertions which will be run on every compute resource with a serverspec configuration:

Configuration.new
  callbacks do
    default ['serverspec_validator']
  end
  sfn_serverspec do
    global_spec_patterns [File.join(Dir.pwd, 'spec/base/*_spec.rb')]
  end
  ...
end

Now, any matching specs in the base directory will be automatically added to our list of assertions:

$ cat spec/base/base_spec.rb
require 'serverspec'

describe 'base' do
  describe port(22) do
    it { should be_listening }
  end

  describe process('cfn-hup') do
    it { should be_running }
  end

  describe process('puppet agent') do
    it { should be_running }
  end

  describe command('hostname -f') do
    its(:stdout) { should match(/^.*.compute-1.amazonaws.com$/) }
  end
end

$ bundle exec sfn serverspec sensu-puppet-test-green-SensuEnterprise-1UZP26F1XXAQL --file sensu_enterprise
[Sfn]: Serverspec validating stack sensu-puppet-test-green-SensuEnterprise-1UZP26F1XXAQL with template sparkleformation/sensu_enterprise.rb:
[Sfn]: Serverspec validating i-30fa8cad (54.86.197.74)

base

  Port "22"
    should be listening
  Process "cfn-hup"
    should be running
  Process "puppet agent"
    should be running

sensu enterprise
  Service "sensu-enterprise"
    should be running
  Service "sensu-enterprise-dashboard"
    should be running
  Service "sensu-client"
    should be running
  Service "redis-server"
    should be running
  File "/etc/sensu/conf.d/client.json"
    should exist

Finished in 4.72 seconds (files took 2.98 seconds to load)
8 examples, 0 failures

The sfn-serverspec callback supports additional configuration, including specifying ssh proxy commands for executing tests via a bastion host. See the project readme for more details.

What’s Next?

As useful as this particular Serverspec testing tool is, we are even more excited by the possibilities that sfn callbacks open up for a variety of use cases. We hope you’ll check out the sfn Callback documentation for more detail on implementing custom callbacks, and let us know how you put callbacks to use in your own deployments.

Happy provisioning!