Day to day, a majority of the software engineers at Red Canary are working on our Portal, a customer-facing Ruby on Rails monolith built using many of the most common Ruby gems and tools out there, including:
As with any monolith, clear and consistent patterns are crucial to onboarding new engineers and maintaining productivity among more tenured staff.
Rails as a framework emphasizes convention over configuration, which is one way to make sure there aren’t too many ways of doing things. While this ethos is often in conflict with gems such as RSpec, it’s a pretty robust strategy that you can extend throughout the Ruby ecosystem.
An RSpec-tator sport
Focusing on RSpec as an example, one of the core philosophies of the framework is that your tests are not only tests, but specifications—they both test and document the expected behavior of your code.
This is evident in the syntax of RSpec: given some state or context, this code is expected to behave in some way or to produce some output. At Red Canary, we may see a test written like this:
context 'when a threat is published' do
it 'updates the alert state' do
...
end
end
A natural question then arises: if RSpec encourages programmers to structure their tests this way, what other features can we take advantage of that allow us to write clean, clear specifications that don’t clutter up test files and remain consistent across a large codebase?
Enter RSpec metadata
Metadata is a feature of RSpec that allows developers to pass information within individual tests—or example groups—to modify the underlying behavior of the test suite and/or add custom configuration.
Extending this feature is often highlighted in the context of filtering tests or adding tags to specific example groups in order to focus your test suite. However, there are some more nuanced uses of metadata that spotlight how it can be used to maintain the narrative-like structure that gives RSpec its power.
For the rspec-rails extension, you may have encountered this in the wild with ‘model’ or ‘request’ specs, which allow you to specify that a particular test is for a controller, and therefore gives you access to helper methods to make requests and test responses:
require 'rails_helper'
describe 'My Controller', type: :request do
it 'renders the new page' do
get new_my_controller_url
expect(response).to render_template(:new)
end
end
In the example above, by including the type: :request metadata, we get access to the get, response, and render_template helpers, as well as access to the Rails router that provides routing methods like new_my_controller_url.
What’s going on under the hood here?
If we look at the rspec-rails configuration code, we can see that via the gem’s settings, we have a request-specific shared context included when we define type: :request in the test.
...
config.include RSpec::Rails::ModelExampleGroup, type: :model
config.include RSpec::Rails::RequestExampleGroup, type: :request
config.include RSpec::Rails::RoutingExampleGroup, type: :routing
...
This example group has a lot of nice features that we might want for testing requests, such as custom matchers for HTTP status codes, as well as access to our application’s defined routes without having to manually load the Rails router in each spec.
In the context of request and controller tests, including the request metadata tag is just syntactic sugar for including a shared context that is helpful for testing this specific domain. Many other Ruby frameworks, such as Capybara and ViewComponent, leverage similar metadata tags to limit where and when these helper methods and variables can be used so that they aren’t exposed globally.
Defining your own metadata
RSpec metadata provides a generous amount of flexibility for defining your own way to run a test.
For example, let’s say I have a service that kicks off an asynchronous Sidekiq job to perform some provisioning action on a resource, and I want a straightforward way to make sure that my service correctly executes this action end to end.
Sidekiq provides some hints at how to test jobs inline rather than just queue them up in a fake testing queue that is not processed.
it 'provisions my resource' do
Sidekiq::Testing.inline! do
expect { call }.to change(Resource.provisioned, :count).by(1)
end
end
This works just fine for us! But developers have to define additional lines of code for each test where we want to confirm this behavior.
If we didn’t want to clutter up individual expectation blocks, another strategy would be to add an around hook that executes each test within an inline! block.
around(:example) do |example|
Sidekiq::Testing.inline! do
example.run
end
end
around hooks like this are common patterns in shared contexts. Given we may want to leverage this test configuration elsewhere, we can use metadata to declare under what conditions we want tests to behave this way:
# spec/rails_helper.rb
config.around(:example, :sidekiq_inline) do |example|
Sidekiq::Testing.inline! do
example.run
end
end
Using metadata, now we can rewrite our test:
it 'provisions my resource', :sidekiq_inline do
expect { call }.to change(Resource.provisioned, :count).by(1)
end
This test now invokes the around hook and runs the example inside of a Sidekiq::Testing.inline! block, getting rid of the need to write out the same code in each test. Thinking back to the syntax and principles of RSpec, this also reads more like a specification that makes it clear that there is something asynchronous happening, and that this asynchronous process is what gives us the outcome we are looking for.
In the case where each one of your tests needs to execute these jobs inline, this metadata can actually be passed to an entire example group, negating the need to define this for each context:
# Multiple examples that all test asynchronous sidekiq jobs
describe MyProvisioningService, :sidekiq_inline do
Through this RSpec metadata extension, we’re modifying how tests are run; much like conditionally including a shared context to expose helper methods and DRY up shared state, we’re creating a mechanism to run tests in an explicit way while making the specification clear and readable when we look to the test for documentation.
Using metadata to manage state (in a limited way)
At Red Canary, we use the Flipper gem to define feature flags, ultimately to disconnect the deployment from the delivery of new features.
Enabling Flipper features in tests can yield many different approaches: from stubbing the Flipper module directly, to conditionally enabling a feature for a single actor, and even just enabling it outright. When it comes to writing consistent tests, we can clean this up and get back to our goal of defining a specification—of writing a description of what we expect to happen under what conditions without worrying about how it’s done.
Clear and consistent patterns are crucial to onboarding new engineers and maintaining productivity among more tenured staff.
In RSpec, metadata is available as a hash that is stored as an attribute called metadata on a singleton class that is programmatically defined when the test is executed. This metadata is used at runtime to know things like what file path and line number the test was defined at, which is displayed in the output when an example fails.
This metadata is available inside a shared context when a test is run, and since Flipper uses an in-memory adapter for tests that resets after every example, we can write a simple module that enables all of the features we expect for that example group:
# spec/support/shared_contexts/flipper_context.rb
module FlipperContext
extend RSpec::SharedContext
let(:enabled_features) { Array.wrap(self.class.metadata[:flipper]) }
before do
enabled_features.each { |feature| Flipper.enable(feature) }
end
end
Then in our Rails spec helper, let’s expose our shared context:
# spec/rails_helper.rb
config.include FlipperContext, :flipper
Now we can write a test that enables a given feature before the example is executed:
context 'when :my_feature is enabled', flipper: :my_feature do
it 'runs what I expect' do
...
end
end
This works for multiple features as well:
context 'when my features are enabled', flipper: [:feature_one, :feature_two] do
You can even tie all of this metadata together to write a test that executes inline Sidekiq jobs with a feature flag enabled:
context 'my async feature', :sidekiq_inline, flipper: :my_feature do
With this example, we can take the guesswork out of how we are going to test something and focus on the outcome that we actually want to test against after the feature is enabled.
This is the same ethos as defining a spec as a request spec: you don’t want to bother yourself with accessing routes or parsing responses; instead, you want to focus on the results.
Specs become much cleaner and the tests can center on what we’re testing rather than how we are testing.
Developers beware
While RSpec appears to be pretty explicit on the surface, many abstractions are executed behind the scenes when a test is run, with very few guardrails for preventing a quagmire of complicated tests.
Like any shared context, the example above with Flipper starts to get into this gray area of altering the state of our specification in a way that muddies what exactly we are testing. When writing a shared context, there are tradeoffs, and it’s worth paying attention to when something ventures from useful into a doctrine, and then into an anti-pattern.
The risk and reward with RSpec—and with Ruby in general—is that it’s as flexible as you want to make it.
Introducing shared logic via metadata encourages developers to set state in the context of a test, which is the preferred pattern of RSpec. This largely fits within the principles of RSpec without abusing the contradictory flexibility that the framework injects into the somewhat rigid Rails ecosystem.
Where this flexibility becomes an anti-pattern, however, would be in dynamically configuring too much underlying state and actually modifying what is being tested. Writing a similar metadata hook that takes a symbol and uses it to dynamically populate an ActiveModel record is the logical extreme of this approach, and is well within the bounds of the tooling RSpec gives us:
self.class.metadata[:class_name].constantize.create!(
self.class.metadata[:my_attribute]
)
However, the way RSpec encourages developers to represent tests is to be as explicit as possible—to clearly define a context in which the test is run. Implicit context definitions make tests difficult to maintain, hence why shared contexts don’t generally have any programmatic definitions in them, even when invoked outside of metadata tags.
Committing to a universal standard
As with any large codebase with many contributors, introducing a new pattern is only part of the work involved with streamlining the developer experience. Encouraging and enforcing standards is almost always more complicated and time consuming, but if done right, it helps take the burden off of individual developers and creates a pattern that people feel empowered to contribute to and develop.
One method that has consistently worked for us has been introducing custom RuboCop linting rules to catch, or at least highlight, potential offenses so that developers are aware of other patterns.
RSpec encourages developers to represent tests as explicitly as possible—to clearly define a context in which the test is run.
Catching invocations of Sidekiq::Testing.inline! or Flipper.enable in specs at least encourages developers to consider whether they need to write something that way, and makes it clear that what they are writing is testing something outside the traditional bounds of what these helpers were intended to be used for.
Where else can RSpec metadata take us?
Since metadata is often a thin wrapper for exposing shared context, almost any situation that might benefit from a shared state could be appropriate for some custom metadata. This could include a small hook that proactively authenticates a user in a third-party integration.
Another case would be to define mocks that allow you to neatly inspect the data passed to a third party. In our case, this could include helper methods around Prometheus metrics that allow us to neatly inspect what data was sent and where, much like the dynamic parsing methods provided in request specs.
The risk and reward with RSpec—and with Ruby in general—is that it’s as flexible as you want to make it. Think about the broader system that you are working in and find a way to build off of that functionality in a way that’s not in conflict with it.
Related Articles
What we learned by integrating with Google Cloud Platform
Shrinking the haystack: The six phases of cloud threat detection
Shrinking the haystack: The six phases of cloud threat detection
Shrinking the haystack: Building a cloud threat detection engine