Modeling infrastructure with security and flexibility in mind

by Richard Harding on 13 September 2017

Juju allows the user to model their infrastructure in a clean and simple repeatable way. Often deployments are repeated across different clouds and regions. Sometimes it's repeated from dev to staging to production. Regardless of the way it's repeated, there are some solid practices that users need to follow when taking a model and reusing it. Some bits need to be unique to each deployment. Most of these are security details that need to be different from deployment to deployment. There might also be some specific bits of configuration that regularly vary. In staging the url in the apache config might be staging.jujucharm.com and in production it's jujucharms.com. You need to be able to reuse the model of how the applications, constraints, and common configuration work but make sure there's a clean and simple method of providing the extra unique bits each time you bring up another model. Let's walk through an example model I've created. I'm going to monitor a pair of Ubuntu machines with Telegraf feeding system details to Prometheus. We'll then use Grafana to visualize those metrics. Finally, we want to setup an HaProxy front end for the Grafana so we can provide a proper SSL terminated web site. After it's up and running it looks a bit like this. The first thing we need to do is use the Juju GUI to export a bundle that will be a dump of our model. That's an EXACT dump of everything we've got. We need to edit out the bits of the model that need to be unique from deployment to deployment. Once it's all cleaned up it looks a bit like this.

applications:
  ubuntu:
    charm: "cs:ubuntu"
    num_units: 2
  telegraf:
    charm: "cs:telegraf"
  prometheus:
    charm: "cs:prometheus"
  grafana:
    charm: "cs:grafana"
    options:
      admin_password: CHANGEME
  haproxy:
    charm: "cs:haproxy"
    expose: true
relations:
  - - "ubuntu:juju-info"
    - "telegraf:juju-info"
  - - "prometheus:target"
    - "telegraf:prometheus-client"
  - - "prometheus:grafana-source"
    - "grafana:grafana-source"
  - - "grafana:website"
    - "haproxy:reverseproxy"

A couple of things to note in there are the config values for the Grafana admin password. We want that to be clear that it should be changed. Other than that though, it's a pretty plain model. Where it gets fun is when we leverage new bundle features in Juju 2.2.3.

Overriding config values at deploy time

Juju 2.2.3 provides a new argument to the deploy command, --bundle-config. This flag allows you to pass a filename where that file will override config for the applications in the bundle file that you're deploying. You might use it like this:

juju deploy ./bundle.yaml --bundle-config=production.yaml

So what can we use this for? Well, let's set a unique password for our Grafana admin user. To provide a file with updated config we just mirror the bundle format and point at the application we're targeting like so. Let's edit the production.yaml file to look like this.

applications:
  grafana:
    options:
      admin_password: ImChanged

Note it looks just like the bundle file above with the same keys and we're just setting an admin password of "ImChanged" to prove it's set. We can then deploy the bundle with the --bundle-config argument and when it's done and brought up we can check it was set.

$ juju config grafana admin_password
ImChanged

Reading complex data from a file

That's handy, but sometimes you don't want to just set a new string value but read content from a file. Prometheus can be used to scrape custom jobs. We've used this in the past to scrape prometheus data from Juju controllers themselves. To set this up we need to add a YAML declaration about the job that Prometheus will process. Let's find out what the IP of our Juju controller is and add that job using another new bundle feature; include-file:// Using include-file:// you can specify a path on disk that will be read and passed to the config value in your bundle. In this way you can easily sent complicated multi-line data (like YAML) to a config value in a clean and easy way. First let's setup our new scape job definition.

juju show-machine -m controller 0
...
ip-addresses:
    - 10.0.0.8
    
vim scapejobs.yaml

  metrics_path: /introspection/metrics
  scheme: https
  static_configs:
    - targets: ['10.0.0.8:17070']
  basic_auth:
    username: user-prometheus
    password: testing

Now let's update our production.yaml file to also read this new scrapejobs.yaml file during deployment.

applications:
  grafana:
    options:
      admin_password: ImChanged
  prometheus:
    options:
      scrape-jobs: include-file://scrapejobs.yaml

In order for this to work the file is defined to be in the current working directory. If you want it elsewhere we'll need to define a full path to the file. Now when we run our deploy command we'll both set the grafana password as well as read the new job for Prometheus.

juju deploy ./bundle.yaml --bundle-config=production.yaml
...
juju config prometheus scrape-jobs
  - job_name: juju
    metrics_path: /introspection/metrics
    scheme: https
    static_configs:
      - targets: ['10.0.0.8:17070']
    basic_auth:
      username: user-prometheus
      password: testing

Awesome, now we can do some work with templating out the file and reusing it providing unique IP address for targets as well as custom usernames and passwords as needed from deployment to deployment while keeping the basics of the model intact and reusable.

base64 the included files

There's a third option for including into this production.yaml and that's include-base64://. This allows reading of a local file and base64'ing the contents before getting set into the config. This is helpful for things like ssl keys and such that are unique to different deployments. In our demo case I want to pass in an SSL key to be used with HAProxy so that I can provide HTTPS for accessing the Grafana dashboard. To do this we need to set the ssl_key and ssl_cert config values int he HAProxy charm. Let's update the production.yaml file for this final bit of configuration overriding.

applications:
  grafana:
    options:
      admin_password: ImChanged
  prometheus:
    options:
      scrape-jobs: include-file://scrapejobs.yaml
        haproxy:
    options:
      ssl_key: include-base64://ssl.key
      ssl_cert: include-base64://ssl.crt

With this in place the next time we deploy we get the config values updated with base64'd values.

juju deploy ./bundle.yaml --bundle-config=production.yaml
...wait a bit...
juju config haproxy ssl_key
LS0tLS1CRUdJTiBSU0EgUFJJV...

Now we've constructed a sharable model that can be reused yet easily follow best practices for not putting our passwords and keys into the model which might leak out in some way. These tools provide you the best ways of collaborating on the operations of software at scale and I can't wait to hear about how you're using this to build out the next level of your operations best practices. Hit a question or want to share a story? Tell us about it in IRC, on the mailing list, or just bug me on twitter @mitechie. IRC: #juju on Freenode Mailing list: https://lists.ubuntu.com/mailman/listinfo/juju