© 2024 Reid D. McKenzie, all rights reserved.

Cram; a new dotfile manager

31 Aug 2022

Ah dotfiles. Love ‘em or hate ‘em we’ve got to live with ‘em. While Rob Pike has words about Unix hidden files, we (almost all) work on computers and with software whose behavior is determined in large part by hidden files in our home directories. There’s probably a .bashrc or .zshrc and a whole .ssh/ and .config/ directories kicking around on your workstation full of stuff that matters a fair bit to your day-to-day and standing up a new work machine is probably a wasted day of trying to remember homebrew incantations and source software.

A traditional answer to this is a git repo, and some sort of installer. GNU Stow is a well-trod solution, providing the ability to install dotfiles, but you probably need to wrap it in some script to install software. And Stow doesn’t do well at uninstalling files. As with Ansible it’s effectively imperative, not a declarative solution to managing the state of your configs.

Puppet or NixOS could solve this problem, but they’re suuuuper heavyweight for just managing your dotfiles in a portable way and they create bootstrapping problems since you can’t count on them being available.

I’d like to present what started life many years ago as my custom script, and has become a fair bit more - cram (repo, v0.2.0 release).

Cram is a single-file Python 3.6+ zip app, designed to be something you can just check in with your dotfiles and run anywhere using a system python interpreter. Cram provides a package abstraction like Stow with the addition of packages which can exec. Most importantly, Cram hews to immutable infrastructure principles with an execution log, dry-run/diff capabilities and supports automatic removal of installed resources.

Let’s take a quick tour!

Let’s get cramming

you can clone the repo here git clone https://git.arrdem.com/arrdem/cram-demo.git and follow along

Cram doesn’t know anything about $XDG_CONFIG_DIR or your OS’s package manager or even dotfiles at all. What cram does know about is packages, profiles and a state log.

To Cram, a package is a directory under packages.d (this is hardcoded). Such a directory may contain a pkg.toml file, which as with other package file formats may describe dependencies, preparation, installation and post-install steps. An example of such a file is as follows -

[cram]
version = 1

[package]
# The package.require list names dependencies
[[package.require]]
name = "packages.d/some-other-package"

# (optional) The package.build list enumerates either
# inline scripts or script files. These exec as a
# package is 'built', before it is installed.
[[package.build]]
run = "some-build-command"

# (optional) Hook script(s) which occur before installation.
[[package.pre_install]]
run = "some-hook"

# (optional) Override installation behavior.
# By default, everthing under the package directory
# (the `pkg.toml` excepted) treated is as a file to be
# stowed using symlinks.
[[package.install]]
run = "some-install-command"

# (optional) Hook script(s) which occur after installation.
[[package.post_install]]
run = "some-other-hook"

Managing files with cram

Cram is used to “apply” changes to a directory under management. The conventional incantation for this is ./cram apply ~/conf ~/, for managing a home directory or dotfiles.

Let’s look at the usage -

$ ./cram apply --help
Usage: __main__.py apply [OPTIONS] CONFDIR DESTDIR

  The entry point of cram.

Options:
  --execute / --dry-run
  --force / --no-force
  --state-file PATH
  --optimize / --no-optimize
  --require TEXT
  --exec-idempotent / --exec-always
  --help                          Show this message and exit.

By default, Cram will “require” the following packages:

But you can override this by passing --require. For the purposes of this demo, we will just install a fake package. Don’t worry, we aren’t going to actually install anything here.

$ ./cram apply --dry-run --require packages.d/fake . ~/
2022-07-28 22:25:26,521 - __main__ - WARNING - No previous statefile .cram.log
- mkdir ~/.config
- chmod ~/.config 16877
- mkdir ~/.config/fake
- chmod ~/.config/fake 16877
- link ./packages.d/fake/.config/fake/b.conf ~/.config/fake/b.conf
- link ./packages.d/fake/.config/fake/a.conf ~/.config/fake/a.conf

--dry-run (which is also the default behavior) instructed Cram to figure out what to do, but not to do anything. This is a changelog of commands which Cram is proposing to execute against your filesystem. All of these commands are generated by the default stow style installer, and produce an installed state. Were you to use apply --execute, Cram would go ahead and make these changes.

That No previous statefile warning is the secret sauce of Cram. Cram works in terms not just of this log of what changes it will make, but in terms of a persisted log of what changes it has made. This allows Cram to optimize repeated executions to remove installation steps that haven’t changed, while still retaining a precise log of how to get where you are now from an empty slate. This also allows Cram to clean up after itself.

Let’s do a real demo of this.

$ ./cram apply --execute --require packages.d/fake . ~/
- mkdir ~/.config
- chmod ~/.config 16877
- mkdir ~/.config/fake
- chmod ~/.config/fake 16877
- link ./packages.d/fake/.config/fake/a.conf ~/.config/fake/a.conf
- link ./packages.d/fake/.config/fake/b.conf ~/.config/fake/b.conf

So now we’ve got two files and a couple directories on the filesystem we may or may not want. We can see the record of this state as follows -

$ ./cram state .
- mkdir ~/.config
- chmod ~/.config 16877
- mkdir ~/.config/fake
- chmod ~/.config/fake 16877
- link ./packages.d/fake/.config/fake/a.conf ~/.config/fake/a.conf
- link ./packages.d/fake/.config/fake/b.conf ~/.config/fake/b.conf

Were you to delete a file, say rm packages.d/fake/.config/a.conf and then inspect changes -

$ rm packages.d/fake/.config/a.conf
$ ./cram apply --require packages.d/fake . ~/
- unlink ~/.config/fake/a.conf

What Cram did here was compute what the install steps for the current state would be, compare that with the PREVIOUSLY EXECUTED steps, identify a file that is no longer to be installed, and include removing that file in the new plan.

And if we apply --execute our changes, note that the state file DOES NOT include the unlink cleanup instruction. It only contains the steps required to produce the now-current state

$ ./cram apply --execute --require packages.d/fake . ~/
- unlink ~/.config/fake/a.conf
$ ./cram state .
- mkdir ~/.config
- chmod ~/.config 16877
- mkdir ~/.config/fake
- chmod ~/.config/fake 16877
- link ./packages.d/fake/.config/fake/b.conf ~/.config/fake/b.conf

Managing software with Cram

We can also manage the software that consumes our dotfiles with Cram! Let’s look at how homebrew would be installed -

$ ./cram list . packages.d/homebrew
packages.d/homebrew: (PackageV1)
requires:
log:
  - exec /tmp ('/bin/sh', PosixPath('/tmp/stow/e5d3a54761ee43023832d565e11ec4661b84f4ec66629042674b6658993e8cb8.sh'))

Not super helpful - let’s take a look at the pkg.toml

$ cat packages.d/homebrew/pkg.toml

[cram]
version = 1

[package]
require = []

[[package.install]]
run = "[ ! -e /opt/homebrew/bin/brew ] && /bin/bash -c \"$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)\""

Cram has a big escape hatch for letting you run scripts under /bin/sh, and it’s providing run= directives for the package installer (or other hooks). So that Cram can determine when an install script or other hook changes, these scripts get extracted and content-hashed. If the content-hash of a script hasn’t changed, Cram won’t run it when an apply reoccurs. We consider exec to be idempotent (--exec-idempotent) although you can override this default (--exec-always).

Installing homebrew this way lets us write other packages which depend on homebrew, for instance zsh

[cram]
version = 1

[[package.require]]
name = "profiles.d/macos/homebrew"

[[package.install]]
run = "brew install zsh"

This works a treat, but can get repetitive for which we don’t have a good story. Another concern is that, because we use a unique subprocess per script $PATH gets discarded so there isn’t a good pattern for ensuring that say /opt/homebrew/bin stays on $PATH between dependent tasks.

Managing larger configurations with Cram

I’ve talked a couple times at the package.require feature of Cram packages. But that doesn’t tell you too much about how to organize larger configurations.

A profile is a directory under profiles.d or hosts.d which may but need not have a pkg.toml specifying requirements. Where a package is fundamentally a set of installation directives, a profile is a group of packages. For instance profiles.d/emacs/doom-emacs is a package of configurations specific to the Emacs package.

The tricky bit is that a profile IMPLICITLY requires all its subpackages. This is useful for profile and host specific packages - you don’t have to have a bunch of macos-foo packages running around, they could live in profiles.d/macos/* and then a given MacOS host can depend on profiles.d/macos to grab all the relevant configuration.

The hosts.d/demo host provides an example of this pattern by depending on the macos and work profiles as meta-packages.

To see how the demo host would be installed, ./cram apply --require profiles.d/default --require hosts.d/demo . ~/ would do the trick.

Limitations of Cram

One of the problems in software is that authors don’t lay out what problems they are and aren’t trying to solve. So without further ado, here’s what Cram does, doesn’t and will never do.

Cram manages files and scripts to be run when setting up an environment. Cram tries to help you make this all idempotent. Cram is designed to work for smallish amounts of configuration.

While Cram may 1.0.0 at some point, as it stands Cram is already the product of years of incrementally optimizing and thinking about my dotfiles managers when the need to bootstrap a new machine arises.

It is tempting to teach Cram how to read Starlark or some other config format and try to make it more of an ur-Nix, but there’s no reason to. The present incarnation of Cram satisfies my needs, and my Cram configs can zero-touch deploy either a new work macbook or my personal machines. Besides some small tweaks to enable other folks to adopt it, I don’t see major changes coming down the pike.

Happy cramming!

^d