f : ⊥ x ⊥ → ⊥ | Clojure

Shelving; Building a Datalog for fun! and profit?

2018-05-17T13:16:00+00:00

This post is my speaker notes from my May 3 SF Clojure talk (video) on building Shelving, a toy Datalog implementation

Databases?

1960s; the first data stores. These early systems were very tightly coupled to the physical representation of data on tape. Difficult to use, develop, query and evolve.
- CODASYL, a set of COBOL patterns for building data stores; essentially using doubly linked lists
- IBM’s IMS which had a notion of hierarchical (nested) records and transactions
1969; E. F. Codd presents the “relational data model”

Relational data

Consider a bunch of data

[{:type ::developer
  :name {:given "Reid"
         :family "McKenzie"
         :middle "douglas"
         :signature "Reid Douglas McKenzie"}
  :nicknames ["arrdem" "wayde"]}
 {:type ::developer
  :name {:given "Edsger"
         :family "Djikstra"}
  :nicknames ["ewd"]}]

In a traditional (pre-relational) data model, you could imagine laying out a C-style struct in memory, where the name structure is mashed into the developer structure at known byte offsets from the start of the record. Or perhaps the developer structure references a name by its tape offset and has a length tagged array of nicknames trailing behind it.

The core insight of the relational data model is that we can define “joins” between data structures. But we need to take a couple steps here first.

Remember that maps are sequences of keys and values. So to take one of the examples above,

{:type ::developer
 :name {:given "Edsger"
        :family "Djikstra"}
 :nicknames ["ewd"]}

;; <=> under maps are k/v sequences

[[:type ::developer]
 [:name [[:given "Edsger"]
         [:family "Djikstra"]]]
 [:nicknames ["ewd"]]]

;; <=> under kv -> relational tuple decomp.

[[_0 :type ::developer]
 [_0 :name _1]
 [_0 :nickname "ewd"]
 [_1 :given "Edsger"]
 [_1 :family "Djikstra"]]

We can also project maps to tagged tuples and back if we have some agreement on the order of the fields.

{:type ::demo1
 :foo 1
 :bar 2}

;; <=>

[::demo1 1 2] ;; under {0 :type 1 :foo 2 :bar}

Finally, having projected maps (records) to tuples, we can display many tuples as a table where columns are tuple entries and rows are whole tuples. I mention this only for completeness, as rows and columns are common terms of use and I want to be complete here.

foo	bar
1	2
3	4

Okay so we’ve got some data isomorphisms. What of it?

Well the relational algebra is defined in terms of ordered, untagged tuples.

Traditionally data stores didn’t include their field identifiers in the storage implementation as an obvious space optimization.

That’s it. That’s the relational data model - projecting flat structures to relatable tuple units.

Operating with Tuples

The relational algebra defines a couple operations on tuples, or to be more precise sets of tuples. There are your obvious set theoretic operators - union, intersection and difference, and there’s three more.

cartesian product

let R, S be tuple sets ∀r∈R,∀s∈S, r+s ∈ RxS

Ex. {(1,) (2,)} x {(3,) (4,)} => {(1, 3,) (1, 4,) (2, 3,) (2, 4,)}

projection (select keys)

Projection is bettern known as select-keys. It’s an operator for selecting some subset of all the tuples in a tuple space. For instance if we have R defined as

A	B	C
a	b	c
d	a	f
c	b	d

π₍a,b₎(R) would be the space of tuples from R excluding the C column -

A	B
a	b
d	a
c	b

selection

Where projection selects elements from tuples, selection selects tuples from sets of tuples. I dislike the naming here, but I’m going with the original.

To recycle the example R from above,

A	B	C
a	b	c
d	a	f
c	b	d

σ₍B=b₎(R) - select where B=b over R would be

A	B	C
a	b	c

Joins

Finally given the above operators, we can define the most famous one(s), join and semijoin.

join (R⋈S)

The (natural) join of two tuple sets is the subset of the set RxS where any fields COMMON to both r∈R and s∈S are “equal”.

Consider some tables, R

A	B	C
a	b	c
d	e	f

and S,

A	D	E
a	1	3
d	2	3

We then have R⋈S to be

A	B	C	D	E
a	b	c	1	2
d	e	f	2	3

semijoin

This is a slightly restricted form of join - you can think of it as the join on some particular column. For instance, if R and S had several overlapping columns, the (natural) join operation joins by all of them. For instance we may want to have several relations between two tables - and consequently leave open the possibility of several different joins.

In general when talking about joins for the rest of this presentation I’ll be talking about natural joins over tables designed for only overlapping field so the natural join and the semijoin collapse.

Enter Datalog

Codd’s relational calculus as we’ve gone through is a formulation of how to view data and data storage in terms of timeless, placeless algebraic operations. Like the Lambda Calculus or “Maxwell’s Laws of Software” as Kay has described the original Lisp formulation, it provides a convenient generic substrate for building up operational abstractions. It’s the basis for entire families of query systems which have come since.

Which is precisely what makes Datalog interesting! Datalog is almost a direct implementation of the relational calculus, along with some insights from logic programming. Unfortunately, this also means that it’s difficult to give a precise definiton of what datalog is. Like lisp, it’s simple enough that there are decades worth of implementations, re-implementations, experimental features and papers.

Traditionally, Datalog and Prolog share a fair bit of notation so we’ll start there.

In traditional Datalog as in Prolog, “facts” are declared with a notation like this. This particular code is in Souffle a Datalog dialect, which happened to have an Emacs mode. This is the example I’ll be trying to focus on going forwards.

State("Alaska")
State("Arizona")
State("Arkansas")

City("Juneau", "Alaska")
City("Phoenix", "Arizona")
City("Little Rock", "Arkansas")

Population("Juneau", 2018, 32756)
Population("Pheonix", 2018, 1.615e6)
Population("Little Rock", 2018, 198541)

Capital("Juneau")
Capital("Phoenix")
Capital("Little Rock")

Each one of these lines defines a tuple in the datalog “database”. The notation is recognizable from Prolog, and is mostly agreed upon.

Datalog also has rules, also recognizable from logic programming. Rules describe sets of tuples in terms of either other rules or sets of tuples. For instance

CapitalOf(?city, ?state) :- State(?state), City(?city, ?state), Capital(?city).

This is a rule which defines the CapitalOf relation in terms of the State, City and Capital tuple sets. The CapitalOf rule can itself be directly evaluated to produce a set of “solutions” as we’d expect.

?city and ?state are logic variables, the ?- prefix convention being taken from Datomic.

That’s really all there is to “common” datalog. Rules with set intersection/join semantics.

Extensions

Because Datalog is so minimal (which makes it attractive to implement) it’s not particularly useful. Like Scheme, it can be a bit of a hair shirt. Most Datalog implementations have several extensions to the fundimental tuple and rule system.

Recursive rules!

Support for recursive rules is one very interesting extension. Given recursive rules, we could use a recursive Datalog to model network connectivity graphs (1)

Reachable(?s, ?d) :- Link(?s, ?d).
Reachable(?s, ?d) :- Link(?s, ?z), Reachable(?z, ?d).

This rule defines reachability in terms of either there existing a link between two points in a graph, or there existing a link between the source point and some intermediate Z which is recursively reachable to the destination point.

The trouble is that implementing recursive rules efficiently is difficult although possible. Lots of fun research material here!

Negation!

You’ll notice that basic Datalog doesn’t support negation of any kind, unless “positively” stated in the form of some kind of “not” rule.

TwoHopLink(?s, ?d) :- Link(?s, ?z), Link(?z, ?d), ! Link(?s, ?d).

It’s quite common for databases to make the closed world assumption - that is all possible relevant data exists within the database. This sort of makes sense if you think of your tuple database as a subset of the tuples in the world. All it takes is one counter-example to invalidate your query response if suddenly a negated tuple becomes visible.

Incremental queries / differentiability!

Datalog is set-oriented! It doesn’t have a concept of deletion or any aggregation operators such as ordering which require realizing an entire result set. This means that it’s possible to “differentiate” a Datalog query and evaluate it over a stream of incomming tuples because no possible new tuple (without negation at least) will invalidate the previous result(s).

This creates the possibility of using Datalog to do things like describe application views over incremental update streams.

Eventual consistency / distributed data storage!

Sets form a monoid under merge - no information can ever be lost. This creates the possibility of building distributed data storage and query answering systems which are naturally consistent and don’t have the locking / transaction ordering problems of traditional place oriented data stores.

The Yak

Okay. So I went and build a Datalog.

Why? Because I wanted to store documentation, and other data.

95 Theses

Who’s ready for my resident malcontent bit?

Grimoire

Grimoire has a custom backing data store - lib-grimoire - which provides a pretty good model for talking about Clojure and ClojureScript’s code structure and documentation.

https://github.com/clojure-grimoire/lib-grimoire#things

lib-grimoire was originally designed to abstract over concrete storage implementations, making it possible to build tools which generated or consumed Grimoire data stores. And that purpose it has served admirably for me. Unfortunately looking at my experiences onboarding contributors it’s clearly been a stumbling block and the current Grimoire codebase doesn’t respect the storage layer abstraction; there are lots of places where Grimoire makes assumptions about how the backing store is structured because I’ve only ever had one.

Grenada

https://github.com/clj-grenada/grenada-spec

In 2015 I helped mentor Richard Moehn on his Grenada project. The idea with the project was to take a broad view of the Clojure ecosystem and try to develop a “documentation as data” convention which could be used to pack documentation, examples and other content separately from source code - and particularly to enable 3rdparty documenters like myself to create packages for artifacts we don’t control (core, contrib libraries). The data format Richard came up with never caught on I think because the scope of the project was just the data format not developing a suite of tools to consume it.

What was interesting about Grenada is that it tried to talk about schemas, and provide a general framework for talking about the annotations provided in a single piece of metadata rather than relying on a hard-coded schema the way Grimoire did.

cljdoc

https://github.com/martinklepsch/cljdoc

In talking to Martin about cljdoc and some other next generation tools, the concept of docs as data has re-surfaced again. Core’s documentation remains utterly atrocious, and a consistent gripe of the community yearly survey over survey.

Documentation for core is higher hit rate than documentation for any other single library, so documenting core and some parts of contrib is a good way to get traction and add value for a new tool or suite thereof.

Prior art

Datomic is closed source
Lots of prior art with no persistence model
Datahike is married to a particular storage model

You can bolt persistence ala carte onto most of the above with transit or just use edn, but then your serialization isn’t incremental at all.

Building things is fun!

Design goals

Must lend itself to some sort of “merge” of many stores
- Point reads
- Keyspace scans
Must have a self-descriptive schema which is sane under merges / overlays
Must be built atop a meaningful storage abstraction
Design for embedding inside applications first, no server

Building a Datalog

Storage models!

Okay lets settle on an example that we can get right and refine some.

Take a step back - Datalog is really all about sets, and relating a set of sets of tuples to itself. What’s the simplest possible implementation of a set that can work? An append only write log!

[[:state "Alaska"]
 [:state "Arizona"]
 [:state "Arkansas"]
 [:city "Juneau" "Alaska"]
 [:city "Pheonix" "Arizona"]
 ...]

Scans are easy - you just iterate the entire thing.

Writes are easy - you just append to one end of the entire thing.

Upserts don’t exist, because we have set semantics so either you insert a straight duplicate which doesn’t violate set semantics or you add a new element.

Reads are a bit of a mess, because you have to do a whole scan, but that’s tolerable. Correct is more important than performant for a first pass!

Schemas!

So this sort of “sequence of tuples” thing is how core.logic.pldb works. It maintains a map of sequences of tuples, keyed by the tuple “name” so that scans can at least be restricted to single tuple “spaces”.

Anyone here think that truely unstructured data is a good thing?

Yeah I didn’t think so.

Years ago I did a project - spitfire - based on pldb. It was a sketch at a game engine which would load data files for a the Warmachine table top game pieces and provide with a rules quick reference and ultimately I hoped a full simulation to play against.

As with most tabletop war games, play proceeds by executing a clock, and repeatedly consulting tables of properties describing each model. Which we recognize as database query.

Spitfire used pldb to try and solve the data query problem, and I found that it was quite awkward to write to in large part because it was really easy to mess up the tuples you put into pldb. There was no schema system to save you if you messed up your column count somewhere. I built one, but its ergonomics weren’t great.

Since then, we got clojure.spec(.alpha) which enables us to talk about the shape and requirements on data structures. Spec is designed for talking about data in a forwards compatible way, unlike traditional type systems which intentionally introduce brittleness to enable evolution.

While this may or may not be an appropriate trade-off for application development, it’s a pretty great trade-off for persisted data and schemas on persisted, iterated data!

https://github.com/arrdem/shelving#schemas

Values (sha512 hash identity (hasch is excellent))
Records (place identity)

(s/def :demo/name string?)
(s/def :demo.state/type #{:demo/state})
(s/def :demo/state
  (s/keys :req-un [:demo/name
                   :demo.state/type]))

(defn ->state [name]
  {:type :demo/state, :name name})

(s/def :demo/state string?)
(s/def :demo.city/type #{:demo/city})
(s/def :demo/city
  (s/keys :req-un [:demo.city/type
                   :demo/name
                   :demo/state]))

(defn ->city [state name]
  {:type :demo/city, :name name, :state state})

(s/def :demo/name string?)
(s/def :demo.capital/type #{:demo/capital})
(s/def :demo/capital
  (s/keys :req-un [:demo.capital/type
                   :demo/name]))

(defn ->capital [name]
  {:type :demo/capital, :name name})

(def *schema
  (-> sh/empty-schema
      (sh/value-spec :demo/state)
      (sh/value-spec :demo/city)
      (sh/value-spec :demo/capital)
      (sh/automatic-rels true))) ;; lazy demo

Writing!

#‘shelving.core/put!

Recursively walk spec structure
- depth first
- spec s/conform equivalent
Generate content hashes for every tuple
Recursively insert every tuple (skipping dupes)
Insert the topmost parent record with either with a content hash ID or a generated ID depending on record/value semantics.
Create schema entries in the db if automatic schemas are on and the target schema/spec doesn’t exist in the db.

Okay so lets throw some data in -

(def *conn
  (sh/open
   (->MapShelf *schema "/tmp/demo.edn"
               :load false
               :flush-after-write false)))
;; => #'*conn

(let [% *conn]
  (doseq [c [(->city "Alaska" "Juneau")
             (->city "Arizona" "Pheonix")
             (->city "Arkansas" "Little Rock")]]
    (sh/put-spec % :demo/city c))

  (doseq [c [(->capital "Juneau")
             (->capital "Pheonix")
             (->capital "Little Rock")]]
    (sh/put-spec % :demo/capital c))

  (doseq [s [(->state "Alaska")
             (->state "Arizona")
             (->state "Arkansas")]]
    (sh/put-spec % :demo/state s))

  nil)
;; => nil

Schema migrations!

Can be supported automatically, if we’re just adding more stuff!

Let the user compute the proposed new schema
Check compatibility
Insert into the backing store if there are no problems

Query parsing!

Shelving does the same thing as most of the other Clojure datalogs and rips off Datomic’s datalog DSL.

(sh/q *conn
  '[:find ?state
    :in ?city
    :where [?_0 [:demo/city :demo/name] ?city]
           [?_0 [:demo/city :demo.city/state] ?state]
           [?_1 [:demo/capital :demo/name] ?city]])

This is defined to have the same “meaning” (query evaluation) as

(sh/q *conn
      '{:find  [?state]
        :in    [?city]
        :where [[?_0 [:demo/city :demo/name] ?city]
                [?_0 [:demo/city :demo.city/state] ?state]
                [?_1 [:demo/capital :demo/name] ?city]]})

How can we achieve this? Let alone test it reasonably?

Spec to the rescue once again! src/test/clj/shelving/parser_test.clj conform/unform “normal form” round-trip testing!

Spec’s normal form can also be used as the “parser” for the query compiler!

Query planning!

Traditional SQL query planning is based around optimizing disk I/O, typically by trying to do windowed scans or range order scans which respect the I/O characteristics of spinning disks.

This is below the abstractive level of Shelving!

Keys are (abstractly) unsorted, and all we have to program against is a write log anyway! For a purely naive implementation we really can’t do anything interesting, we’re stuck in an O(lvars) scan bottom.

Lets say we added indices - maps from ids of values of a spec to IDs of values of other specs they relate to. Suddenly query planning becomes interesting. We still have to do scans of relations, but we can restrict ourselves to talking about subscans based on relates-to information.

Take all lvars
Infer spec information from annotations & rels
Topsort lvars
Emit state-map -> [state-map] transducers & filters

TODO:

Planning using spec cardinality information
Simultaneous scans (rank-sort vs topsort)
Blocked scans for cache locality

Goodies

API management!

shelving.core is the intentional API for users
shelving.impl is the implementer’s API
import-vars https://github.com/ztellman/potemkin/blob/master/src/potemkin/namespaces.clj#L77

Documentation generation!

Covered previously on the blog - I wrote a custom markdown generator and updater to help me keep my docstrings as the documentation source of truth, and update the markdown files in the repo by inserting appropriate content from docstrings when it changes.

More fun still to be had

What makes Datalog really interesting is that among the many extensions which have been proposed is support for recursive rules.

Negation!

Really easy to bolt onto the parser, or enable as a query language flag
Doesn’t invalidate any of the current stream/filter semantics
Closed world assumption, which most databases happily make

Recursive rules!

More backends!

Transactions!

Local write logs as views of the post-transaction state
transact! writes an entire local write log all-or-nothing
server? optimistic locking? consensus / consistency issues

Ergonomics!

The query DSL wound up super verbose unless you realy leverage the inferencer :c

Actually replacing Grimoire…

Should have just used a SQL ;) but this has been educational

Docs Sketches

2018-02-26T06:18:00+00:00

At present I’m working on developing Shelving, a clojure.spec(.alpha) driven storage layer with a Datalog subset query language. Ultimately, I want it to be an appropriate storage and query system for other project of mine like Stacks and another major version of Grimoire.

In building out this tool, I’ve been trying to be mindful of what I’ve built and how I’m communicating it. What documentation have I written? Is there a clear narrative for this component? Am I capturing the reason I’m making these decisions and the examples that motivated me to come here?

Writing good docs is really hard, doubly so when you’re trying to write documentation largely in docstrings. Docs you write in the docstrings drift rapidly from docs written elsewhere. The only surefire way I’ve come up with to keep the two in sync is to consider your docstrings to be authoritative and export them to other media. So I wanted a tool for that.

Another major problem is documentation organization. In Clojure, namespaces have long known limitations as the unit of structure for your API. Clojure’s lack of support for circular references between namespaces and the need to use explicit forward declarations lead to large namespaces whose order reflects the dependency graph of the artifact not perhaps the intentional API or relative importance of its members. In my first iterations of shelving, I was trying to just use a large .core namespace rather than maintain separate namespaces. I wanted a tool that would let me take a large API namespace and break its documentation up into small subsections which had conceptual relationships.

The result of these two needs is the following script, which I used for a while to generate the docs/ tree of the shelving repo. To work around the issues of having a large, non-logical .core namespace, I annotated vars with ^:categories #{} metadata. This allowed me to build up and maintain a logical partitioning of my large namespace into logical components.

Documentation files are generated by opening an existing file, truncating any previously generated API docs in that category out, and appending newly generated docs to it. This allows me to have Eg. a Markdown title, some commentary, examples and what have you, then automatically splat the somewhat formatted and source linked docs for the relevant category at the end after that header.

I won’t say the results are good. But they beat the heck out of not having any docs at all. Mainly they succeeded in being easier to browse, cross-check and consequently made copy editing docstrings much easier.

With the introduction of distinct “user” and “implementer” APIs to Shelving which happen to share some vars, this categorization is no longer so clear cut. Some docs now belong in one place, some in multiple places. This is an inadequate tool for capturing those concerns. Also it still sorts symbols by line numbers when generating docs 😞

So I’ll be building another doodle, but thought that this one was worth capturing before I do.

Some sample output -

(ns compile-docs
  "A quick hack for building the doc tree based on `^:category` data."
  (:require [shelving.core :as sh]
            [clojure.java.io :as io]
            [clojure.string :as str]))

(def category-map
  {::sh/basic  (io/file "docs/basic.md")
   ::sh/schema (io/file "docs/schema.md")
   ::sh/rel    (io/file "docs/rel.md")
   ::sh/util   (io/file "docs/helpers.md")
   ::sh/query  (io/file "docs/queries.md")
   ::sh/spec   (io/file "docs/spec.md")
   ::sh/walk   (io/file "docs/walk.md")})

(defn ensure-trailing-newline [s]
  (if-not (.endsWith s "\n")
    (str s "\n") s))

(defn relativize-path [p]
  (str/replace p (.getCanonicalPath (io/file ".")) ""))

(defn compile-docs [category-map nss]
  (let [vars (for [ns              nss
                   :let            [_ (require ns :reload)
                                    ns (if-not (instance? clojure.lang.Namespace ns)
                                         (the-ns ns) ns)]
                   [sym maybe-var] (ns-publics ns)
                   :when           (instance? clojure.lang.Var maybe-var)]
               maybe-var)

        groupings (group-by #(-> % meta :categories first) vars)]
    (println groupings)
    (doseq [[category vars] groupings
            :let            [category-file (get category-map category)]
            :when           category-file
            v               (sort-by #(-> % meta :line) vars) ;; FIXME: better scoring heuristic?
            :let            [{:keys [categories arglists doc stability line file]
                              :as   var-meta} (meta v)]]
      (println v)
      (with-open [w (io/writer category-file :append true)]
        (binding [*out* w]
          (printf "## [%s/%s](%s#L%s)\n"
                  (ns-name (.ns v)) (.sym v)
                  (relativize-path file) line)
          (doseq [params arglists]
            (printf " - `%s`\n" (cons (.sym v) params)))
          (printf "\n")
          (when (= stability :stability/unstable)
            (printf "**UNSTABLE**: This API will probably change in the future\n\n"))
          (printf (ensure-trailing-newline
                   (-> doc
                       (str/replace #"(?<!\n)\n[\s&&[^\n\r]]+" " ")
                       (str/replace #"\n\n[\s&&[^\n\r]]+" "\n\n"))))
          (printf "\n"))))))

(defn recompile-docs [category-map nss]
  (doseq [[_cat f] category-map]
    (let [buff      (slurp f)
          truncated (str/replace buff #"(?s)\n+##.++" "\n\n")]
      (spit f truncated)))

  (compile-docs category-map nss))

(defn recompile-docs!
  "Entry point suitable for a lein alias. Usable for automating doc rebuilding."
  [& args]
  (recompile-docs category-map
                  '[shelving.core
                    shelving.query
                    shelving.spec
                    shelving.spec.walk]))

;; In lein:
;; :profiles {:dev {:aliases {"build-docs" ["run" "-m" "compile-docs/recompile-docs!"]}}}
;;
;; Works best with lein-auto to keep your doctree fresh

Intermediate Abstraction

2017-07-27T00:00:00+00:00

This talk was presented at the Bay Area Clojure meetup, hosted by Funding Circle. These notes are presented here with thanks to the organizers and attendees.

1. Preface
2. Context
3. Complexity
4. Abstraction
5. Performance
6. Fin: Purism

1 Preface

Hey folks! For those of you who don't know me, I'm Reid McKenzie. I've been writing Clojure on and off for about four years now. These days I work for Twitter as a Site Reliability Engineer.

Twitter Site Reliability is really small. We're not even 150 engineers tasked with ensuring that all the infrastructure and services required to support the business stay online. It's clear that as the business continues to grow, we in SRE can't continue be traditional reactive system operators. We must be software engineers developing automation and designing systems capable of intervention-less recovery.

Achieving quality in software engineering means we must develop an understanding of how to exercise our powers of abstraction, and that we appreciate and managing the complexity with which we are surrounded. This talk presents some formal and philosophical concepts intended to elevate the ways we approach problem solving which will I hope be useful to you.

2 Context

Programming is a young profession all things told.

If we trace the roots of our profession to Lovelace as she is the first to have written of a machine which could be used for general purpose and perhaps program or modify itself then our profession such as it is dates to the 1830s. Just shy of the two century mark.

Date	Who	Event
1928	Hilbert	Entscheidungsproblem
1933	Godel, Herbrand	μ-recursive functions (simple arithmetic)
1936	Church	λ calculus, models Entscheidungsproblem with reduction
	Turing	Turing machine, models Entscheidungsproblem with halting
1941	Zuse	First programmable, general purpose computer
194{5,6}	Von Neumann	Von Neumann single memory architecture
…
1950	USGOV, Banks	Computers become common place for census & data processing
1956	Bell Labs	Transistors are invented
1970	IBM, Commodore	First transistorized PCs come to the market
1976	Wozniak	Apple I
…
2017		Us, here, today, floundering variously

Computation as a science is grounded in the concepts of evaluation and analysis. The science is in the theoretical limits of what a machine could do. The question of how to achieve something on a given machine is the province of software engineering.

Software engineering is a younger discipline still. We've only really been writing software since the '50s (Excepting Lovelace's programs which never had a machine), so call it 60 years of recognizable programming during the course of which huge shifts have occurred in the industry.

In retrospect one can only wonder that those first machines worked at all, at least sometimes. The overwhelming problem was to get and keep the machine in working order.

– Dijkstra "The Humble Programmer" EWD340 (1972)

The problem with software engineering is, well, for the most part we never get to do it. The profession of programming as adopted in industry is rooted in the now seemingly long tradition of just getting the machine to work at all and then "debugging" it which has carried since the first programs in the '50s. The operations tradition from which SRE programs are usually developed is particularly guilty of this.

Rather than have a concrete body of science to reference and with which to inform decisions, every working programmer develops an aesthetic taste for what seems to them like the right way to solve problems; what kept the computer or systems of computers in working order and solved the problem best. The problem with taste is that it is deeply subjective. It reflects our personal experiences in education, what javascript frameworks we've played with on the side and how we did it at $LASTJOB. When tastes conflict, they can't inform each-other except with great difficulty because there's no underlying theory which can be used to relate and moderate them.

The question is how do we earn the E in our titles. And the answer is not to continue artistically operating the systems we have or continuing to do what seems to work. We must develop and communicate formal understandings for the tools and tactics we use.

3 Complexity

Simple; Simplex; To have but a single strand, to be one.

Complex; To be woven from several strands.

Complect (archaic); To plait together.

Easy; from the French alsie meaning comfortable and tranquil. Not difficult, requiring no great labor or effort.

– Rich Hickey "Simple Made Easy" (2012)

Often, we miss-use simple to mean easy.

Ideally, we want to write code which is both simple and easy. This requires that we be concious both of complexity and of ergonomics. However we only rarely have objective critera for these two critical dimensions.

The fact that a perfectly reasonable engineer may consider a tool sufficient and easy while another may just as reasonably a tool baroque and overbearing is a core motivator for this talk. Sharing an objective metric of simplicity clarifies otherwise subjective questions and enables us to agree upon measurements of solution quality.

Thankfully there is prior art on the subject of attempting to estimate complexity. Karl Popper's Logic of Scientific Discovery builds a framework of formal logic with which to formalize the scientific process itself.

Popper bases his project upon falsifiability and corroboration through experiment. A hypothesis for Popper is a formal statement which is falsifiable because it implies testable outcomes. An experiment cannot confirm a hypothesis, because to do so would imply that no other experiment (state of the entire universe) could possibly falsify the hypothesis. However, we can fail to falsify the claims of the hypothesis, and we can even show that the experiment produced a state which corroborates the prediction(s) of the hypothesis.

Popper proposes that the complexity of a theory can be measured by reasoning about the set of states of the universe which would falsify the theory. These sets can be related to each other and relative complexity measured by establishing implication and subset relations, but this is far from enough to be a really working rule of measurement. Comparing infinities (sets of states of the universe) is hardly intuitive.

The good news is that there are ways we can do this!

Cyclomatic complexity (McCabe) measures the number of control paths through a program. This approach works, and even gives an obvious heuristic for recognizing complexity in the simple number of branches but it is uni-dimensional and doesn't capture the fact that our programs have and manipulate state.

Dataflow complexity is a related family of metrics which try to measure the possibility space of the data (state) in a program. Unfortunately, beyond giving an intuition for the idea that complexity grows with the memory footprint of a program there's not an efficient way to quantify or relate this kind of complexity.

We get into this mess of having to count states in order to talk about complexity because both of these complexity estimators use the model of a Von Neumann finite state machine machine whose states cannot easily be related to each other.

If we go back to Popper, if we have two logical reduction statements P and Q, we can estimate their relative complexity by simply comparing the number of degrees of freedom of each expression. Likewise using the μ calculus (simple arithmetic and reduction) or the λ calculus (function application and reduction) we instantly regain the property that we can understand the complexity of an expression or program merely by looking at the number of terms it involves and counting them.

This intuition is supported by cyclomatic complexity and dataflow complexity metrics because (for programs which halt under reduction) the complexity of a program written with reduction is, well, one. Each datum and program point is unique and occurs only once.

λx.xx λx.xx

→ λx.xx λx.xx

That this must be an estimate, not a precise metric as a divergent combinator such as ω will mess this whole thing right up. But per the halting problem there's not much we can do here so if we accept (as we must) that this is an incomplete analysis and restrict ourselves to the domain of terminating expressions we can still get a lot of utility out of this rule of thumb.

Now, we don't write purely functional programs, and when we do write programs which use functional tools and abstractions there's still global state lying around because we compute on Turing machines not graph reduction engines (those are really hard to build). How do we estimate complexity for real programs then?

It has been suggested that there is some kind of law of nature telling us that the amount of intellectual effort needed grows with the square of program length. But, thank goodness, no one has been able to prove this law. And this is because it need not be true.

– EWD, "The Humble Programmer"

There is room for reasonable disagreement here, but I'd propose a very simple heuristic for estimating complexity; the product of the following properties of a program or program segment.

1 + Number of input parameters
1 + Number of branches
1 + Number of back-edges
1 + Amount of REACHED CAPTURED state which is accessed (maybe 2x cost of a parameter)
1 + Amount of REACHED CAPTURED state which is modified (maybe more than 2x the cost of a parameter)

(defn map [f coll]
  (when coll
    (lazy-seq
      (cons (f (first coll))
            (map f (rest coll))))))

That is, a function which accepts more parameters and takes many branches using them is more complex than a function which accepts a function and a few parameters as parameters and merely delegates to the argument function. The real branching behavior of such an applied higher order function may be complex, but the function itself is simple. It captures little state and does little directly.

(defn log*
  "Attempts to log a message, either directly or via an agent; does not check if
  the level is enabled.
  For performance reasons, an agent will only be used when invoked within a
  running transaction, and only for logging levels specified by
  *tx-agent-levels*. This allows those entries to only be written once the
  transaction commits, and are discarded if it is retried or aborted.  As
  corollary, other levels (e.g., :debug, :error) will be written even from
  failed transactions though at the cost of repeat messages during retries.
  One can override the above by setting *force* to :direct or :agent; all
  subsequent writes will be direct or via an agent, respectively."
  [logger level throwable message]
  (if (case *force*
        :agent  true
        :direct false
        (and (clojure.lang.LockingTransaction/isRunning)
             (*tx-agent-levels* level)))
    (send-off *logging-agent*
              (fn [_#] (impl/write! logger level throwable message)))
    (impl/write! logger level throwable message))
  nil)

...

(declare ^:dynamic *logger-factory*)

...

(defmacro log
  "Evaluates and logs a message only if the specified level is enabled. See log*
  for more details."
  ...
  ([logger-factory logger-ns level throwable message]
   `(let [logger# (impl/get-logger ~logger-factory ~logger-ns)]
      (if (impl/enabled? logger# ~level)
        (log* logger# ~level ~throwable ~message)))))

An expression which does little, but does so using values which it draws from global state such as a configuration value may still be simple, but it is not so simple as a function which accepts that same configuration structure directly as a parameter. This complexity becomes evident when testing, as a test for the simple function merely requires calling the simple function with the appropriate configuration value where testing the globally configured function requires manipulating the state of the entire application so that the shared mutable configuration state conforms to the tests requirements.

We can see that the log macro is actually quite complex becuase it reaches a significant amount of global state despite the fact that the log macro itself doesn't actually do that much. The log macro has to check in a global mutable registry of namespaces to logger implementations for a logger, check the whether that logger is enabled and then do the logging, potentially manufacturing a new logger using the factory from global state if it has to.

(defn mongo!
  "Creates a Mongo object and sets the default database.
      Does not support replica sets, and will be deprecated in future
      releases.  Please use 'make-connection' in combination with
      'with-mongo' or 'set-connection!' instead.
       Keyword arguments include:
       :host -> defaults to localhost
       :port -> defaults to 27017
       :db   -> defaults to nil (you'll have to set it anyway, might as well do it now.)"
  {:arglists '([:db ? :host "localhost" :port 27017])}
  [& {:keys [db host port]
      :or   {db nil host "localhost" port 27017}}]
  (set-connection! (make-connection db :host host :port port))
  true)

A function which changes state may be complex compared to a function which produces no effects and returns a value, but it introduces far more whole program complexity if global state is modified especially if it is modified many times.

This supports the intuition that perhaps sorting a list received as an argument in place doesn't add so much whole program complexity because the effect is restricted in scope to the caller and wherever the list to be modified was sourced from whereas reading and manipulating the ERRNO global value in a C program may directly impact any number of other program behaviors. Such modification defeats referential transparency, and forces us to use sequential logics at the level of the whole program to achieve understanding.

We use multiplication rather than addition to reflect that really what we're trying to approximate is the VOLUME of a multi-dimensional state space, which is measured by the product of the length of the sides, not their sum.

With apologies to Dijkstra, I note that it is quite possible for a program's complexity by this metric alone to grow at least in proportion to the square of the program's length. However there's also still plenty of room at the bottom for simple programs to achieve near-linear complexity growth.

4 Abstraction

We all know that the only mental tool by means of which a very finite piece of reasoning can cover a myriad cases is called “abstraction”; as a result the effective exploitation of his powers of abstraction must be regarded as one of the most vital activities of a competent programmer. In this connection it might be worth-while to point out that the purpose of abstracting is not to be vague, but to create a new semantic level in which one can be absolutely precise.

– EWD, "The Humble Programmer"

What is an abstraction?

Abstractions can be considered to consist of a model, interface and an environment.

The model of an abstraction is the set of operational semantics which it exposes to a user. For instance a queue or channel as an abstraction provides the model that a user writes to it and a consumer reads from it. Reads may be in some order such as FIFO or LIFO, or un-ordered with respect to writes. A bounded queue may provide the additional semantics that, if the writer outpaces the reader it can only get so far ahead before the queue put operation blocks the writer. A Concurrent Sequantial Processes queue provides the operational semantics that when a write occurs the reader is awakened and so only the reader or the writer is ever operating at once.

The interface is the set of operations which the abstraction provides to the user. For instance the channel can have elements enqueued and dequeued. Perhaps the queue can also be closed at one end, denoting that either the source or the destination is finished whereupon the other end will eventually (or immediately) also close.

Abstractions don't exist in a vacuum. They have expectations which they may demand of the outside world (externalities) such as restrictions on how they are used or in what context they are employed. Everything that the model omits is ultimately an assumption about the environment.

It may be helpful to note that the environment is the dual of the model. Anything that is not part of the model must be part of the environment, and by growing the model we can shrink dependency on the environment.

This is deeply significant, in that mismatch between environments and reality results in friction for the user, and tends to be resolved not by abandoning the abstraction but by humans adapting to the restrictions imposed upon them by inadequate tools.

Abstractions may be compared - one said to be simpler than the other - on the basis of the size of the model, interface and expectations imposed on the environment. The model after all is a space of states, the interface a set of functions above, and externalities may be either a set of propositions or a space of states as the two are equivalent.

The ergonomics of abstractions may also be compared - one said to be more consistent than the other - on the basis of the consistency of the names and structure of the abstraction's interface with each-other and predictability of the relationship between the interface, the model and its externalities.

4.1 Examples of Abstraction

Per the Dijkstra quote above, abstraction is all about changing or choosing semantics. I'll give a treatment of linguistic abstraction shortly, but by naming things we can build semantic levels - languages - which convey what we want.

We can abstract over control flow by using higher order functions which provide for instance conditional execution giving us a tool with which we can talk about for instance conditionally applying a transformation.

We can abstract over values by choosing types or decomposing values into their components, using aggregate logical values and references which enable us to refer to parts of our data.

However it is not abstractive to perform computation. When we "compute" (reduce) an expression, we discard information irrecoverably. For instance an average is not meaningfully an abstraction over a time series. It is a datom, the product of some other irrecoverable data under a function. By computing this new value, we've given ourselves an aggregate view which no longer carries the same semantics and cannot meaningfully be considered to be equivalent to its source.

4.2 Limits of Abstractions

Unfortunately, abstractions do have limits.

Traditional types which you may be familiar with, your int and char, are really just raw presentations of what the physical machine can do with a given value. Rarely however do these presentations of machine capabilities map to the senses which are meaningful to us. They are the poorest possible of abstractions - none at all.

#include <stdio.h>
#include <stdint.h>

#define EVAL(...) printf("%s: %d\n", #__VA_ARGS__, __VA_ARGS__)

int main(int argc, char** argv) {
  EVAL((int32_t)(1<<31)-1);
  EVAL((int32_t)(~(1<<31)));
  EVAL((int32_t)(1<<31));
  return 0;
}

(int32_t)(1<<31)-1: 2147483647
(int32_t)(~(1<<31)): 2147483647
(int32_t)(1<<31): -2147483648

How many times in Java have you actually wanted precisely a value with 2³² states of which 2³¹-1 states are said to represent positive numbers, 2³¹ states represent negative numbers and the zeroed bit string represents … zero. And arithmetic wraps around, flipping the sign should we excede the bounds of [-2³¹, …, +2³¹-1]. We also know that while we have a vector of bits … somewhere and one of them is a sign, and we can count on the 32nd bit being the sign we can't actually make endianness assumptions about the bytes in our integer(s). So much for bitmasking tricks.

This abstraction of an int type provides addition, subtraction, multiplication, division, absolute value and of course comparison. All of which we associate with the concept of an integer. Furthermore this abstraction supplies, by specifying model of the machine representation, all the operations we customarily associate with a vector of bits - and, or, xor, shifting, inversion.

If all we want is addition and comparison, the rest of this behavior is not just semantic baggage, it adds complexity. What if I want to count to 2³¹? I have to be aware that this abstraction's model says I'll get -2³¹+1, not the value I expect. Or perhaps an out of band signal that an IntegerOverflowException occurred, which constitutes another cross-cutting concern because now the abstraction implies a system of exceptions with which I must be familiar.

For a large domain of problems, this abstraction does well enough. However we must be constantly aware of its failure modes. We as professionals bear a constant cognitive cost in conforming ourselves to the limits which this abstraction presents to us.

It should be recognized here that Haskell, Lisp(s) and other languages which provide arbitrary precision integer types out of the box capture a simpler programming model by default. A model which has more implementation complexity to be sure and consequently expects more of the environment for instance the presence of a memory manager but the interface is smaller and less thorny.

4.3 Inheritance, Composition & Re-use

Software quality is defined first by solving the business needs of the here and now. If software or strategy or any other tool doesn't do that, it's worthless. Software is a form of physical capital in the same sense as a steam engine or a factory. A factory which can be re-tooled to produce articles of a different design is more valuable than a factory which will require full replacement should the product change. Likewise an engine or assembly line which must be significantly reworked in order to add capacity or change task is less valuable than a design which naturally features a pattern along which it can be changed without significant alteration.

Object Oriented Programming (that is, programming via encapsulation and inheritance) promised that it would improve among other things software reusability when it broke into the scene.

The problem with inheritance as a strategy in practice is that any tool which wants to participate in some abstraction is forced to conform to the expectations of the interface established by the parent which the child wishes to participate in. The interface must accumulate.

Furthermore, reuse through inheritance has the problem of capturing mutable state. An extension or wrapper class must be intimately familiar with the state of the class it wraps and the expectations it may make. The model accumulates.

In "Simple Made Easy" Rich gave a good treatment of this, pointing out that code partitioning doesn't imply decoupling and pointing to Interface Oriented Programming is the response to this problem, because interfaces only bring with them their direct set of expectations.

To be precise about what's happening here - the model of inheritence oriented programming forces us to accreet interfaces and externalities. Thankfully this is not an essential property of abstraction, merely a property of this particular technique.

(defn mapcat [f & colls]
  [f & colls]
  (apply concat (apply map f colls)))

When we take two abstractions and put them together, we say that we've composed them. For instance two functions compose together to make a third. The map function composes with the concatenate function to give us mapcat. map lets us apply a function over many values, and concat allows us to join sequences together. We can thus define mapcat to be a function which lets us join the results of applying a function which produces sequences to many values. The composite model of mapcat requires only the additional constraint that f: a → [b], that is the function to be mapped produces results which can be concatenated.

(def evens #(keep even? %1))

We can build abstractions which have smaller interfaces. For instance if we take a function of a configuration and a datom and we partially apply it with a configuration, we now have a fixed function of a datom. Its model is smaller - we know how it will behave because we've specified whatever configuration(s) the larger base model depends on and the interface is smaller - it consists only of the datom to manipulate.

Building a small wrapper around a large Java API would be a good example of such a composite abstraction with a smaller interface.

We can also build composite abstractions which traverse the model and externality trade-off. For instance we can build abstractions which simplify the set of externalities by providing input or state validation where previously we had unchecked expecatations. This expands the model since now the expectations are explicitly modeled and checked. We can also reduce the complexity of the model by choosing to make assumptions.

4.4 Naming

Credit: Zachary Tellman, Elements of Clojure (unreleased).

Linguistic abstraction in the form of naming is the most fundamental tool available to us as both artisans and engineers.

Names being the tools we use to understand our tools, choosing good names (whatever that means) is of primary importance. But what characterizes a "good" name?

Frege suggests that a name consists of three properties - the sign, the textual representation of the name, the referent being the entity referred to by the sign, and finally the sense, being the properties ascribed to the referent.

The traditional example is that Phosphorous (morning star) and Hesperus (evening star) are both Greek celestial bodies which we now understand to both name the planet Venus. The sign and referent are according to Frege insufficient to understand these two terms which are prima facie synonyms because they don't actually interchange.

A good name must satisfy two properties - it must be narrow and consistent.

A narrow name excludes things it cannot represent; its sense is only as broad as the context of its use requires. For instance if the only expectation to be made of a reference is that it will be a mapping, then map or its contraction m is a perfectly acceptable name if that is the only sense in which the referent is used.

A consistent name shares the same sense for a referent as is used for that referent in other related contexts. To continue the example of a mapping, to subsequently sign the same referent as dict elsewhere would be less than ideal both because it's a broader name which implies a specific kind of mapping and thus no longer shares the same sense. A simple mapping m may only imply clojure.lang.ILookup, whereas hashmap implies java.util.HashMap and side-effectful update. Naming and naming conventions are tremendously important in the context of dynamically checked systems wherein names largely depend on their sense to communicate the type or at least intent of the use of the referent.

Lets consider some examples (adapted from Zach's fine book)

;; This is a Very Bad expression.
;;
;; The left referent is very broad, when we clearly use it in a narrow sense. It
;; should be more narrowly named.
;;
;; The value we get out of our broad referent is named very narrowly. The single
;; string contains far too much sense.
(get m "sol-jupiter-callisto")

;; This is a better equivalent expression, because we communicate a more
;; appropriate sense for our previously over-broad sign, and we unpacked the
;; previously overly precise key into a set of more general structures each of
;; which communicates less sense.
(get-in starsystem ["jupiter" "callisto"])

;; Better yet capture the logical operation, the fact that Callisto is a moon of
;; Jupiter is an assumption in all of the above expressions.
(get-body-by-name system "callisto")

;; It would also be better to be completely precise about the sense of the name
;; Callisto by doing something like
(-> (get-system "sol")     ; This communicates that we want to get a "system" named sol
    (get-body "callisto")) ; And we care about the component "body" named callisto

;; This expression further communicates the sense that Callisto is a moon of Jupiter.
(-> (get-system "sol")
    (get-body "jupiter")
    (get-moon "callisto"))

;; Both of these two latter expressions are better because more of the relevant
;; sense of the name callisto is captured explicitly, rather than being implied
;; upon or expected of the seemingly broad mapping referent.

4.5 Achieving Abstraction

As suggested in the discussion of the consistency of signs, part of the problem here is that we're forced by the tools we use to project our sense directly upon the sign.

Types: A heresy	static sense	dynamic sense
static dispatch	Haskell, C, Java	Java (class casting)
dynamic dispatch	C (dyn linking)	Smalltalk, Python, Ruby

Static type systems provide a model with which we can talk about the sense we associate to a name by capturing at least some of the sense in which we intend to use the referent as a type and knowing that if we can determine a type (and the type must conform to a type of which we knew ahead of time) then we can begin to understand the behavior of our systems through the lense of the types we've assigned to our data.

Some systems allow us to use this information statically to check that at least some part of the sense or type is consistently respected in the use of a referent.

Some systems enable you to capture more sense than others, by encoding the sense statically in what we usually call a type.

Type systems can be compared in terms of how much sense they enable you to capture, and how difficult it is to do so.

Lets say that we wanted to count up from zero, to continue the example of positive integers from above. We could choose a sign with which to reference our value which indicates that it's an index such as the traditional i, or idx or even index depending on individual taste. But sometimes we want to play clever tricks with the fact that sometimes negative indexes are defined, or we want to be tricky and reach back to some location "before" in memory where we're currently indexed to and soforth. The choice of sign implies only sense, it cannot constrain the referent.

However, we can choose abstractions which let us do so. For instance, we could build an abstraction which only allows us to capture ℕ, the natural or counting numbers.

from collections import namedtuple
from functools import total_ordering

@total_ordering
class N(namedtuple("N", ["value"])):
  """A numeric value constricted to N (the counting numbers)."""

  def __new__(cls, value):
    if value < 0 or isinstance(value, float):
      raise ValueError("N is constrained to [0, 1, ...)")
    return super(N, cls).__new__(cls, value)

  def __int__(self):
    return self.value

  def __add__(self, other):
    return N(int(self) + int(other))

  def __sub__(self, other):
    return N(int(self) - int(other))

  def __eq__(self, other):
    return int(self) == int(other)

  def __lt__(self, other):
    return int(self) < int(other)

This isn't so different from the full Haskell implementation of Numeric.Positive.

There are techniques which can be used to try and capture more sense. This technique is best known as "smart constructors". Rather restrict the sense to the sign, we've defined a type which can only take on values conforming to our desired sense.

The additional semantic restrictions we can impose on ourselves from the type free us from either having to check the sense ourselves over and over again which is the most verbose and fragile path, or from settling for the integer sense we get "for free" and hoping that it's enough.

This tactic can be applied to any domain where we want to give a type to a subset of some other space of values. For instance strings of a known format. It is just window dressing, but frequently it's enough that we can approximate the semantics we actually want.

To take another example, what if we were to try and make a type for a Start of Authority (SOA) version? We could use a simple string, after all that's what the SOA becomes when we lay it out in a zone file, and the only requirements we have of an SOA is that it increase to define an ordering among zone file versions.

Generally however, strings (even constrianed strings!) should be avoided. Text is the least structured form of data. Parsing is, even with the aid of parser combinators and regular expressions, difficult and error prone. It's far better to use fully realized data representations which can be rendered to strings than to keep data as a string and decode it.

Which carries more semantic meaning - the string "20170706001", or the tuple

from collections import namedtuple
from scanf import scanf  # 3rdparty

class SOA(namedtuple("SOA", ["year", "month", "day", "version"])):
  """A tuple capturing a DNS SOA format."""

  PATTERN = "%04d%02d%02d%03d"

  def __new__(cls, text):
    """Decode a SOA string, returning a SOA tuple"""
    return super(SOA, cls).__new__(cls, *scanf(text, self.PATTERN))

  def __str__(self):
    """Encode this SOA as a string"""
    return self.PATTERN % self  # self is an appropriate 4-tuple already

print SOA("20170706001")
# SOA(year=2017, month=7, day=6, version=1)

The tuple can be rendered to and parsed from text as required, and above all it captures the sense in which we use SOAs which leaving the SOA as a plain string does not. If strings are just passed through and not manipulated, then perhaps it's acceptable to keep them represented as strings and skip decoding but structures like this which capture more of the sense of our data should be preferred.

Unfortunately there's often lots of other sense we may care about - for instance that we have a number within the index bounds of some structure or that there exists a record in the database with an id primary key of this value for any possible value. Checking these senses statically may well be impossible. There could be a concurrent change to the database and a once valid identifier becomes dangling and soforth.

Static types in the Haskell sense of things help, they allow you to capture some of the relevant sense - enough in my experience that software development may be easier because the type system helps you especially when you need to change abstractions but thanks to the completeness theorem we know all too well that they can never check every property we may care about.

4.6 Utility Is Contextual

It's tempting to think that we can, through great skill or shear luck, discover perfect abstractions; Platonic ideal or Martian crystalline forms of a concept which satisfy all our needs for all time. The problem with this view is that different contexts have different needs. Many objects may generate the same shadow on the wall of Plato's cave.

For instance above I attacked the "stock" numeric abstractions which include hardware implementation details, suggesting that arbitrary precision math is a better default. I stand by that argument, but it must be recognized that there are occasions when we must make different tradeoffs in choosing our models.

Sometimes we need to interface with the machine. Sometimes we want to use (or author) bit vectors. Sometimes we don't have a memory manager and can't afford truely arbitrary precision arithmetic. Sometimes the performance degredation of operating on a double floating point number as opposed to a single or half width floating point number is meaningful and we must make a choice to increase complexity and sacrifice ergonomics in the name of utility. That some machine type is the shared interface of some other abstraction and that the externality of the machine's particular wall time performance relate to our context changes what is appropriate.

Just as we should exercise caution in choosing our names so that we don't choose overly narrow or broad names, so we must exercise caution in choosing our abstractions. The simplest possible abstraction with the fewest possible externalities may well not be the most appropriate to the domain, and worse still a simple abstraction is not likely to lend itself to change. Like other crystals, abstractions shatter or shear rather than bend from their natural shape.

Coupling of abstractions - allowing the tools you choose to share externalities and to expect one-another is dangerous because it means that even small deviation from the original requirements and context may prove cause for significant change with rippling consequences for your entire system.

5 Performance

The greatest performance improvement of all is when a system goes from not-working to working.

– Osterhaut "My Favorite Sayings"

Performance and efficiency generally are factors in the measurement of how well a tool solves or responds to the needs of the "user", whether they be a person, business or other entity.

A code breaking machine which can decode a daily message in 32 hours isn't sufficient. It can't decode messages fast enough for them to be relevant. Such a machine is not useless, but it doesn't solve the problem set for it adequately. Three such machines together will provide, at an 32 hour delay each, the previous day's message and perhaps a fourth or even pairs of machines may be supplied for reliability in order to provide a more-or less continuous feed of intelligence but it would still be yesterday's data today and delivered at great cost.

In this example we have an objective criterion of how much performance is "enough". The cracking machine must have a cracking rate of at least 1 msg per 24 hrs, and we have an objective measurement that we're only able able to crack at the rate of 1 msg per 32 hrs.

There are two primary approaches to achieving performance where it is required. The first approach is to simply do less work - either by reducing the problem using algorithmic optimization, or by reducing the size of inputs by firing customers or negotiating system scope. When optimization must occur, algorithmic or organizational optimization should be by far preferred in the interest of overall program simplicity with all its other virtues.

The second approach is to tune what work must be done to better suit the machinery we use to do it. This is what we usually mean when optimization comes up, although in general we should prefer to find ways to simplify and reduce the problem.

Two opinions about programming date from those days. … The one opinion was that a really competent programmer should be puzzle-minded and very fond of clever tricks; the other opinion was that programming was nothing more than optimizing the efficiency of the computational process, in one direction or the other.

– EWD, "The Humble Programmer"

To be clever means to be "sharp", "quick" and "skillful". The old English roots suggest a sense of "cutting" and "keenness". It would be cleverness to design a tool which does the difficult or seemingly impossible within resource constraints by leveraging intimate knowledge of the system.

This formulation of a "clever trick" implies that a really clever trick must also have a high complexity. This is the root of the trouble with optimizations of the second kind. Clever tricks while perhaps relevant in the moment are in the long term likely to prove troublesome.

Worse, clever tricks of this kind require specializing some part or all of your program so that it reflects performance considerations. This has the particular negative consequence of meaning that your code cannot be so simple and algorithmic anymore - it must be reflective of some state(s) or trick(s) used to achieve acceptable performance because performance is now a crosscutting concern which must be accounted for by your abstractions.

Frequently, reasoning from first principles is not enough to produce fully optimized code. Modern high performance sorting algorithms are absolutely fascinating, because while the algorithmic complexity may suggest that provably optimal algorithms such as mergesort or quicksort should be universally adopted it turns out there are frequently ways to cheat and do better. Modern collections libraries use adaptive sorting strategies which benefit from existing sortedness in the input collection and frequently contain multiple different sorting implementations which are applied to different sizes of collection.

Even apparently simple statements like "doing two string concatenations will be slower than doing one" are impossible to evaluate without a real effort at profiling. The JVM among other platforms recognize that string manipulation is "expensive" due to the need to resize and copy buffers. However, both the Oracle JDK and OpenJDK actually include several optimizations which detect code doing trivial string concatenations and silently replace it with equivalent code which uses the StringBuilder class which delays buffer resizing as long as possible.

If you want fast, start with comprehensible. You can't make it fast if you can't change it.

– Paul Phillips, "We're Doing It All Wrong" (2013)

Unless there are known constraints, such as here the hard throughput requirement of a message a day question of "is it fast enough?" and the statements "this isn't performant" or "we need to do this for performance" are totally meaningless. Worse, they detract from the goal which should be to ship a tool which serves a purpose and introduce artificial pressure to conform the choice of abstractions to arbitrary false goals.

6 Fin: Purism

Niklaus Wirth's work is notable superficially because Wirth is himself a prodigious hacker. He's responsible for, among other projects, no fewer than five separate programming languages and two operating systems. What's the trick? How does he do it?

In the implementation of the Modula-2 compiler, Wirth and his students used two metrics to estimate and track its quality over the course of its evolution. One was the simple line count of the program - a poor proxy for complexity but a data point none the less. Another was the time which it took the compiler to compile itself.

One of Wirth's students implemented an elegant hash table backed by binary trees, for use in mapping strings to symbols - to look up local variables and implement lexical scope. Doing so introduced a new, complex datastructure, increased the line count of the compiler and added at best no performance benefit for it turned out most scopes in real programs were small for which a simple association list could be faster. Wirth ripped the hash table out.

Wirth was also a great user of tools, such as Backus-Naur Form and parser generators for expressing precisely the meaning which he wished to ascribe to a given set of symbols and the notation which his languages should represent. Even in these, Wirth's ascetic attachment to simplicity is clear - all of his languages are simple and can be efficiently implemented without any clever tricks whatsoever.

Simplicity and quality aren't properties which code or systems get by osmosis or by grant of the gods. They are the product of deliberate effort by programmers to resist the entropic pressure applied by reality and customer requirements on systems and to in spite of that pressure carve out cohesive, appropriate abstractions which leave room for growth.

It would be equally absurd, then, to expect that in unjust, cowardly, and voluptuous action there should be a mean, an excess, and a deficiency; for at that rate there would be a mean of excess and of deficiency, an excess of excess, and a deficiency of deficiency. But as there is no excess and deficiency of temperance and courage because what is intermediate is in a sense an extreme, so too of the actions we have mentioned there is no mean nor any excess and deficiency, but however they are done they are wrong; for in general there is neither a mean of excess and deficiency, nor excess and deficiency of a mean.

– Aristotle, "Nichomachean Ethics", Book II, Ch. 6

In the "Nichomachean Ethics", Aristotle (or rather his student Plato writing in his master's name) develops a system of ethics on the basis that one must always choose the mean between excess and deficiency.

There is a fashion in the software community to be "pragmatic". That is, to be tempered or practical with respect to an attitude or policy. Unfortunately, if you temper temperance with an added dose of reality, you no longer have "pragmatism", you have a regression. Software as an industry is built on an incrementalism of "progress" which is informed by experience but largely unaware and often dismissive of the possibility space.

Before we can claim to be pragmatic, we must have a vision of the perfection from which we turn aside in practical interests. It is my opinion that the overwhelming majority of working programmers are not conscious of what could be; of how higher order tools could help them; of how the pains they feel are the fault of specific failures of their tools and are not essential.

My goal with this talk was to share with you some of the philosophy and purism which I carry with me and which I hope helps me to strive for more perfect solutions. I hope that it helps you too.

Thanks.

Immutable Env Things

2016-06-13T00:00:00+00:00

As with some of my other posts, this was originally an email which turned into enough of a screed I thought it would be of general interest and worth posting. Some op/ed material has been removed, but the technical details are unaltered if not clarified.

So here’s the rundown I’ve got so far on a lisp with immutable environments. I’m merely gonna sketch at some stuff, because giving a full treatment would require dusting off and finishing a bunch of stuff I came up with for Ox and put down again.

I also threw out the Haskell sketch I was dinking with because it was just distracting from writing this :P

So lets start from the top.

Clojure is a form-at-a-time language which happens to support reading files of forms (or rather Readers which produce textual representations of forms, where they come from is magical).

Binding is achieved through a global, thread shared, transactional, mutable mapping from Symbols to Namespaces, which are themselves transactional mappings from Symbols to Vars, which are themselves a triple (QualifiedSymbol, Metadata, Binding). Vars happen to also support dynamic binding (pushing and popping), but this is less used. From here on out I’ll treat them simply as a global mostly static binding mechanism, which is their primary use case anyway.

Control and local bindings are achieved entirely using lambda/fn forms compiled to JVM methods, produced by the opaque compiler as opaque IFn/AFn objects. Top level forms are compiled and executed for effect by being wrapped in an anonymous (fn [] <form>) which can be blindly invoked by the compiler to realize whatever effects the form may have.

Circular dependencies are achieved via Var indirection. A Var is first created with no binding, so that it can be referenced. With that environment binding, code which will depend on (call) the Var’s final bound value can be compiled since it just needs the target Var to exist in order to compile. The depended on Var may then be redefined, having both itself and its dependent visible in the compilation context and so the two Var bindings can be mutually recursive.

While traditional and lispy, this approach has a number of problems which I’m sure I don’t need to reiterate here. The goals of an immutable namespace system then should be to

Make Namespaces the compilation unit rather than Forms
Make Namespaces reloadable (change imports/exports/aliases w/o system reboot)
Enable Namespace compilation to fail sanely
Enable refactoring/analysis

The sketch of this I’ve been playing with is as follows:

Lets assume an interpreter. If you can interpret you can compile as an implementation detail of eval and interpretation is easier to implement initially. Assume a reader. So we read a form, then we have to evaluate it. Evaluation is first macroexpansion, then sequential evaluation of each resulting form in the environment. Evaluation at the top level is eval Env Form -> Env (we discard the result) which is fine. Because we aren’t really doing compilation, only interpretation, Clojure’s tactic of having Vars and analyzing against Var bindings works 1:1. Create an unbound symbol, analyze/bind against it, then add a binding later.

The (global) environment structure I had in mind is pretty simple: one unqualified symbol names the current namespace (*ns*), a mapping exists from unqualified symbols to Namespaces.

Namespaces are essentially compilation contexts, and really just serve to immutably store the alias mapping, import mapping, the mapping of the Vars in the namespace to bindings, and a list of the public Vars/exports from the namespace.

Vars are a fully qualified name, metadata and a value. That’s it.

Compilation occurs at the namespace level. All the forms in a namespace are sequentially read and evaluated in interpretation mode. All macros expand normally and type hints are processed they just don’t do anything.

Once the whole namespace has been successfully loaded in interpretation mode, a second pass may be made in which static linking is performed. Because everything in the namespace has been fully realized already it’s possible to statically link even mutually recursive calls within the same namespace. Full type inference within the namespace is also possible here, etc.

Aside: Interestingly because it is single pass, the existing Clojure compiler doesn’t (can’t) handle hinted mutually recursive function calls at all. It relies on Var metadata to store arglists (and hints for primitive invocations), so in order to emit a hinted recursive call the forward declaration of the var has to carry the same ^:arglists metadata (hints and all!) which will be added by defn whenever it is finally defined.

Once a namespace has been fully loaded and compiled (including the second pass) the global environment with the resulting namespace and var mappings is the result of whatever caused loading. If an error occurs during compilation, we just abort and return the environment state before anything else happened. I think there are tricks to be played with generated garbage class names and classloader isolation here so that this works even during the 2nd static compile pass, but it should be possible to fully clean up a failed compile.

So this works really great for module loading, and even for macros which expand into multiple def forms. This model of eval :: env -> form -> (env, result) starts to break down when you want to talk about macros that do evaluation during code loading, and likewise the REPL. Basically your interpreter winds up holding an additional reference, *env* or something, which is intrinsically mutable, but which references an immutable environment and holds shall we say the state continuation of the entire REPL. Consider user code which actually calls eval during execution. Whatever state changes that evaluation creates need to be preserved in the “global” continuation.

In this model when you compile a form at the REPL, the runtime could obviously interpret (this may be appropriate for macroexpansion) and can also compile. This is only tricky when the user recompiles a symbol which was already bound/compiled. In this case, the runtime could either eagerly recompile all the code which links against this function using the same interpret then statically link tactic or could just invalidate any compiled bytecodes and lazily recompile later. The former is probably more predictable.

Once a namespace has been reloaded/altered, all namespaces importing it must also be recompiled in topsort order using the same tactics. That we already have per-symbol dependency information and per-module dependency information helps with building refactoring tools which otherwise have to derive all this themselves. Ideally top level expressions/definitions would also expose local variable tables tables and local variable dataflow graphs so that analysis there is also possible.

Aside: It may be possible to allow cyclic dependencies between namespaces (see submodules in racket for some study on this problem). In the common case it may well be that macros are well behaved and that cyclic dependencies between modules work out just fine. However because macros perform arbitrary computation it’s also pretty easy to cook up pathological cases where macro generated code never reaches a fixed point in repeated interpretive analysis which can be statically compiled. For this reason I’m inclined towards formalizing a ns or module special form and throwing out require, refer, import and friends as legal top level forms altogether. Namespaces should be declarative as in Java/Haskell.

There’s probably a way which I just haven’t seen yet to treat updates to the “global” namespace mapping and updates within a namespace the same way via the same mechanism since they’re both dependent on the same dependency tracking and recompile machinery.

Writing this has made me think about having an ImmutableClassLoader which is derived entirely from a immutable environment and loads classes by lazily (statically) compiling forms.

That’s kinda all I got. ztellman gets credit for the mutable *env* in the repl bit which I spent literally weeks trying to do without last year. Maybe something with monad transformers can get you there but I couldn’t figure it out. mikera has some sketches of all this with types in his KISS project.

I’ve already prototyped an environment system that looks a lot like this in the Ox repo if you want to go dig. Originally there was this ox/lang/environment.clj then I started dinking with a Java implementation of the environment structure ox/lang/environment which also has a couple different stack based contextual binding types for the interpreter.

As written about in the Jaunt essays, I’ve kinda concluded that whatever this immutable ns language winds up looking like is so divorced from what Clojure or at least the way that most people write Clojure that you’ll loose so much value in the changeover it won’t pay off for a very long time. The mount library is a perfect example of this in that it’s deeply imperative and takes advantage of code loading order to achieve start and stop. It’s not the only bit of code I’ve seen which does effects at load time either. Hence Jaunt which tries to be less flawed around namespace reloading/refactoring without going whole hog on ns immutability.

The more I kick this stuff around the more I find that the whole endeavor verges on a Haskell clone in sexprs and the more I decide I’d rather take the time to learn Haskell properly than mess with an immutable lisp or Jaunt. Besides then you get nice stuff like typeclasses and fully typed instance dispatch and equality that actually works and theorems and native bytecode and yeah.

Deprecation warnings in Jaunt

2016-03-08T06:56:12+00:00

This week I’m working on getting Jaunt’s 1.9.0 release nearer the door, on which note I’m happy to properly demo one the features of this first release: deprecation warnings.

A bare REPL will do for these demos.

$ cd $(mktemp -d)
$ wget https://clojars.org/repo/org/jaunt-lang/jaunt/1.9.0-RC4/jaunt-1.9.0-RC4.jar
$ java -jar jaunt-1.9-RC4.jar

Jaunt #2 (not my best ticket ever in terms of hygiene) was the first work I did on what became Jaunt and was really pretty critical for getting my teeth into the project and validating that there were tractable incremental improvements to be made over Clojure.

This change introduced four new switches to the Jaunt compiler, the most interesting of which is :warn-on-deprecated which enables or disables compiler warnings in support the use of ^:deprecated metadata on functions and namespaces. :warn-on-deprecated is on (true) by default, although it can be disabled globally with the JVM system property clojure.compiler.warn-on-deprecated=false, or by set!ing clojure.core/*compiler-options* say to have (set! *compiler-options* (dissoc *compiler-options* :warn-on-deprecated)) in a namespace where you want these checks to be disabled.

The semantics of deprecation are simple:

A namespace is deprecated if and only if it has the ^:deprecated metadata.
A definition (Var) is deprecated if either it occurs within a ^:deprecated namespace, or is itself marked ^:deprecated.

Vars and namespaces may become deprecated in any SemVer minor version or major version, but may only legally be deleted only on a major version.

Deprecation warnings are only emitted when a deprecated namespace or definition is accessed from a context which is not deprecated. Deprecated contexts are namespaces which are deprecated, and the bodies of deprecated definitions.

So for instance, in the following code:

(defn ^:deprecated bad-idea [x]
  (println "Oh noez!")
  (throw (new Exception "Bad code is bad")))
;; => #'user/bad-idea

(defn ^:deprecated also-bad-idea [x]
  (bad-idea x))
;; => #'user/also-bad-idea

(defn almost-better-idea [x]
  (bad-idea x))
;; Warning: using deprecated var: #'user/bad-idea ...
;; => #'user/also-bad-idea

The definition of bad-idea itself is innocuous. Simply evaluating deprecated definitions is fine. Warning that also-bad-idea uses bad-idea would be superfluous, because that fact is innocuous so long as neither is used. Suppressing warnings in this case allows Jaunt to load arbitrarily much deprecated code without drowning a user with false positive warnings.

Users should only have to care about deprecation at the edge between current code and deprecated code. Thus compiling almost-better-idea will emit a warning that it makes use of bad-idea, since almost-better-idea is not itself deprecated. Likewise at the repl invoking either deprecated function would also generate a warning.

As to namespaces, similar principles apply. Consider the two namespaces and trace:

(ns ^:deprecated com.proj.code)
;; => nil

(def some-const 3)
;; => #'com.proj.code/some-const

;;------------------------------------------------------------------------
(ns com.proj.new-code
  (:require [com.proj.code :as old :refer [some-const]]))
;; Warning: aliasing deprecated ns: com.proj.code  ...
;; Warning: referring deprecated var: #'com.proj.code/some-const ...
;; => nil

(def g 4)
;; => #'com.proj.new-code/g

(def h (+ old/some-const g))
;; Warning: using deprecated var: #'com.proj.code/some-const ...
;; => #'com.proj.new-code/h

As the whole com.proj.code namespace is deprecated, merely referencing it and requiring that it be loaded is cause for a warning. Should that namespace ever be removed, the require would break regardless of whether anything from the required namespace is used or not.

If symbols which are deprecated are referred into a namespace which is not deprecated, each one of those will generate a warning as well for the same reason. However Jaunt tries to be smart about this, suppressing such warnings if a (require ' [... :refer :all]) or a (use ...) directive has been issued which constitutes an indefinite referral.

The symbol com.proj.code/some-const, is deprecated by dint of being defined in a deprecated namespace. Consequently a warning is emitted when it is used outside of a deprecated context, here in the definition of com.proj.new-code/h.

One last demo, disabling warnings!

(ns com.proj.no-warnings)
;; => nil

(set! *compiler-options* 
  (dissoc *compiler-options* :warn-on-deprecated))
;; elided...

(def ^:deprecated a 3)
;; => #'com.proj.no-warnings/a

(def b (+ a 4))
;; => #'com.proj.no-warnings/b

There’s more than just this coming down the line in Jaunt, so if you’re interested check out the 1.9 CHANGELOG, watch the repo or stay tuned here for more demos. Issues, ideas and especially pull requests are welcome :D

Jaunt - A friendly Clojure fork

2016-02-22T13:00:00+00:00

TL;DR

I’ve started a fork of Clojure which I intend to operate in a community directed manner; Jaunt. Check out the demo script or just keep reading for more.

The Long Version

After several months of dinking with Oxlang on and off I came to a realization. Ox, while an annoyingly persistent daydream was doomed to be one of two things. Either it would be something akin to Haskell in S expressions, or it would be indistinguishable from Clojure with some tweaks under the hood.

I’d gotten Ox’s reader sort of working, and was starting to experiment with notations, explore notions of higher kinded types and generally was having a ball. However even getting that far was a highly educational exercise in how much Clojure already does fairly well. To continue down the road of Ox was to admit that I’d have to reinvent all of Clojure’s datastructures, most of the Clojure compiler and a whole boatload of the standard library. It was also to admit that due to limitations of the JVM as a target, I wouldn’t be able to play some of the many tricks which the GHC folks get to play even were I to go down the road of a pure language.

Why do all this? Because it’s hard? I’m no Heracles seeking labors or yaks for their own sake. I’d rather be learning something new or playing Dota or… a thousand other things.

YOUR FINEST YAK SIR
— tail -f /dev/ardm0 (@arrdem) February 13, 2016

I swear I would.

Besides, even though Michiel Trimpe was kind enough to mention Ox alongside a number of other “conversational” programming languages in his ClojureX talk, I’m not convinced that it’s a good idea.

In looking back at that particular talk, one thing struck me. Sure in a pure functional programming language such as Haskell you could stick an arbitrary AST in a Merkle tree and have a globally distributed version control and distribution system with strong guarantees about tree uniqueness and reuse safety.

Unfortunately, Clojure is no such beast. We have no language level concept of effects (okay we have the io! macro but that’s hardly the same). While you could implement immutable namespaces (and in fact they are backed by immutable maps already) the concept doesn’t make a whole lot of sense to me.

The very notion of a conversational language is that you can reference any definition from any version of the program as if it were the current version. What does this look like at a REPL? Are namespaces (or rather full states) identified with a commit ID and compilation occurs in terms of a fully qualified “commit” and Var? Now we need notation for that. Maybe version/name/namespace or something. Some sort of explorer to browse older commits is in order… the list goes on and on. In short, I think you wind up reinventing a bunch of stuff that git already does pretty well.

This is not to say that Clojure’s existing mutable namespaces are ideal. For me, they work against the ns or file level code loading style I use when developing Clojure code. Old habits from C die hard as it were. A common failure mode I encounter is that I’ll rename a function but not all of its uses. IntelliJ and refactor-nrepl can be of some help here, but I’ve had them break down and I’m not always in a “full ide” configuration. Because namespaces are mutable, even if I were to perform a rename correctly, the old name is still present in the namespace. If I got it wrong, my code reloads, all symbols resolved then I get Really Cool Unexpected Behavior when parts of my code call the new function and parts call the old function. Or worse my code works fine until I restart my repl or the test server and everything explodes. Or the old name just sticks around cluttering up my autocompletion.

This becomes a real UX problem for me, because I rely heavily on refactoring. I start by sketching out what amount to types, or deriving operations from types I already have, and keep working renaming and moving code around until I get something I’m happy with.

@arrdem your problem is all the refactoring. Why isn't the code right from the start?
— Chas Emerick (@cemerick) December 23, 2015

And Yet I Shave

The thing which came to me now a few weeks ago is that there’s an interesting halfway house between these two extremes of mutable and immutable namespaces. What if we gave Namespaces and Vars version numbers? When a Namespace is re-defined (the ns form is evaluated) then the Namespace’s version is incremented. When a def is evaluated, the version of the Var bound by the def is set to be the version of the Namespace in which that Var exists.

This allows traditional REPL style interaction with Namespaces. You may enter a Namespace, add bindings, aliases, imports and so forth as you will. However the version numbers give you important static information about the state of the Var in its Namespace.

If I have some def which I entered at the REPL, when I do something to ns backing file and reload it, that def persists. But, its version is now out of sync with the Namespace it exists within. When we reloaded the file, the Namespace’s version was incremented and then every definition was reevaluated. If my code is still using an old symbol which is not textually present in the file and was was not reloaded, its version doesn’t increase. Now the compiler can detect the version disparity and emit warnings that I’m using old code.

While this solves dependency freshness issues within a single def and a single Namespace, solving them globally requires more leg work. The compiler needs to be adapted to track the use and reach sets of each compiled expression, and to inter that information into the Var binding forms so that it’s possible for a newly compiled form to emit warnings about transitive use of stale vars.

These changes don’t alter the semantics of the Namespace at all outside of reloading. If you evaluate a new def or a single form in a Namespace it works just as before. However if you take this whole file reloading approach, the language can offer added support which papers over a rather painful pitfall.

A related change is that Namespaces can be made to purge their imports and aliases on reload. Right now, Namespaces persist imports and aliases until the VM is restarted. This leads to a nasty set of refactoring problems where you want to change the name of an alias, but can’t reuse a name because aliases are never cleared. If we consider a whole Namespace and its source file(s) as the unit of compilation we can again do better. This is another language level change which would meaningfully improve my day to day use of the language without breaking existing use patterns.

Both of these changes turn out to be quite simple to implement. In the course of only a few days I went from the concept of these changes to a build of Clojure (pr, build) which features Namespace clearing and Var versions. I’m using now daily because it better supports the way that I want to work with the language.

Waxing Political

Rich, Stuart Sierra, Stu Halloway, Alex and the rest of the Clojure/Core crew (see Clojure/Huh?) built Clojure and have shepherded it this far. I’m immensely grateful to them for the work they’ve done to date in building a tool which I’ve had great fun using. But I think our priorities diverge.

From what I’ve been able to gather by talking to people who’ve been around longer and what Rich has written, it should be clear that Clojure is not a community project in the same way Python or Rust is. Clojure is Rich’s personal project, which he is so kind as to share with the rest of us and in in its refinement Rich chooses to accept some amount of input from us as users. I’m somewhat ashamed to admit that it took me two years to realize this.

Open-source development problem in one sentence:
"Users assume they are customers"
— Aleksey Shipilëv (@shipilev) January 20, 2016

Discussion of particulars, and the reasons for the status quo is I think a somewhat useless exercise. Rich has his project and administers it as he sees fit. As Clojure is Rich’s project, the priority of its governance is to conserve Rich’s time and it has repeatedly been made clear that there is no desire to change this. This is an eminently reasonable goal I can find no fault in. Rich doesn’t owe me or anyone else jack, let alone time. However the consequence of this structure and of not delegating is the slow and steady path which frustrates my desires for more rapid iteration.

A Fork in the Road

With loyalty the path of the last 18 months and clearly one of frustration, and voice ineffective due to the (reasonable!) priorities of Clojure/Core, The only course of action left to me is some form of exit.

I think that there’s a lot of possible improvements that can still be made to the language. Currently there doesn’t seem to be a lot of appetite for exploring that in Clojure itself. The namespace reloading stuff sketched above is just the first thing that popped to mind when I started thinking about how to improve my day to day experience and implementing that alone has shaken loose a number of other small but worthy improvements.

The consequence of this opinion is that I’ve started Jaunt.

Jaunt is my personal fork of Clojure which seeks to deal with the various organizational and technical limitations expressed above.

Technically, with Jaunt I want to explore a space of refinements to the language such as the Namespace reloading changes sketched above which add value without breaking compatibility with the ecosystem of libraries and tools like CIDER which I’ve come to rely on every day.

Organizationally, I want Jaunt to be a community driven affair. I don’t claim that my judgment is infallible, nor do I think I’ll have all the good ideas, nor will I have infinite time to consider changes even were these restrictions lifted. I’d love to involve contributors who want to bring improvements to the project within the stated bounds of preserving compatibility at least with the documented parts of Clojure.

Demo

Because otherwise how do you believe that any of this works, here’s a script that’ll build a empty leiningen project in a temporary directory and run a quick demo of some of the stale Var stuff.

No Promises

If you want a stable language. If you’re running code in production. If you don’t want to get down and dirty with the language implementation or the development process, stick with Clojure.

As the EPL states:

THE PROGRAM IS PROVIDED ON AN “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED INCLUDING, WITHOUT LIMITATION, ANY WARRANTIES OR CONDITIONS OF … FITNESS FOR A PARTICULAR PURPOSE.

Jaunt should be a drop in replacement for Clojure. I want to keep it compatible. I foresee no reason to break the documented Clojure API. But some amount of drift is inevitable. I’ve already moved some stuff around inside the Java implementation. If you were depending on those undocumented yet public implementation details, all bets are off. I’ve changed how metadata behaves on vars (I think it’s a better approach but it does impose behavioral differences). I’ve added deprecated warnings. I added clojure.core/*line* and clojure.core/*column* (pr) which the compiler totally had internally and just didn’t expose forcing weirdness. And this list of differences (perhaps improvements) is just going to grow.

Whether Jaunt proves to be something I depend on in the future, whether I wander off into the land of types and leave it to someone else or whether it diverges too much from Clojure and dies as a result of the weight of its mutations, I’ve overheard enough interest in a project such as this that I think this is a worthwhile endeavor and I hope you’ll join me in this little adventure.

Update: Clojarr has been renamed to Jaunt, updated article accordingly.

Hacking like it's 2288

2015-11-17T08:22:39+00:00

I’m an a diehard Fallout franchise fan. While school currently precludes me from playing, I took some time to automate solving the hacking minigame which is used to unlock computer terminals and secured areas in-game.

When “hacking”, a player is presented with a list of words one of which is the “password” and has five (or fewer) tries to try and find the password. Every time the user selects a candidate, a response of how many characters match between the answer and the candidate is given so that the user can refine their guess.

The strategy for playing this game is uninteresting at best, you choose an initial word either heuristically or randomly, and then based on the number of characters which that word shares with the answer you can refine the whole list of words by removing all words which do not share exactly that many characters with the word you tried. This simple guess and refine algorithm one need only iterate until only one answer remains.

While an obvious algorithm, it’s a little tedious to do this by hand because there’s no good way to keep track of all the candidates which have been removed from consideration. Also computing string overlaps is boring. Given an algorithmic problem, write an algorithmic solition! What do we need to solve this problem in Clojure?

Well first we need our word “intersection” scoring function. To do this, we’ll take the frequencies of the left word, the frequencies of the right word. Since each character can only correspond to another single character, this means that given N as on the laft and M bs on the right, we have (min N M) common characters. We can simply compute this common character count for every character in the left string (characters occuring only in the right string won’t factor in anyway) and take the sum over those counts.

(defn letters-in-common [left right]
  (let [fl (frequencies left)
        fr (frequencies right)]
    (->> (for [[k vl] fl
               :let   [vr (get fr k 0)]]
           (min vl vr))
         (apply +))))

(letters-in-common "foo" "foooo")
;; => 3
(letters-in-common "cat" "dog")
;; => 0
(letters-in-common "cat" "rat")
;; => 2

So now we need a guess making function which we can write in terms of our scoring function. What word do we want to choose as our guess? Well we want to guess the word which will give us the most information about all the other words, that is to say is most similar to as many other words as possible.

Once we’ve made a guess we can use the word similarity score we get back to refine our dictionary and contrain our search space.

This must be the optimal choice by the usual greedy choice argument, since any other word we could guess would tell us less about the other words, and in the case of a tie (two words with equal similarity to all other words) we can’t know without guessing which one is the better choice so we can choose randomly.

(defn make-guess
  "Given a population of words, locates and returns a word with maximum
  \"matching potential\", that is the word which shares the most
  characters with every other word.

  If there is a tie, since the two words have equal potentials it
  doesn't matter which one we choose and so the choice is arbitrary.

  Makes use of scoped memoization for performance on lots of words,
  but realistically this should never be a factor."
  [words]
  (let [-score-fn (memoize
                   (fn [col]
                     (let [[l r] (seq col)]
                       (letters-in-common l r))))
        score-fn  (fn [word]
                    (let [s (sorted-set word)]
                      (->> (for [w words
                                 :when (not= w word)]
                             (-score-fn (conj s w)))
                           (apply +))))]
    (last (sort-by score-fn words))))

Now we need our dictionary pruning function. If a word does not have exactly n letters in common with the last guess where n is the reported similarity of the last guess with the answer, that word cannot be a solution.

We know that for multiple guesses, no word could be an answer which does not have the reported similarity with all previous guesses. Thus we don’t accidentially exclude any possible answers by simply filtering the dictionary as such after each guess.

(defn trim-pop [words guess commonality]
  (for [w     words
        :when (= commonality (letters-in-common w guess))]
    w))

(trim-pop ["foo" "bar" "cat" "rat"] "cat" 2)
;; => ("rat")

So lets put these two to use! We’ll wire up make-guess to letters-in-common which is our scoring function anyway and just let it run on a small dictionary and a goal word. Just to make the point that this algorithm does converge.

;; An example hacking loop. The guess and refine algorithm with a
;; sample input.
(let [words ["LOWER" "CREED" "JAMES" "CAGES" "CARES" "OFFER" "CAVES" "TIRED"]
      answer "LOWER"]
  (loop [words words]
    (when (= (count words) 1)
      (first words)
      (let [guess  (make-guess words)
            score  (letters-in-common guess answer)
            words' (trim-pop words guess score)]
        (println "Guessed" guess "reduced population to" words')
        (recur words')))))

Yay! So that totally works.

And because I don’t think anyone wants to manually run all that state through a REPL while trying to play Fallout 4, here’s a quick REPL wrapper to help with hacking.

(defn autohack [words]
  (if-not (= 1 (count words))
    (let [guess (make-guess words)
          _     (println "Try>" guess)
          score (read)]
      (recur (trim-pop words guess score)))
    (println "It's" (first words))))

Happy wandering!

Strictly Tagged Clojure

2015-08-03T19:00:00+00:00

This summer I’ve been an intern at Factual, and this is an experience report from the semiannual internal hackathon where Alan ‘amalloy’ Malloy and I experimented with using Alexander Yakushev’s Skummet fork of Clojure to emit lean(er) bytecode.

Some motivation

One of Clojure’s primary use cases is as a more palatable tool with which to interact with the rich Java ecosystem and existing Java libraries. Because of its facilities for such inter-operation, Clojure is even sometimes used to write performance sensitive code which would otherwise be written in Java. However there are limitations to the success with which this may be done.

While JVM bytecode is statically typed, Clojure is an aggressively dynamically checked language which makes pervasive use of the Object type to delay typechecking. To this end, Clojure will use JVM Object reflection to resolve instance fields and methods when performing interoperation. While correct for unknown types, because reflective access is slow compared to direct access for known types it has long been possible to write type hints which advise Clojure about the runtime JVM type of a value and enable Clojure to use direct access and direct method invocation rather than reflective access.

However these hints are not types in the sense of a static type being a contract on the domain of values, they are merely hints for reflection elimination and place no contract on the domain of a hinted value.

This hinting behavior for reflection elimination comes at the cost of emitting checkcast instructions. As the JVM is statically typed, one cannot simply swear that a value is of a type, a checking cast must be used. Clojure, when emitting non-reflective method calls and field accesses, does not statically know (and makes no attempt to prove) that the value or expression in play is in fact of the type which you may have tagged it. All local variables and function parameters which are not JVM primitives are stored as Objects and so must be checkcasted to the desired type on every use.

So are we stuck trading slow reflective access for checkcast instructions (which are faster to be sure but cause method bloat when doing lots of interop on previously checked values)?

Of course not! While Clojure does not currently have support for real contractual types when using tags, we can sure add such behavior!

Now. Before you burn me at the stake for being a heretic, clearly since Clojure does not currently have strict local types, we can’t just make tags strict. TEMJVM actually makes that mistake, and as a result cannot compile clojure/core because among others, clojure.core/ns-publics makes use of a type hint which while safe for nonstrict type tags is not correct in the context of strict tags. This has to be an additive, opt-in change.

So, what Alan and I did was create new special fn metadata flag ^:strict. If a fn body being compiled has the ^:strict tag, then and only then are are type tags treated as a contract rather than being advisory. This is a strictly additive change because stock Clojure will ignore the metadata and emit less efficient but still correct code.

So as an example, let’s consider the following fn:

1
2
3
4
5
6
(defn sum ^long [^Iterable xs]
  (let [iter (.iterator xs)]
    (loop [tot (long 0)]
      (if (.hasNext iter)
        (recur (+ tot (long (.next iter))))
        tot))))

Eliding a bunch of implementation details for brevity, this fn compiles on stock Clojure 1.7 to the following JVM bytecodes:

public final long invokePrim(java.lang.Object xs);
aload_1 [xs]
aconst_null
astore_1 [xs]
checkcast java.lang.Iterable [47]
invokeinterface java.lang.Iterable.iterator() : java.util.Iterator [51] [nargs: 1]
astore_2 [iter]
lconst_0
nop
lstore_3 [tot]
aload_2 [iter]
checkcast java.util.Iterator [53]
invokeinterface java.util.Iterator.hasNext() : boolean [57] [nargs: 1]
ifeq 51
lload_3 [tot]
aload_2 [iter]
checkcast java.util.Iterator [53]
invokeinterface java.util.Iterator.next() : java.lang.Object [61] [nargs: 1]
invokestatic clojure.lang.RT.longCast(java.lang.Object) : long [64]
invokestatic clojure.lang.Numbers.add(long, long) : long [70]
lstore_3 [tot]
goto 15
goto 52
pop
lload_3
lreturn
    Local variable table:
      [pc: 15, pc: 52] local: tot index: 3 type: long
      [pc: 12, pc: 52] local: iter index: 2 type: java.lang.Object
      [pc: 0, pc: 52] local: this index: 0 type: java.lang.Object
      [pc: 0, pc: 52] local: xs index: 1 type: java.lang.Object

// Method descriptor #77 (Ljava/lang/Object;)Ljava/lang/Object;
// Stack: 5, Locals: 2
public java.lang.Object invoke(java.lang.Object arg0);
aload_0 [this]
aload_1 [arg0]
invokeinterface clojure.lang.IFn$OL.invokePrim(java.lang.Object) : long [79] [nargs: 2]
new java.lang.Long [30]
dup_x2
dup_x2
pop
invokespecial java.lang.Long(long) [82]
areturn

So here we have two methods. The first one, the invokePrim, takes an Object and returns a primitive long since we long hinted our function. The invoke method is a wrapper around the invokePrim method which provides for “boxing” (wrapping in an object) the primitive result of calling invokePrim. This allows our fn to be used by code which wants and can use a long, and code which doesn’t know/care and just wants an Object back like a normal fn would return.

So lets dig into the invokePrim method.

Load xs off the arguments stack. It’s just typed Object because that’s the parameter type.
Load the constant nil.
Store the nil to the local named xs, thus clearing it. Note that in the locals table, xs has the type Object. This means that when we get xs from the local, we have to checkcast it again because we’ve lost type information by storing and loading it.
checkcast the xs we loaded to Iterable since we don’t really know what it is.
invokeinterface of the .iterator method to get an Iterator from our now guaranteed Iterable.
Store our Iterable into the iter local (discarding type information as above)
Load the constant 0 from the class constant pool.
Store the tot local (primitive typed)
Load our iter
checkcast because in storing it we forgot that it’s an Iterator
invokeinterface to see if there are elements left, producing a primitive boolean
Branch on the boolean going to 21 (in this list) if false
Load tot from the local
Load iter from the local
checkcast that iter is still Iterator
invokeinterface to get the next value from the iterator, producing an Object
invokestatic to call the static clojure.lang.RT method for converting Objects to primitive longs.
invokestatic to add the two primitive longs on the stack
Store the new value of tot
Loop back to 10.
Clear the stack
Load tot
return

So with the exception of the first checkcast to make sure that the Object we got as an argument that should be Iterable is in fact an instance of Iterable, the checkcasts after load are all provably uncalled for. The static types of these values is known because their Java signatures are known, and the only reason that we have to emit all these checks is that the Compiler throws that information away by storing these values in untyped (Object) locals.

The Hack

Every Expr in clojure.lang.Compiler already knows (or can state) its type either as tagged or inferred, and whether it has such a tag. However, these stated Java classes are lies! A function invocation (IFn.invoke call site) is statically typed to return Object (unless it’s a primitive call site but we know that as well) no matter what the tag on the IFn being invoked may say. For example clojure.core/str is tagged ^String and does indeed return a String, however after invoking the appropriate IFn the JVM doesn’t know that there’s a String on the stack because the IFn interface discards that type information. It just knows it has an Object. The fix is that we add a Expr.needsCast method and implement it for every instance of Expr in Compiler.java. So now when in strict mode, we know that unless Expr.needsCast returns true, the value on the stack after Expr.emit absolutely is of type Expr.getJavaClass. Otherwise we cannot avoid the checkcast.

We also have to change the behavior of locals so that we can emit locals with types other than Object. By typing locals we preserve their type information as tagged or inferred across loads and stores. This allows the Expr representing a local use to report that it only needs a cast when the usage of the local doesn’t have the same tag as the type of the binding and we cannot statically show no cast is required.

With these changes, our modified Compiler.java can indeed produce and use strictly typed locals. So lets add our annotation…

1
2
3
4
5
6
(defn ^:strict sum ^long [^Iterable xs]
  (let [iter (.iterator xs)]
    (loop [tot (long 0)]
      (if (.hasNext iter)
        (recur (+ tot (long (.next iter))))
        tot))))

And generate bytecode on our modified version of Skummet 1.7-RC1-r4 (again abbreviated).

public final long invokePrim(java.lang.Object);
  Code:
aload_1
aconst_null
astore_1
checkcast     #30                 // class java/lang/Iterable
invokeinterface #34,  1           // InterfaceMethod java/lang/Iterable.iterator:()Ljava/util/Iterator;
astore_2
lconst_0
nop
lstore_3
aload_2
invokeinterface #40,  1           // InterfaceMethod java/util/Iterator.hasNext:()Z
ifeq          43
lload_3
aload_2
invokeinterface #44,  1           // InterfaceMethod java/util/Iterator.next:()Ljava/lang/Object;
invokestatic  #49                 // Method clojure/lang/RT.longCast:(Ljava/lang/Object;)J
ladd
lstore_3
goto          15
goto          44
pop
lload_3
lreturn
  LocalVariableTable:
    Start  Length  Slot  Name   Signature
       15      29     3   tot   J
       12      32     2  iter   Ljava/util/Iterator;
        0      44     0  this   Ljava/lang/Object;
        0      44     1    xs   Ljava/lang/Object;

public java.lang.Object invoke(java.lang.Object);
  Code:
aload_0
aload_1
invokeinterface #59,  2           // InterfaceMethod clojure/lang/IFn$OL.invokePrim:(Ljava/lang/Object;)J
new           #13                 // class java/lang/Long
dup_x2
dup_x2
pop
invokespecial #62                 // Method java/lang/Long."<init>":(J)V
areturn

The win compared to the original bytecode should be obvious. Sure enough in the invokeStatic method we only emit the one checkcast we absolutely have to have because the xs argument could really be anything. The tot and iter locals are both statically typed, and so we can just load them and invoke the appropriate interfaces directly.

In some simple benchmarks, this optimization on this fn translates to a 5%-10% performance improvement which isn’t too impressive. However other fns like clojure.core/str in our testing were able to get up to 20% performance improvements from strict locals.

Disclaimer

This is the product of a two day hack. While Alan and I have been able to get it to work and emit working code, honestly this isn’t something we’re comfortable taking to production yet. Some clear wins such as being able to emit typed fn arguments by popping arguments, checking them and then putting them in typed local bindings for use and being able to take advantage of types on closed over locals remain on the table.

What Didn’t (seem) To Work

While Alan debugged Compiler.java and picked off some types related wins we hadn’t gotten yet I worked on adding more inlining behavior to clojure.core. Much of core, especially the lexically early fns are just thin wrappers around interop on clojure.lang.RT, which does have reasonable interface types on most of its methods.

The hope was that what with the typed locals work, preserving more type information across calls to the Clojure standard library and inlining the Clojure standard library where possible to interop calls with clearly inferable types we would be able to produce demonstrably faster code.

While in theory this should be at least break even and probably a win, we haven’t managed to benchmark it well and show a clear win from the aggressive inlining work. In fact, the really interesting case of possibly aggressive inlining being an into which is able to use a typed, static Transient loop is impossible because into is implemented in terms of reduce, which takes the reducing fn as a value and then dispatches via clojure.lang.IReduce in order to get fast iteration over chunked seq. However we can’t statically inline a call site through being taken as a value so that’s the end of the line for that idea.

Inline Unrolling

We were however able to fully inline some interesting cases of clojure.core/assoc and clojure.core/conj. A common pattern in Clojure is to write a function f which has a zero arguments case returning a constant, a one argument case returning the argument unmodified and a two or more arguments case in which the operation f is reduced over the arguments provided. Rather than directly implement IFn, functions emitted by Clojure instead extend clojure.lang.AFn (source), which provides some out of the box support for functions of variable arity and function application.

Next Steps

These changes were motivated by internal performance requirements and will likely get polished until they are ready to be upstreamed back to Skummet. While we expect that the patch we have developed will never be included into Clojure as-is if only due to high impact, we hope to see this behavior or something like it enter the mainline in a future version of Clojure.

Edit 1: Skummet pull request submitted

Toothpick: A theory of bytecode machines

2015-01-28T22:50:37+00:00

In working on my Batbridge simulators and other adventures in computing hardware, one of the problems I have repeatedly encountered is the need to generate bit encodings of symbolic instructions. When working with “well known” architectures with existing tooling support, one can usually find an “off the shelf” assembler either in the GCC collection or elsewhere. However when you’re working against a custom instruction set you’re on your own for tooling.

So lets invent yet another assembler!

An assembler is simply a program that describes how to construct a binary instruction encoding from a set of symbolic human-writable mnemonics representing data manipulating directives to computing hardware. add, sub[tract], mul[tiply] and jump are good examples of such mnemonics. Each operation which a given “machine” (hardware/software interpreter) will accept is typically specified as a sequence of bits and bit encoded fields (sometimes abbreviated in octal or hexadecimal notation). So Batbridge for instance specifies that the add instruction is the bit string 110000tttttaaaaabbbbbiiiiiiiiiii where the prefix 110000 or 0x30 indicates the addition operation, the bits named t encode a number t∈[0..30] being the register to which the result of the addition will be stored, the bits named a subject to the same constraints as t name the register from which the “left” operand will be drawn and the b bits name the “right” operand same as the t and a operands. i is an arbitrary signed 10 bit integer (sign is the 11th bit).

Sitting down with a specification, you or I could construct a set of functions from values appropriately constrained to bit encoded instructions as laid out in an instruction set specification. However in doing such an assembler implementation, you will discover that you are simply encoding in the programming language of your choice a translation table laid out in full by the instruction set documentation. However realizing that the assembler which we are constructing by hand represents a printer to the bytecode machine’s reader, we can think of bytecodes as a language in the same sense that any other set of strings and productions constitutes a language.

A Theory of Bytecode Machines

It happens that we have good formal theories about languages and their structures. If we stop and squint, we can see that the individual opcodes are analogous to terminals or verbs. If we allow ourselves to model opcodes as words in a language, then a program (a production of words) is a sentence for which we can establish a grammar. For instance a ELF formatted program could be considered to be a sequence of blocks of instructions (verbs) defined in terms of words (opcodes) and formatted with an appropriate header/footer (the ELF file structure). As these are all linguistic constructs, a program capable of generating these values must be a direct production of the specification which lays out these productions in the same way that a lex/yacc lexer parser pair is a mechanical production of the lexing and parsing rules which define a computer language.

This leads to the idea that, just as we now rarely write parsers and lexers preferring to derive them automatically from specifications of the languages we wish them to process since a as suggested above a bytecode is simply a language on bit strings rather than ASCII or Unicode characters the consequent that we can use the same class of tools to generate assemblers and disassemblers as we use to generate parsers and printers should be obvious. We just need to construct formal languages describing the instructions our machines accept.

An Assembler Generator

Just as we have theories about “interesting” sets of languages and the parser techniques which may be applied to efficiently recognize and process them, so too rather than being able to automatically assemble arbitrary programs to arbitrary bytecodes we must first recognize that as a bytecode is a language on bit strings and as there exist languages which require a Turing machine to recognize in addition to uncomputable languages consequently one could invent an “interesting” (but I will argue in a minute silly) bytecode for which no assembler or recognizer can be automatically generated.

So what features define “interesting” bytecodes? In hardware terms as I discuss elsewhere memories and lookup tables are just about the most expensive thing you can build in terms of chip area and time. As a result, the overwhelming majority of computer architectures are not what I shall call “micro-programmable”, that is users cannot give meaning to new sequences of bits and rather interact with the computer by chaining together a few bit-verbs or opcodes which are built into the chip. There do exist such micro-programmable computers, FPGAs, but they tend to be special purpose prototyping machines and are not in wide usage compared to fixed instruction set machines.

Another characteristic of “interesting” bytecodes is in the name, bytes or words. Many years have gone by since bit addressed machines were last built with commercial intent. Instead commercial machines and off the shelf memories tend to address “blocks” of bits or “lines” being strings a power of two in length which may be divided into substrings smaller powers of two in length as the user may see fit at no additional hardware complexity cost by simply selecting bits through selecting predefined subdivisions of lines known as “words” or using bit masking and shifting to select strings smaller than a single word. Strings larger than a single word must be dealt with by using multiple operations on words.

This theory of a fixed word size, fixed instruction set machine characterizes the overwhelming majority of instruction sets and machines designed and built by Intel, ARM, Symbolics, Thinking Machines, and others over the past decades due mainly to the ease of implementation and efficiency of such designs.

So what does this theory about a “useful” bytecode machine buy us? It implies that we can describe a “useful” machine as a bitstring length denoting word size. Endianness must also be considered. Then given an enumeration of the various opcodes and how to generate them as bitstrings, you can completely describe an “interesting” instruction set as characterized above.

Assemblers are simply programs that deal with translating instructions set mnemonics writable by a programmer or more often than not a compiler to bytecodes which a machine can execute. The details of computing label addresses are entirely for the most part independent of the specific architecture in so much as they are determined by properties of the specific target bytecode described above. This suggests that there must exist a function

(assemble p ← Platform c ← Codes)

Which makes sense if you consider a single target assembler to be the partial application of such an assembler to a platform.

Talk is cheap, show me the code

First things first, we need a way to represent an “interesting” instruction set.

(deftype Architecture
    "A structure representing an \"interesting\" bytecode architecture
    from which a parser or a generator can be computed."
  [word-width         ;; ← Num, how many bits in a word
   pointer-resolution ;; ← Num, how many bits forwards does *(x+1) go
   opcodes            ;; ← Map [Memonic → Opcode], opcode formatting rules
   ])

So what do individual opcodes look like? We’re gonna need some bit vector formatting engine to do real code bytecode generation with.

I will define an opcode to be a bitstring of fixed length (this is important, there are instruction sets with variable length “opcodes” however as the different length forms have different operational semantics I shall define them to be different operations and declare the instruction set architects silly people) composed of a sequence of concatenated parameters or fields and constants which may be either signed or unsigned. Fields are named or anonymous. The set of named field names in a single opcode must have no repetitions. We want to be able to write something like the following

;; IFLT 0x20  100000 _____ aaaaa bbbbb iiiiiiiiiii
;; execute the next instruction IFF (< a b)
(opcode :iflt
  (const-field            :icode 6  0x20)
  (enforced-const-field   :_     5  0)
  (parameter-field        :a     5  register?)
  (parameter-field        :b     5  register?)
  (signed-parameter-field :i     11 literal?))

So in this example instruction from Batbridge, we define an instruction as the ordered bit string where the bits [0..5] are named :icode and fixed with the constant value of 0x20. We then have an anonymous field 5 bits long taking the bits [6..10] which we force to have the value of 0 since they are defined to be ignored by the instruction set. We then have a pair of (unsigned) 5-bit registers, which must satisfy some guard predicate ensuring that the value in question fits within the value domain which the instruction set allows for registers. Finally we have a signed field 11 bits long, which again is guarded with a value domain predicate. This notation encodes the bit format string, as well as the value domains for each element of the bit format string in a form that can be easily inspected by either an assembler or a disassembler program.

So what does that expression build out to as a data structure?

{:width 32,
 :params (:_ :a :b :i),
 :fields ({:offset 26,
           :width 6,
           :name :icode,
           :type :const,
           :value 35}
          {:offset 21,
           :width 5,
           :name :_,
           :type :enforced-const,
           :value 0}
          {:offset 16,
           :width 5,
           :name :a,
           :type :unsigned-field,
           :pred #<batbridge$register_QMARK_ toothpick.isa.batbridge$register_QMARK_@4bd4095>}
          {:offset 11,
           :width 5,
           :name :b,
           :type :unsigned-field,
           :pred #<batbridge$register_QMARK_ toothpick.isa.batbridge$register_QMARK_@4bd4095>}
          {:offset 0,
           :width 11,
           :name :i,
           :type :signed-field,
           :pred #<batbridge$literal_QMARK_ toothpick.isa.batbridge$literal_QMARK_@38334886>})}

Given this datastructure, and a sequence (opcode . params), we can trivially compute a map (zipmap (:params (get isa opcode)) params), and then fold right over the fields of the instruction computing each field then shifting and anding it into place in the bit vector to construct the bit encoding of the instruction.

As implemented this is a little fragile because it assumes that the computer running the assembler assembler has a larger or equal integer size than the target, as attempting to construct an int aka bit vector that’s too large will yield… interesting issues. Future fixes are to use a real sequence of bits, or to rewrite this logic in terms of multiplication by two and addition.

Deriving a parser for this structure is also (fairly) straightforwards, one simply can simply generate an alternation matcher on bitstring matchers matching the individual instructions. “interesting” ISAs tend to put the opcode specification in the “low” bits of each instruction word so as a we can simply generate a jump table by masking on the least significant bits but that’s not implemented yet.

Also this representation falls into jneen’s type tagging pitfall, but this is old code and implementation details thereof so I’ll forgive myself and fix it later.

We can then define an “architecture” by constructing a composite map as above having a word size and pointer interval specification and a map from keywords naming instructions to many such instruction maps.

So, we can describe an instruction stream as a sequence of instructions, and we can assemble individual instructions by interpreting from their representation above, so if we wanted to encode a sample program

[[:label :top]
 [:op [:add [:param [:register 0]]
            [:param [:const 0]]
            [:param [:const 1]]]]
 [:label 1]
 [:op [:add [:param [:register 1]]
            [:param [:const 0]]
            [:param [:const 2]]]]
 [:label 2]
 [:op [:add [:param [:register 0]]
            [:param [:register 0]]
            [:param [:register 1]]]]
 [:label 3]]

We can simply foldr a label location computing operation since all of our opcodes are defined to be fixed length even if they are abstractly variable length, then in a second pass we assemble each instruction against the label location context and the instruction set, giving us a sequence of bytecodes. Done! ∎

This is essentially my Toothpick project, which seeks to provide exactly this meta-assembler facility and has already proven its worth to me in automating the generation of assemblers for the various toy architectures with which I’ve worked at school. It’s not done yet and as noted above it needs some spit and shoeshine but I think it represents a remarkably simple and powerful idea.

The real power of this approach is that, extended, this can enable the automatic generation of LLVM backends! If an instruction specification were to include its operational semantics written in terms of LLVM instructions one could imagine a trivial derived LLVM backed which sequentially scans emitted LLVM assembly matching out the platform translation of the instruction “at point” and then assembling the resulting instruction stream as above. Since LLVM already provides a massively portable C compiler infrastructure, this means that given only a formal platform specification one could construct a mediocre C compiler only from the platform specification.

To take this idea even further, one could also imagine generating a platform simulator from either symbolic instructions or bit encoded instructions from only the augmented ISA with semantics instructions.

Generating muxing logic in Verilog or VHDL should also be trivial from this structure.

Generate all the things! Correctness by specification & construction! Toy architectures for everyone!

Oxcart going forwards

2014-12-11T14:35:07+00:00

When I last wrote about Oxcart work pretty much went on hiatus due to my return to school. As there has been some recent interest in the status of Lean Clojure overall I thought I’d take the opportunity to review the state of Oxcart and the plan for Oxcart going forwards.

Oxcart and Clojure

The static linking approach and lack of run time dynamism found in Oxcart is explicitly at odds with the philosophy of core Clojure. Where Clojure was designed to enable live development and makes performance sacrifices to enable such development as discussed here, Oxcart attempts to offer the complement set of trade offs. Oxcart is intended as a pre-deployment static compiler designed to take a working application and to the greatest extent possible wring more performance out of unchanged (but restricted) Clojure as PyPy does for Python. As Oxcart explicitly avoids the dynamic bindings which Clojure embraces, Alex Miller, the Clojure Community Manager, has repeatedly stated that he expects to see little cross pollination from Oxcart and related work to Clojure itself.

This would be all well and good, were it not for the existing behavior of Clojure’s clojure.lang.RT class. As currently implemented in Clojure 1.6 and 1.7, RT uses its <initc> method to compile the following resources with clojure.lang.Compiler.

“clojure/core”
“clojure/core_proxy”
“clojure/core_print”
“clojure/genclass”
“clojure/core_deftype”
“clojure/core/protocols”
“clojure/gvec”
“clojure/instant”
“clojure/uuid”

These represent about 10799 lines of code, all of which could easily be statically compiled and most importantly tree shaken ahead of time by Oxcart or another tool rather than being loaded at boot time. This also means that the unavoidable booting of Clojure itself from source can easily dominate loading user programs especially after static compilation to raw classes. A quick benchmark on my machine shows that booting a Clojure 1.6 instance, loading a ns containing only a -main that only prints “hello world” takes ~2.8 seconds from source compared to ~2.5 seconds booting the same program compiled with Oxcart suggesting that the cost of booting Clojure is the shared ~2.5 second boot time. This is the test.hello benchmark in Oxcart’s demos.

$ git clone git@github.com:oxlang/oxcart.git &&\
  cd oxcart &&\
  git checkout 0.1.2 &&\
  bash bench.sh test.hello
Running Clojure 1.6.0 compiled test.hello....
Hello, World!

real    0m1.369s
user    0m3.117s
sys     0m0.083s
Oxcart compiling test.hello....
Running Oxcart compiled test.hello....
Hello, World!

real    0m1.212s
user    0m2.487s
sys     0m0.073s

Then there’s the test.load benchmark. This benchmark as-is pushes credulity because it compiles 502 functions of which only the -main which uses none of the other 501 will be invoked. This reflects more on program loading time than on the loading time of “clojure/core”, but I still think instructive in the costs of boot time compilation, showing a ~7s boot time for Clojure compared to a ~2.5s boot time for Oxcart. As arbitrary slowdowns from macroexpansions which Thread/sleep would be entirely possible I consider this program within the bounds of “fairness”.

A Fork in the Road

There are two solutions to this limitation, and both of them involve changing the behavior of Clojure itself. The first is my proposed lib-clojure refactor. Partitioning Clojure is a bit extreme, and in toying with the proposed RT → Util changes over here I’ve found that they work quite nicely even with a monolithic Clojure artifact. Unfortunately there seems to be little interest from Clojure’s Core team (as judged via Alex’s communications over the last few months) in these specific changes or in the static compilation approach to reducing the deployment overhead of Clojure programs. The second is to fork Clojure and then make lib-clojure changes which solves the problem of convincing Core that lib-clojure is a good idea but brings its own suite of problems.

Oxcart was intended to be my undergraduate thesis work. While the 16-25% speedup previously reported is impressive, Oxcart does nothing novel or even interesting under the hood. It only performs four real program transformations: lambda lifting, two kinds of static call site linking and tree shaking. While I suppose impressive for an undergrad, this project also leaves a lot on the table in terms of potential utility due to its inability to alter RT’s unfortunate loading behavior. I also think there is low hanging fruit in doing unreachable form elimination and effect analysis, probably enough that Oxcart as-is would not be “complete” even were its emitter more stable.

I’m reluctant to simply fork Clojure, mainly because I don’t think that the changes I’ve been kicking about for lib-clojure actually add anything to Clojure as a language. If I were to fork Clojure, it’d be for Oxlang which actually seeks to make major changes to Clojure not just tweak some plumbing. But writing a language so I can write a compiler is frankly silly so that’s not high on the options list. The worst part of this is that forking Clojure makes everything about using Oxcart harder. Now you have dependencies at build time (all of “stock” Clojure) that don’t exist at deployment time (my “hacked” Clojure). Whatever hack that requires either winds up complicating everyone’s project.clj or in an otherwise uncalled for leiningen plugin just like lein-skummet. Tooling needs to be able to get around this too when every library you’d want to use explicitly requires [org.clojure/clojure ...] which totally goes away once Oxcart emits the bits you need and throws the rest out. Most of all I don’t want to maintain a fork for feature parity as time goes on. However I also don’t see any other a way to get around RT’s existing behavior since the RT → Util refactor touches almost every java file in Clojure.

Flaws in the Stone

Oxcart itself also needs a bunch of work. While I think that Nicola has done an awesome job with tools.analyzer and tools.emitter.jvm I’m presently convinced that while it’s fine for a naive emitter (what TEJVM is), it’s a sub-optimal substrate for a whole program representation and for whole program transforms.

Consider renaming a local symbol. In the LLVM compiler infrastructure, “locals” and other program entities are represented as mutable nodes to which references are held by clients (say call sites or use sites). A rename is then simply an update in place on the node to be changed. All clients see the change with no change in state. This makes replacements, renames and so forth constant time updates. Unfortunately due to the program model used by tools.analyzer and tools.emitter.jvm, such efficient updates are not possible. Instead most rewrites degenerate into worst case traversals of the entire program AST when they could be much more limited in scope. Cutaway is one experiment in this direction, but it at best approximates what clojure.core.logic.pldb is capable of. I hope that over Christmas I’ll have time to play with using pldb to store, search and rewrite a “flattened” form of tools.analyzer ASTs.

Oxcart is out of date with tools.emitter.jvm and tools.analyzer. This shouldn’t be hard to fix, but I just haven’t kept up with Nicola’s ongoing work over the course of the last semester. This will probably get done over Christmas as well.

Oxcart doesn’t support a bunch of stuff. As of right now, defmulti, defmethod, deftype, defprotocol, proxy, extend-type and extend-protocol aren’t supported. I’m pretty sure all of these actually work, or could easily work, they just didn’t get done in the GSoC time frame.

Finally and I think this is the thing that’s really blocking me from working on Oxcart: it can’t compile clojure.core anyway. This is a huge failing on my part in terms of emitter completeness, but it’s a moot point because even if I can compile clojure.core with Oxcart RT is gonna load it anyway at boot time. I also suspect that this is an incompleteness in the project as a whole which probably makes it an unacceptable thesis submission although I haven’t spoken with my adviser about it yet.

The Endgame

As of right now I think it’s fair to call Oxcart abandoned. I don’t think it’s a worthwhile investment of my time to build and maintain a language fork that doesn’t have to be a fork. I talked with Alexander, one of the clojure-android developers and a fellow GSoC Lean Clojure student/researcher about this stuff and the agreement we reached was that until 1.7 is released there’s no way that the lib-clojure changes will even get considered and that the most productive thing we can do as community members is probably to wait for 1.8 planning and then try to sell lib-clojure and related cleanup work on the basis of enabling clojure-android and lean clojure/Oxcart. Practically speaking in terms of my time however, if it’s going to be a few months until 1.7 and then a year until 1.8, that only gives leaves me my last semester of college to work on Oxcart against an official version of Clojure that can really support it. If that’s what it takes to do Oxcart I’ll likely just find a different thesis project or plan on graduating without a thesis.

(with-plug
  That said, serious interest in Oxcart as a deployment tool or another
  contributor would probably be enough to push me over the futility hump
  of dealing with a Clojure fork and get Oxcart rolling.)

The Future of the LispM

2014-11-28T05:10:38+00:00

This is also addressed to @lojikil towards whom I have threatened to write this post on multiple occasions. It is expected to piss off loper-os. Above all, it represents my opinions backed up only by a cursory research effort. You have been warned.

Background Of Historical LispMs

LispMs occupy a transitional period of computing industry history between time shared mainframes and the single user workstations that would become today’s desktop computers. Prior to John McCarthy’s invention of time sharing, all computers had been single program machines which multiple users shared via batch scheduling. The introduction of time sharing opened the way to exploring how to make computers useful interactively for many users sitting at terminals. Under bach processing, users would present fully formed programs written by hand or with little tool assistance to computer techs who would run the programs and return printed output.

The concept of time sharing and interactivity lead to the development of more dynamic programming environments including the Read Eval Print Loop. However at the same time, the falling costs of integrated circuits (and DoD funding for research into the design of such VLSI machines) began to put the first workstations, single user machines featuring large memories, within the financial reach of well funded AI laboratories which previously dependent on time shared mainframes.

In order to improve interactive performance, Richard Greenblatt and Thomas Knight developed the cons machine, the first of the MIT Lisp machines. Priced at ~$50,000 in 1970s dollars cons and the successor cadr machine were single user workstations built primarily it seems to address the restrictions on memory and memory performance and subsequent detrimental impact on general program performance faced on time shared systems due to the large memory footprint of Lisp programs and page thrashing. Hardware/microcode support was added for maintaining the relatively complex Lisp environment structures, and for many of the primitive lookup/load/store operations on cons cell structures that characterize a traditional Lisp implementation’s data access pattern. This combination of machine support and microcoding enabled Lisp “compilers” to be far simpler and offloaded the responsibility for maintaining complicated structures such as the environment from the compiler to the microcode system. Best of all as the microcode closely corresponded relatively directly to Lisp, on the rare occasions that truly writing microcode for say device drivers was called for doing so was easy because users could transparently invoke and define microcode from their native Lisp environment.

Seeing a business opportunity, in the early ’80s a number of MIT AI lab researchers departed and founded Symbolics Inc. and Lisp Machines Inc. for the purpose of building and selling Knight machine derived Lisp workstations. Ultimately however, Lisp machines did fall out of favor during the AI winter(s) and lost to what we now call the personal computer. I have seen the argument made that they were too big-ticket and missed the mass market as a result. Thus our story really begins.

Of Modern Hardware

In the years since the demise of the dedicated LispMs, Intel and the other major chip manufacturers have progressed from single cycle to pipelined, cache accelerated, superscalar and out of order machines. Knowledge of these designs is assumed. I suggest my earlier introduction series to these topics if you are not already familiar with them. At a high level however, each step of this progression trades off chip area, transistors (complexity) and power budget to wring more instruction level parallelism out of existing programs.

The Symbolics 3600 technical report from ‘83 claims a best case cycle time of 180ns (~5MHz) and a worst case of 250ns (4MHz). Arbitrary reads from main memory are priced at 600ns (3x the cost of a single instruction). This low (by modern standards) slowdown between the processor and memory meant that one could write programs that did lots and lots of random memory access and still have reasonable performance expectations. However, as a stack machine architecture, there is no opportunity for the instruction level parallelism exploited by modern architectures as the exact state of the stack which is depended on by every instruction changes with every instruction. This forces all program execution to wait during slow operations like main memory reads, division and floating point math due to the sequential data dependency on the stack. This is a fundamental architectural failure common to all stack machines and skirted by modern software stack machines like the JVM by compiling literal stack instructions to a register machine in order to recover instruction level parallelism.

Modern processors claim cycle times of 0.3̅ns for a 3GHz machine, often running multiple instructions per cycle. Unfortunately while processors core clocks have gotten faster since the days of the 3600, memories have not fundamentally improved. On modern hardware, 100ns for a random main memory read seems to be more or less normal. This represents a 27x speedup in processor speed mismatched with a 6x speedup in main memory latency for an effective 4.5x slowdown on memory reads. Now this is admittedly worst case memory latency. Thanks to the L1 and L2 cache layers, memory read times of 1ns to 3ns are not uncommon meaning that for cache efficient programs it is quite reasonable to expect less than a 2ns or 6 cycles for memory reads hitting in the L1 or L2 cache.

I mention this, because techniques like cdr coding serve to bring the average case of list access and construction down from their worst case behavior of generating massive heap fragmentation (and thus defeating caching and prefetching) towards the behavior of nicely caching sequential data layouts (classical arrays) typically via tree like structures or SRFI-101. While traditional linked list structures due to potential heap fragmentation provide potentially atrocious cache locality and this cripple the performance of modern cache accelerated machines, more modern datastructures can simultaneously provide the traditional cons list pattern while improving cache locality and thus performance characteristics. Guy Steele himself has even come out arguing this point.

Without literal implementation as a stack machine traversing linked lists and providing special hardware support for binding and soforth, the LispM instantly looses much of the vaunted simplicity which makes it attractive to program directly and becomes simply another register machine in the mold of modern ARM or Intel machines with strange long running microcoded instructions reminiscent more of the VAXen than of modern RISC inspired processors. In short due to the many performance limitations of linked lists as data structures we as an industry will never again build machines as the original LispMs were for the sole purpose of traversing and manipulating linked list data structures. Modern hardware simply offers better performance with more general applicability.

Of Modern Languages

We’ve also come a long way in language design and implementation. Compilers, once slow, have gotten faster and smarter. Virtual machines like the JVM, JavaScript and the CLR are becoming widely used deployment targets. The ML and Haskell families of languages have introduced us to concepts of real abstract types and abstract effects which can be used to build programs coupled only by the abstract properties of the data being consumed, generated and effects produced. Type inference is even making such fancy behavior manageable by mere mortals, while providing language implementations with more and more information with which to perform both program level optimization and micro-optimization not possible in traditional naive lisps.

Of LispMs Future

While we’ll never build a hardware LispM again, I suspect that we will see LispM like systems return one day in the not too distant future. Not as hardware, but as software or a virtual machine designed to run atop existing and entirely adequate modern hardware. Now in 10 years someone may make a commercial hardware LispM at which point I will be happy to eat my words, but as of right now I don’t see it happening.

The JVM and the .net CLR have proven to be amazing platforms. While their allocation requirements perhaps prohibit their use for implementing operating systems and driver code (not that people haven’t tried this sort of thing) they do offer excellent and standardized platforms. It is my personal belief that, by leveraging the flexibility that Lisp dialects have for both data driven DSLs and macro DSLs that it would be more or less trivial to implement an operating system using a DSL for generating platform specific assembly. As the “kernel” OS is implemented in the final “host” LispM language editing the kernel is possible albeit difficult due to the inherent difficulties of hot swapping operating systems.

Add a JIT (no this isn’t easy), preferably implemented in the same assembler DSL as the host operating system, and the stage is set for building a LispM environment as a virtual machine running atop modern hardware and using modern JIT to achieve reasonable performance characteristics. From this vantage the way is clear to implementing the rest of the operating system and user land in a fully fledged lisp with good reloading and introspection support atop this kernel of a JIT and enough of an OS to provide memory and process management.

This is, I think, an entirely reasonable and practical project for a smallish (~3 or fewer) team and a single target architecture (Intel). There was a project, Dream, an r4rs x86 assembler dsl in which an r4rs interpreter and operating system were implemented. It booted and worked more or less fine, and while it lacked polish I think it serves as an excellent proof of concept that such a self hosting Lisp (assembler os runtime) triple is viable. I don’t argue that such an implementation will be trivially portable to a wide variety of target hardware because as experience shows porting nominally portable Linux, FreeBSD or FreeRTOS is actually quite hard and porting a JIT can at best be as hard due to tight coupling with the specific behavior of the target machine but this is why we specify an abstract machine and then let implementations deal with the mess as the JVM does.

I also think that Lisp as a language family could stand to learn some new tricks if someone were to make the effort to build a full Lispy OS/VM thing.

Clojure is awesome. I’ve written a boatload of it. It’s an amazingly simple and elegant tool, and its persistent/immutable datastructures are absolutely worth stealing as is its concept of hygenic macros via symbol qualification and probably the concept of protocols/interfaces which Clojure kinda inherits from its primary host Java.

Racket is also awesome although I can’t claim to have written a bunch of it. Its approach to static compilation, strong support for documentation and examples via Scribble, its pattern matching and typechecking facilities are totally worth stealing.

Shen is interesting if only due to its built in Prolog engine enabling search for arbitrary proofs with regards to arbitrary program properties. While criticized by the Haskell community for disregarding completeness being able to write and search for proofs with regards to arbitrary properties of programs not just types is I think an amazingly powerful one albeit an active research area.

But is it a LispM?

There are some people who will rail against this vision of mine that the “OS” is as low as we need to try and push a language. To my mind, the advantages of pushing a language to the hardware level are scant enough as argued above that it simply does not justify the investment or the delay. Even the OS may be too low to push a language and that while the adventure of building an OS with a language to host the language ala Oberon because while integrating the language with the OS does offer maximal conceptual unity across the combined platform and provide a huge leap in ease of integrating pipelines of different applications it also forcibly discards programs not written in HOST_LANG for HOST_PLATFORM. This is a problem if only because the Unix model of computing as implemented on Linux with the help of the GNU user land is second only to the mach hiding inside of Mac OS X in terms of apparent developer adoption. Developers booting the next Dream for the first time won’t find familiar text editors or tools. It will be a brave new world.

Maybe that’s a good thing. Maybe we can escape the legacy of Unix with untyped byte streams and instead revive some of Plan 9 with a basis in lispy goodness and with more types everywhere. Maybe there’s even a place in this for algebraic effects. Maybe we can join urbit in looking towards a functional, fully netwoked future. Maybe. If only we can look past the obsolete machines and operating systems of yesteryear with which we seem content to circlejerk and move forwards with appropriate context and vision.

Or maybe we’ll just be stuck with Linux, Unix, OpenSSL and the rest of our somewhat suspect tower of legacy code out of sheer inertia and continue getting caught up in language of the month rages out of collective ADD.

 We have not wings, we cannot soar;
       But we have feet to scale and climb
 By slow degrees, by more and more,
       The cloudy summits of our time.

 The mighty pyramids of stone
       That wedge-like cleave the desert airs,
 When nearer seen, and better known,
       Are but gigantic flights of stairs.

 The distant mountains, that uprear
       Their solid bastions to the skies,
 Are crossed by pathways, that appear
       As we to higher levels rise.

 The heights by great men reached and kept
       Were not attained by sudden flight,
 But they, while their companions slept,
       Were toiling upward in the night.

~ Longfellow

Compiler introduction of transients to pure functions

2014-09-17T23:13:10+00:00

In Clojure and pure functinal languages, the abstraction is provided that values cannot be updated, only new values may be produced. Naively, this means that every update to a value must produce a full copy of the original value featuring the desired change. More sophisticated implementations may opt for structural sharing, wherein updated versions of some structure share backing memory with the original or source value on the substructures where no update is performed. Substructures where there is an update must be duplicated and updated as in the naive case, but for tree based datastructures this can reduce the cost of a typical update from O(N) on the size of the updated structure to O(log(N)) because the rest of the structure may be shared and only the “updated” subtree needs to be duplicated.

This means that tree based structures which maximize the ammount of sharable substructure perform better in a functional context because they minimize the fraction of a datastructure which must be duplicated during any given update.

Unfortunately however, such structural sharing still carries a concrete cost in terms of memory overhead, garbage collection and cache and performance when compared to a semantically equivalent update in place over a mutable datastructure. A mutable update is typically O(1), with specific exceptions for datastructures requiring amortized analysis to achieve near O(1) performance.

Ideally, we would be able to write programs such that we preserve the abstraction of immutable values, while enabling the compiler or other runtime to detect when intentional updates in place are occurring and take the opportunity to leverage the performance improvements consequent from mutable data in these cases while ensuring that no compiler introduced mutability can become exposed to a user through the intentional API as built by the programmer.

In such a “pure” language, there is only one class of functions, functions from Immutable values to Immutable values. However if we wish to minimize the performance overhead of this model four cases become obvious. λ Immutable → Immutable functions are clearly required as they represent the intentional API that a programmer may write. λ Mutable → Mutable functions could be introduced as implementation details within an λ Immutable → Immutable block, so long as the API contract that no mutable objects may leak is preserved.

Consider the Clojure program

(reduce (partial apply assoc)
    {}
    (map vector
	   (range 10000)
	   (range 10000)))

This program will sequentially apply the non-transient association operation to a value (originally the empty map) until it represents the identity mapping over the interval [0,9999]. In the naive case, this would produce 10,000 full single update coppies of the map. Clojure, thanks to structural sharing, will still produce 10,000 update objects, but as Clojure’s maps are implemented as log₃₂ hash array mapped tries, meaning that only the array containing the “last” n % 32 key/value pairs must be duplicated, more the root node. This reduces the cost of the above operation from T(~10,000²) to T(10,000*64) ≊ T(640,000) which is huge for performance. However, a Sufficiently Smart Compiler could recognize that the cardinality of the produced map is max(count(range(10000)), count(range(10000))), clearly being 10000. Consequently an array map of in the worst case 10000 elements is required given ideal hashing, however assuming a load factor of 2/3 this means our brilliant compiler can preallocate a hashmap of 15000 entries (presumed T(1)), and then perform T(10000) hash insertions with a very low probability of having to perform a hash table resize due to accounting for hash distribution and sizing the allocated table to achieve a set load factor.

Clearly at least in this example the mutable hash table would be an immense performance win because while we splurge a bit on consumed memory due to the hash table load factor (at least compared to my understanding of Clojure’s hash array mapped trie structure) the brilliantly compiled program will perform no allocations which it will not use, will perform no copying, and will generate no garbage compared to the naive structurally shared implementation which will produce at least 9,967 garbage pairs of 32 entry arrays.

The map cardinality hack is it’s own piece of work and may or may not be compatible with the JVM due to the fact that most structures are not parametric on initial size and instead perform the traditional 2*n resizing at least abstractly. However, our brilliant compiler can deduce that the empty map which we are about to abuse can be used as a transient and made static when it escapes the scope of the above expression.

Consider the single static assignment form for the above (assuming a reduce definition which macroexpands into a loop) (which Clojure doesn’t do).

	[1 ] = functionRef(clojure.core/partial)
	[2 ] = functionRef(clojure.core/apply)
	[3 ] = functionRef(clojure.core/assoc)
	[4 ] = invoke(2, 3, 4)                   ;; (partial apply assoc)
	[5 ] = {}
	[6 ] = functionRef(clojure.core/map)
	[7 ] = functionRef(clojure.core/vector)
	[8 ] = functionRef(clojure.core/range)
	[9 ] = 10000
	[10] = invoke(8, 9)                      ;; (range 10000)
	[11] = invoke(6, 7, 10, 10)              ;; (map vector [10] [10])
	[12] = functionRef(clojure.core/first)
	[13] = functionRef(clojure.core/rest)
loop:
	[14] = phi(5,  18)
	[15] = phi(11, 19)
	[16] = if(13, cont, end)
cont:
	[17] = invoke(12, 14)
	[18] = invoke(4, 14, 15)
	[19] = invoke(13, 15)
	[20] = jmp(loop)
end:
	[21] = return(18)

Where the phi function represents that the value of the phi node depends on the source of the flow of control. Here I use the first argument to the phi functions to mean that control “fell through” from the preceeding block, and the second argument to mean that control was returned to this block via instruction 20.

This representation reveals the dataflow dependence between sequential values of our victim map. We also have the contract that the return, above labeled 21, must be of an Immutable value. Consequently we can use a trivial dataflow analysis to “push” the Immutable annotation back up the flow graph, giving us that 18, 14 and 5 must be immutable, 5 is trivially immutable, 18 depends on 14, which depends on 18 and 5, implying that it must be immutable as well. So far so good.

We can now recognize that we have a phi(Immutable, Immutable) on a loop back edge, meaning that we are performing an update of some sort within the loop body. This means that, so long as Transient value is introduced into the Immutable result, we can safely rewrite the Immutable result to be a Transient, and add a persistent! invocation before the return operation. Now we have phi(Immutable, Transient) → Transient which makes no sense, so we add a loop header entry to make the initial empty map Transient giving us phi(Transient, Transient) → Transient which is exactly what we want. Now we can rewrite the loop update body to use assoc! → Transient Map → Immutable Object → Immutable Object → Transient Map rather than assoc → Immutable Map → Immutable Object → Immutable Object → Immutable Map.

Note that I have simplified the signature of assoc to the single key/value case for this example, and that the key and value must both be immutable. This is required as the persistent! function will render only the target object itself and not its references persistent.

This gives us the final operation sequence

	[1 ] = functionRef(clojure.core/partial)
	[2 ] = functionRef(clojure.core/apply)
	[3 ] = functionRef(clojure.core/assoc!)
	[4 ] = invoke(2, 3, 4)                      ;; (partial apply assoc)
	[5 ] = functionRef(clojure.core/transient!)
	[6 ] = {}
	[7 ] = invoke(5, 6)
	[8 ] = functionRef(clojure.core/map)
	[9 ] = functionRef(clojure.core/vector)
	[10] = functionRef(clojure.core/range)
	[11] = 10000
	[12] = invoke(10, 11)                       ;; (range 10000)
	[13] = invoke(8, 9, 12, 12)                 ;; (map vector [12] [12])
	[14] = functionRef(clojure.core/first)
	[15] = functionRef(clojure.core/rest)
loop:
	[16] = phi(7,  20)
	[17] = phi(13, 21)
	[18] = if(17, cont, end)
cont:
	[19] = invoke(14, 17)
	[20] = invoke(4, 16, 17)
	[21] = invoke(15, 17)
	[22] = jmp(loop)
end:
    [23] = functionRef(clojure.core/persistent!)
	[24] = invoke(23, 20)
	[25] = return(24)

Having performed this rewrite we’ve one. This transform allows an arbitrary loop using a one or more persistent datastructures a accumulators to be rewritten in terms of transients if there exists (or can be inferred) a matching Transient t → Transient t updater equivalent to the updater used. Note that if a non-standard library updater (say a composite updater) is used, then the updater needs to be duplicated and if possible recursively rewritten from a Persistent t → Persistent t by replacing the standard library operations for which there are known doubles with their Transient t counterparts until either the rewrite fails to produce a matching Transient t → Transient t or succeeds. If any such rewrite fails then this entire transform must fail. Also note that this transformation can be applied to subterms… so as long as the Persistent t contract is not violated on the keys and values here of the map in a nontrivial example their computation as well could be rewritten to use compiler persisted transients.

Now yes you could have just written

(into {}
   (map vector
      (range 10000)
	  (range 10000)))

which would have used transients implicitly, but that requires that the programmer manually perform an optimization requiring further knowledge of the language and its implementation details when clearly a relatively simple transformation would not only reveal the potential for this rewrite but would provide it in the general case of arbitrarily many mutable accumulators rather than into’s special case of just one.

On Future Languages

2014-08-26T05:05:56+00:00

As Paul Philips notes at the end of his talk We’re Doing It All Wrong [2013], ultimately a programming language is incidental to building software rather than critical. Ultimately the “software developer” industry is not paid to write microcode. Rather, software as an industry exists to deliver buisness solutions. Similarly computers themselves are by a large incidental. Rather, they are agents for delivering data warehousing, search and transmission solutions. Very few people are employed to improve software and computer technology for the sake of doing so compared to the hordes of industry programmers who ultimately seek to use computers as a magic loom to create value for a business. In this light I think it’s meaningful to investigate recent trends in programming languages and software design as they pertain to solving problems not to writing code.

As the last four or five decades of writing software stand witness, software development is rarely an exercise in first constructing a perfect program, subsequently delivering it and watching pleased as a client makes productive use of it until the heat death of the universe. Rather, we see that software solutions when delivered are discovered to have flaws, or didn’t solve the same problem that the client thought that it would solve, or just didn’t do it quite right, or the problem changed. All of these changes to the environment in which the software exists demand changes to the software itself.

Malleability

In the past, we have attempted to develop software as if we were going to deploy perfect products forged of pure adamantium which will endure forever unchanged. This begot the entire field of software architecture and the top down school of software design. The issue with this model as the history of top down software engineering stands testament is that business requirements change if they are known, and must be discovered and quantified if they are unknown. This is an old problem with no good solution. In the face of incomplete and/or changing requirements all that can be done is to evolve software as rapidly and efficiently as possible to meet changes as Parnas argues.

In the context of expecting change, languages and the other tools used to develop changing software must be efficient to re-architect and change. As Paul Philips says in the above talk, “modification is undesirable, modifiability is paramount”.

Looking at languages which seem to have enjoyed traction in my lifetime, the trend seems to have been that, with the exception of tasks for which the language in which the solution was to be built was a requirement the pendulum both of language design and of language use has been swinging away from statically compiled languages like Java, C & C++ (the Algol family) towards interpreted languages (Perl, Python, Ruby, Javascript) which trade off some performance for interactive development and immediacy of feedback.

Today, that trend would seem to be swinging the other way. Scala, a statically checked language base around extensive type inference with some interactive development support has been making headway. Java and C++ seem to have stagnated but are by no means dead and gone. Google Go, Mozilla Rust, Apple Swift and others have appeared on the scene also fitting into this intermediary range between interactive and statically compiled with varying styles of type inference to achieve static typing while reducing programmer load. Meanwhile the hot frontier in language research seems to be static typing and type inference as the liveliness of the Haskell ecosystem is ample proof.

Just looking at these two trends, I think it’s reasonable to draw the conclusion that interpreted, dynamically checked, dynamically dispatched languages like Python and Ruby succeeded at providing more malleable programming environments than the languages which came before them (the Algol family & co). However while making changes in a dynamically checked language is well supported, maintaining correctness is difficult because there is no compiler or type checker to warn you that you’ve broken something. This limits the utility of malleable environments, because software which crashes or gives garbage results is of no value compared to software which behaves correctly. However, as previously argued, software is not some work of divine inspiration which springs fully formed from the mind onto the screen. Rather the development of software is an evolutionary undertaking involving revision (which is well suited to static type checking) and discovery which may not be.

As a checked program must always be in a legal (correct with respect to the type system) state by definition, this precludes some elements of development by trial and error as the type system will ultimately constrain the flexibility of the program requiring systematic restructuring where isolated change could have sufficed. This is not argued to be a flaw, it is simply a trade off which I note between allowing users to express and experiment with globally “silly” or “erroneous” constructs and the strict requirement that all programs be well formed and typed when with respect to some context or the programmer’s intent the program may in fact be well formed.

As program correctness is an interesting property, and one which static model checking including “type systems” is well suited to assisting with, I do not mean to discount typed languages. Ultimately, a correct program must be well typed with respect to some type system whether that system is formalized or not.

Program testing can be used to show the presence of bugs, but never to show their absence!

~ Dijkstra (1970)

Static model checking on the other hand, can prove the presence of flaws with respect to some system. This property alone makes static model checking an indispensable part of software assurance as it cannot be replaced by any other non-proof based methodology such as assertion contracts or test cases.

Given this apparent trade off between flexibility and correctness, Typed Racket, Typed Clojure and the recent efforts at Typed Python are interesting, because they provide halfway houses between the “wild west” of dynamic dispatch & dynamic checking languages like traditional Python, Ruby and Perl and the eminently valuable model checking of statically typed languages. This is because they enable programmers to evolve a dynamically checked system, passing through states of varying levels of soundness towards a better system and then once it has reached a point of stability solidify it with static typing, property based testing and other static reasoning techniques without translating programs to another language which features stronger static analysis properties.

Utility & Rapidity from Library Support

Related to malleability in terms of ultimately delivering a solution to a problem that gets you paid is the ability to get something done in the first place. Gone (forever I expect) are the days when programs are built without using external libraries. Looking at recent languages, package/artifact management and tooling capable of trivializing leveraging open source software has EXPLODED. Java has mvn, Python has pip, Ruby has gem, Haskell has cabal, Node has npm and the Mozilla Rust team deemed a package manager so critical to the long term success of the language that they built their cargo system long before the first official release or even release candidate of the language.

Why are package managers and library infrastructure critical? Because they enable massive code reuse, especially of vendor code. Building a webapp? Need a datastore? The investment of buying into any proprietary database you may choose has been driven so low by the ease with which $free (and sometimes even free as in freedom) official drivers can be found and slotted into place it’s silly. The same goes for less vendor specific code… regex libraries, logic engine libraries, graphics libraries and many more exist in previously undreamed of abundance (for better or worse) today.

The XKCD stacksort algorithm is a tongue in cheek reference to the sheer volume of free as in freedom forget free as in beer code which can be found and leveraged in developing software today.

This doesn’t just go for “library” support, I’ll also include here FFI support. Java, the JVM family of languages, Haskell, Ocaml and many others gain much broader applicability for having FFI interfaces for leveraging the decades of C and C++ libraries which predate them. Similarly Clojure, Scala and the other “modern” crop of JVM languages gain huge utility and library improvements from being able to reach through to Java and leverage the entire Java ecosystem selectively when appropriate.

While it’s arguably unfair to compare languages on the basis of the quantity of libraries available as this metric neglects functionally immeasurable quality and utility, the presence of any libraries is a legitimate comparison in terms of potential productivity to comparative absence.

What good is a general purpose building material, capable of constructing any manner of machine, when simple machines such as the wheel or the ramp must be reconstructed by every programmer? Not nearly so much utility as a building material providing these things on a reusable basis at little or no effort to the builder regardless of the ease with which one may custom build such tools as needed.

So What’s the Big Deal

I look at this and expect to see two trends coming out of it. The first of which is that languages with limited interoperability and/or limited library bases are dead. Stone cold dead. Scheme, RⁿRS, and Common Lisp were my introductions to the Lisp family of languages. They are arguably elegant and powerful tools, however compared to other tools such as python they seem offer at best equal leverage due to prevailing lack of user let alone newbie friendly library support compared to other available languages.

I have personally written 32KLoC in Clojure. More counting intermediary diffs. That I can find on my laptop. Why? Because Clojure unlike my experiences with Common Lisp and Scheme escapes the proverbial lisp curse simply thanks to tooling which facilitates library and infrastructure sharing at a lower cost than the cost of reinvention. Reinvention still occurs, as it always will, but the marginal cost of improving an existing tool vs writing a new one is in my experience a compelling motivator for maintaining existing tool kits and software. This means that Clojure at least seems to have broken free of the black hole of perpetual reinvention and is consequently liberating to attack real application critical problems rather than distracting programmers into fighting simply to build a suitable environment.

It’s not that Clojure is somehow a better language, arguably it ultimately isn’t since it lacks the inbuilt facilities for many interesting static proof techniques, but that’s not the point. As argued above, the language(s) we use are ultimately incidental to the task of building a solution to some problem. What I really want is leverage from libraries, flexibility in development, optional and/or incremental type systems and good tooling. At this task, Clojure seems to be a superior language.

The Long View

This is not all to say that I think writing languages is pointless, nor that the languages we have today are the best we’ve ever had let alone the best we ever will have at simultaneously providing utility, malleability and safety. Nor is this to say that we’ll be on the JVM forever due to the value of legacy libraries or something equally silly. This is however to say that I look with doubt upon language projects which do not have the benefit of support from a “major player”, a research entity or some other group willing to fund long term development in spite of short term futility simply because the literal price of bootstrapping a “new” language into a state of compelling utility is expensive in terms of man-years.

This conclusion is, arguably, my biggest stumbling block with my Oxlang project. It’s not that the idea of the language is bad, it’s that a tradeoff must be carefully made between novelty and utility. Change too much and Oxlang will be isolated from the rest of the Clojure ecosystem and will loose hugely in terms of libraries as a result. Change to little and it won’t be interesting compared to Clojure. Go far enough, and it will cross the borders of hard static typing, entering the land of Shen, Ocaml and Haskell and as argued above I think sacrifice interesting flexibility for uncertain gains.

Of Oxen, Carts and Ordering

2014-08-06T01:40:32+00:00

Well, the end of Google Summer of Code is in sight and it’s long past time for me to make a report on the state of my Oxcart Clojure compiler project. When last I wrote about it, Oxcart was an analysis suite and a work proposal for an emitter.

To recap the previous post: Clojure uses Var objects, an atomic, threadsafe reference type with support for naming and namespace binding semantics to create a literal global hashmap from symbols to binding vars, with a literal stack of thread local bindings. These vars form the fundamental indirection mechanism by which Clojure programs map from textual symbols to runtime functions. Being atomically mutable, they also serve to enable dynamic re-binding and consequently enable REPL driven development.

But for this second aspect of providing for dynamic redefinition of symbols, Clojure could be statically compiled eliminating var indirection and achieving a performance improvement.

Moreover, in the style of Clojurescript, exposing the full source of the language to an agressive static compiler could yield total program size improvements in comparison to programs running on the official Clojure compiler/runtime pair.

So. This was the premise upon which my Project Oxcart GSoC began. Now, standing near the end of GSoC what all has happened, where does the project stand and what do I consider results?

As of this post’s writing, e7a22a09 is the current state of the Oxcart project. The Oxcart loader and analyzer, built atop Nicola Mometto’s tools.analyzer.jvm and tools.emitter.jvm, is capable of loading and generating a whole program AST for arbitrary Clojure progrms. The various passes in the oxcart.passes subsystem implement a variety of per-form and whole program traversals including λ lifting, use analysis, reach analysis and taken as value analysis. There is also some work on a multiple arity function reduction system, but that seems to be a problem module at present. The oxcart.emitter subsystem currently features two emitters, only one of which I can claim is of my authoring. oxcart.emitter.clj is a function from an Oxcart whole program AST to a (do) form containing source code for the entire program as loaded. This has primarily been a tool for debugging and ensuring the sanity of various program transformations.

The meat of Oxcart is oxcart.emitter.jvm, a wrapper around a modified version of Clojure.tools.jvm.emit which features changes for emitting statically specialized bytecode which doesn’t make use of var indirection. These changes are new, untested and subject to change but they seem to work. As evidenced by the bench-vars.sh script in the Oxcart project’s root directory, for some programs the static target linking transform done by Oxcart can achieve a 24% speedup.

Times in ms.
Running Clojure 1.6.0 compiled test.vars....
1603.894357
1369.60414
1348.747718
1345.55669
1343.380462
1341.73058
1346.342237
1345.655224
1395.983262
1340.063011
1339.7823
1399.282023
1451.58462
1353.231654
1348.458151
Oxcart compiling test.vars....
Running Oxcart compiled test.vars....
1256.391888
1066.860551
1033.764971
1031.380052
1032.09911
1030.032666
1040.505754
1040.419533
1041.749353
1040.038301
1041.699772
1044.441135
1095.809551
1072.494466
1047.65782

How/why is this possible? The benchmark above is highly artificial in that it takes 500 defs, and tests over several thousand iterations of selectively executing half of them at random. This benchmark is designed to exaggerate the runtime cost of var indirection by being inefficient to inline and making great use of the (comparatively) expensive var dereference operation.

So what does this mean for Clojure? Is Oxcart proof that we can and should build a faster Clojure implementation or is there a grander result here we should consider?

Clojure’s existing compiler operates on a “good enough” principle. It’s not especially nice to read nor especially intelligent, but it manages to produce reasonably efficient bytecode. The most important detail of the reference Clojure compiler is that it operates on a form by form basis and is designed to do so. This is an often forgotten detail of Clojure, and one which this project has made me come to appreciate a lot more.

When is static compilation appropriate and valuable? Compilation is fundamentally an analysis and specialization operation designed to make a trade off between “start” or “compile” time and complexity and runtime performance. This suggests that in different contexts different trade offs may be appropriate. Furthermore, static compilation tends to inhibit program change. To take the extreme example of say C code which is directly linked change in a single function, if a single function increases in bytecode size so that it cannot be updated in place. In this case all code which makes use of it (worst case the rest of the program) must be rewritten to reflect the changed location of the changed function. While it is possible to build selective recompilation systems which can do intelligent and selective rebuilding (GHCI being an example of this), achieving full compilation performance at interactive development time is simply a waste of time on compilation when the goal of REPL driven development is to provide rapid feedback and enable exploration driven development and problem solving.

While vars are clearly inefficient from a perspective of minimizing function call costs, they are arguably optimal in terms of enabling this development style. Consider the change impact of altering a single definition on a Clojure instance using var indirection rather than static linking. There’s no need to compute, recompile and reload the subset of your program impacted by this single change. Rather the single changed form is recomputed, and the var(s) bound by the recompiled expression are altered. In doing so, no changes need be made to the live state of the rest of the program. When next client code is invoked the JVM’s fast path if any will be invalidated as the call target has changed, but this is handled silently by the runtime rather than being a language implementation detail. Furthermore var indirection means that compiling an arbitrary Clojure form is trivial. After handling all form local bindings (let forms and soforth), try to resolve the remaining expressions to vars in the mapped namespace, and to a class if no var is found. Brain dead even, but sufficient and highly effective in spite of its lack of sophistication.

While, as Oxcart is proof, it is possible to build a static compiler for some subset of Clojure, doing so produces not a static Clojure but a different language entirely because so much of the Clojure ecosystem and standard library even are defined in terms of dynamic redefinition, rebinding and dispatch. Consider for instance the multimethod system, often lauded with the claim that multimethod code is user extensible. Inspecting the macorexpand of a defmethod form you discover that its implementation far from being declarative is based on altering the root binding of the multimethod to install a new dispatch value. While it would be possible to statically compile this “feature” by simply collecting all defmethods over each multimethod and AOT computing the final dispatch table, however this is semantics breaking as it is strictly legal in Clojure to dynamically define additional multimethod entries just as it is legal to have an arbitrary do block containing defs.

So, in short, by trying to do serious static compilation you largely sacrifice REPL development and to do static compilation of Clojure you wind up defining another language which is declarative on defs, namespaces and dependencies rather than imperative as Clojure is. I happen to think that such a near-Clojure static language, call it Clojurescript on the JVM, would be very interesting and valuable but Clojure as implemented currently on the JVM is no such thing. This leads me to the relatively inescapable conclusion that building a static compiler for Clojure is putting the cart before the ox since the language simply was not designed to benefit from it.

Moving Forwards

Now. Does this mean we should write off tools.emitter.jvm, Clojure in Clojure and Oxcart? By no means. tools.emitter.jvm uses the same var binding structure and function interface that Clojure does. Moreover it’s a much nicer and more flexible single form level compiler that I happen to think represents a viable and more transparent replacement for Clojure.lang.Compiler.

So what’s that leave on the table? Open question. Clojure’s compiler has only taken major changes from two men: Rich and Stu. While there is merit in considered design and stability, this also means that effort towards cleaning up the core of Clojure itself and not directed at a hard fork or redesign of Clojure is probably wasted especially in the compiler.

Clearly while var elimination was a successful optimization due to the value Clojure derives from the Var system it’s not a generally applicable one. However it looks like Rich has dug the old invokeStatic code back out for Clojure 1.7 and the grapevine is making 10% noises, which is on par with what Oxcart seems to get for more reasonable inputs so we’ll see where that goes.

While immutable datastructures are an awesome abstraction, Intel, AMD and ARM have gotten very good at building machines capable of exploiting program locality for performance and this property is fundamentally incompatible with literal immutability. Transients can help mitigate Clojure’s memory woes, and compiler introduction of transients to improve memory performance could be interesting. Unfortunately again this is a whole program optimization which clashes with the previous statement of the power that Clojure derives from single form compilation.

Using core.typed to push type signatures down to the bytecode level would be interesting, except that since almost everything in Clojure is a boxed object and object checkcasts are cheap and this would probably result in little performance improvement unless the core datatypes were reworked to be type parametric. Also another whole program level transform requiring an Oxcart style whole program analysis system.

The most likely avenue of work is that I’ll start playing with a partial fork of Clojure which disassociates RT from the compiler, data_readers.clj, user.clj and Clojure/core.clj. While this will render Oxcart even more incompatible with core Clojure, it will also free Oxcart to emit Clojure.core as if it were any other normal Clojure code including tree shaking and escape the load time overhead of bootstrapping Clojure core entirely.

This coming Monday I’ll be giving a talk on Oxcart at the Austin TX Clojure meetup, so there’s definitely still more to come here regardless of the Clojure/Conj 14 accepted talk results.

Of Mages and Grimoires

2014-07-12T04:43:40+00:00

When I first got started with Clojure, I didn’t know (and it was a while before I was told) about the Clojure.repl toolkit which offers Clojure documentation access from within an nREPL instance. Coming from the Python community I assumed that Clojure, like Python, has excellent API documentation with examples that a total n00b like myself could leverage to bootstrap my way into simple competence.

While Clojure does indeed have web documentation hosted on GitHub’s Pages service, they are eclipsed in Google PageRank score if not in quality as well by a community built site owned by Zachary Kim: ClojureDocs. This wouldn’t be a real problem at all, were it not for the fact that ClojureDocs was last updated when Clojure 1.3 was released. In 2011.

While Clojure’s core documentation and core toolkits haven’t changed much since the 1.3 removal of Clojure.contrib.*, I have recently felt frustrated that newer features of Clojure such as the as->, and cond->> which I find very useful in my day to day Clojure hacking not only were impossible to search for on Google (being made of characters Google tends to ignore) but also didn’t have ClojureDocs pages due to being additions since 1.3. Long story short: I finally got hacked off enough to yak shave my own alternative.

**drumroll**

[Grimoire](http://conj.io/)

Mission

Grimoire seeks to do what ClojureDocs did, being provide community examples of Clojure’s core functions along with their source code and official documentation. However with Grimoire I hope to go farther than ClojureDocs did in a number of ways.

I would like to explore how I find and choose the functions I need, and try to optimize accessing Grimoire accordingly so that I can find the right spanner as quickly as possible. Part of this effort is the recent introduction of a modified Clojure Cheat Sheet (thanks to Andy Fingerhut & contributors) as Grimoire’s primary index.

Something else I would like to explore with Grimoire is automated analysis and link generation between examples. In ClojureDocs, (and I admit Grimoire as it stands) examples were static text analyzed for syntax before display to users. As part of my work on Oxcart, I had an idea for how to build a line information table from symbols to binding locations and I’d like to explore providing examples with inline links to the documentation of other functions used.

Finally I’d like to go beyond Clojure’s documentation. ClojureDocs and Grimoire at present only present the official documentation of Clojure functions. However some of Clojure’s behavior such as the type hierarchy is not always obvious.

< amalloy> ~colls

< Clojurebot> colls is http://www.brainonfire.net/files/seqs-and-colls/main.html

Frankly the choice of the word Grimoire is a nod in this direction… a joke as it were on Invoker’s fluff and the Archmagus like mastery which I see routinely exhibited on Freenode’s Clojure channel of the many ins and outs of the language while I struggle with basic stuff like “Why don’t we have Clojure.core/atom?” and “why is a Vector not seq? when it is Sequable and routinely used as such?”. “Why don’t we have Clojure.core/sequable?”?

Clojure’s core documentation doesn’t feature type signatures, even types against Clojure’s data interfaces. I personally find that I think to a large degree in terms of the types and structure of what I’m manipulating I find this burdensome. Many docstrings are simply wanting if even present. I think these are usability defects and would like to explore augmenting the “official” Clojure documentation. Andy Fingerhut’s thalia is another effort in this direction and one which I hope to explore integrating into Grimoire as non-invasively as possible.

Status

Much of what I have talked about here is work that needs to be done. The first version of Grimoire that I announced originally on the Clojure mailing list was a trivial hierarchical directory structure aimed at users who sorta kinda knew what they were looking for and where to find it because that’s all I personally need out of a documentation tool for the most part after two years of non-stop Clojure. Since then I’ve been delighted to welcome and incorporate criticism and changes from those who have found Grimoire similarly of day to day use, however I think it’s important to note that Grimoire is fundamentally user land tooling as is Leiningen and as is Thalia.

As such, I don’t expect that Grimoire will ever have any official truck with Rich’s sanctioned documentation. This and the hope that we may one day get better official docs mean that I don’t really foresee migrating Grimoire off of my personal hosting to a more “Clojure-ey” domain. Doing so would lend Grimoire an undue level of “official-ness”, forget the fact that I’m now rather attached to the name and that my browser goes to the right page with four keystrokes.

Future

As far as I am concerned, Grimoire is in a good place. It’s under my control, I have shall we say a side project as I’m chugging along on “day job” for GSoC and it seems judging by Google Analytics that some 300 other Clojureists have at least bothered to stop by and some 70 have found Grimoire useful enough to come back for more all in the three days that I’ve had analytics on. There is still basic UI work to be done, which isn’t surprising because I claim no skill or taste as a graphic designer, and there’s a lot of stuff I’d like to do in terms of improving upon the existing documentation.

Frankly I think it’s pretty crazy that I put about 20 hours of effort into something, threw it up on the internet and suddenly for the first time in my life I have honest to god users. From 38 countries. I should try this “front end” stuff more often.

So here’s hoping that Grimoire continues to grow, that other people find it useful, and that I manage to accomplish at least some of the above. While working on Oxcart. And summer classes. Yeah…..

Oxcart and Clojure

2014-06-26T20:18:45+00:00

Well, it’s a month into Google Summer of Code, and I still haven’t actually written anything about my project, better known as Oxcart beyond what little per function documentation I have written for Oxcart and the interview I did with Eric N.. So it’s time to fix that.

Motivation

Clojure isn’t “fast”, it’s simply “fast enough”. Rich, while really smart guy with awesome ideas isn’t a compilers research team and didn’t design Clojure with a laundry list of tricks that he wanted to be able to play in the compiler in mind. Instead in the beginning, Rich designed a language that he wanted to use to do work, built a naive compiler for it, confirmed that JVM JITs could run the resulting code sufficiently fast, and got on with actually building things.

The consequent of Rich’s priorities is that Clojure code is in fact fast enough to do just about whatever you want, but it could be faster. Clojure is often criticized for its slow startup time and its large memory footprint. Most of this footprint is not so much a consequent of fundamental limitations of Clojure as a language (some of it is but that’s for another time) as it is a consequent of how the existing Clojure compiler runtime pair operate together.

So Clojure wasn’t designed to have a sophisticated compiler, it doesn’t have such a compiler, and for some applications Clojure is slow compared to other equivalent languages as a result of not having these things. So for GSoC I proposed to build a prototype compiler which would attempt to build Clojure binaries tuned for performance and I got accepted.

Validating complaints

Okay, so I’ve made grand claims about the performance of Clojure, that it could be faster and soforth. What exactly do I find so distasteful in the language implementation?

Vars are the first and primary whipping boy. Vars, defined over here, are data structures which Clojure uses to represent bindings between symbols and values. These bindings, even when static at compile time, are interred at runtime in a thread shared global bindings table, and then thread local bindings tables contain “local” bindings which take precedence over global bindings. This is why (Clojure.core/alter-var-root!) and the other var manipulation functions in the Clojure standard library have the -! postfix used to annotate transactional memory mutation, because only a transaction can modify the root bindings of a thread shared and thread safe Var.

Now Vars are awesome because they are thread shared. This means that if you drop a REPL in a live program you can start re-defining Vars willy nilly and your program will “magically” update. Why does this work? Because Clojure programs never hold a reference to a “function”, instead they hold a thread synchronized var which names a function and get the latest function named by the var every time they have to call that function.

This is great because it enables the REPL driven development and debugging pattern upon which many people rely, however for the overwhelming majority of applications the final production form of a program will never redefine a Var. This means that consequently the nontrivial overhead of performing thread synchronization, fetching the function named by a var, checking that the fn is in fact Clojure.lang.IFn and then calling the function is wasted overhead that the reference Clojure implementation incurs every single function call. The worst part about this is that Var has a volatile root which poisons the JVM’s HotSpot JIT by providing a break point at function boundaries which the JIT can’t inline or analyze across.

IFn is another messy part of Clojure programs. The JVM does not have semantics for representing a pointer to a “function”. The result is that Clojure doesn’t really have “functions” when compiled to JVM bytecode, instead Clojure has classes providing .invoke() methods and instances of those classes, or more often Vars naming IFns may be passed around as values. This isn’t a bad thing for the most part, except that IFns use Vars to implement their .invoke() methods and vars are slow.

The real reason that this can be an issue is why do we need an IFn with multiple invoke methods? Because in Clojure functions can have multiple arities, and the single instance of an IFn could be invoked multiple ways where a JVM Method cannot be.

The real issue with IFns is that they exist at all. Every single class referenced by a JVM program must be read from disk, verified, loaded and compiled. This means that there is a load time cost to each individual class in a program. Clojure exacerbates this cost by not only generating a lot of small classes to implement IFns. When a namespace is required or otherwise loaded by a compiled Clojure program, the Clojure runtime loads foo__init.class, which creates instances of every IFn used to back a Var in the namespace foo and installs those definitions in the global var name table. Note that this loading is single threaded, so all synchronization at load time is wasteful. Also note that if there are top level forms like (println "I got loaded!") in a Clojure program those are evaluated when the namespace containing the form is loaded.

The sales pitch

So what’s the short version here? Clojure has Vars in order to enable a dynamic rebinding model of programming which deployed applications do not typically need. Because applications do not tend to use dynamic binding for application code, we can discard Vars and directly use the IFn classes to which the Vars refer. This could be a significant win just because it removes that volatile root on Vars that poisons the JIT.

This opens up more opportunities for playing tricks in the compiler, because we don’t really need IFn objects for the most part. Remember that IFn objects are only needed because methods aren’t first class on the JVM and to support dynamic redefinitions we need a first class value for Vars to point to. If all definitions are static, then we don’t need Vars, so we can find the fully qualified class and method that a given function invocation points to, freeing a Clojure compiler to do static method invocation. This should be a performance win as it allows the JVM JIT to escape type checking of the object on which the method is invoked and it allows the JIT to inline in the targeted method among other tricks.

If we can throw away IFns by implementing functions as say static methods on a namespace “class” rather than having seperate classes for each function, then we cut down on program size in terms of classes which should somewhat reduce the memory footprint of Clojure programs on disk and in memory in addition to reducing load time.

Oxcart

So what is Oxcart? Oxcart is a compiler for a subset of Clojure which seeks to implement exactly the performance hat tricks specified above and a few more. For the most part these are simple analysis operations with well defined limitations. In fact, most of what’s required to use Oxcart to compile Clojure code is already built and working in that Oxcart can currently rewrite Clojure programs to aid and execute the above transformations.

Oxcart is also a huge exercise in the infrastructure around the Clojure.tools.analyzer contrib library, as Oxcart is the first full up compiler to use tools.analyzer, tools.analyzer.jvm and tools.emitter.jvm as more than the direct compilation pipeline which tools.emitter.jvm implements. This means that Oxcart has interesting representational issues in how passes and analysis data are handled and shared between passes, let alone how the data structure describing the entire program is built and the performance limitations of various possible representations thereof.

So what’s Oxcart good for? right now: nothing. Oxcart doesn’t have an AOT file emitter yet, and relies on tools.emitter.jvm for eval and as such is no faster for evaling code in a live JVM than Clojure is. At present I’m working on building an AOT emitter which will enable me to start doing code generation and profiling Oxcart against Clojure. I hope to post an initial emitter and a trivial benchmark comparing a pair of mutually recursive math functions between Clojure and Oxcart.

Before you go

I’ve know I’ve said this entire time that Oxcart is a Clojure compiler. That’s a misnomer. Oxcart doesn’t compile Clojure and never will. Clojure has stuff like eval, eval-string, resolve, load-string, load and all the bindings stuff that allow Clojure programmers to reach around the compiler’s back and change bindings and definitions at runtime. These structures are not and never will be supported by Oxcart because supporting them would require disabling optimizations. Oxcart also doesn’t support non-def forms at the top level. Oxcart programs are considered to be a set of defs and an entry point. Oxcart also assumes that definitions are single. Redefining a var is entirely unsupported, abet not yet a source of warnings.

Some of these differences are sufficiently extreme that I’m honestly on the fence about whether Oxcart is really Clojure or some yet undefined “Oxlang” more in the style of Shen, but for now I’ll stick to building a prototype :D

Typing the processor

2013-11-13T00:00:00+00:00

Previously I implemented a pipelined version of a simple processor for my batbridge assembly architecture. This time, I’m going to take that architecture a step further by using the Clojure.core.typed library by @ambrosebs to get some level of type safety in preparation for a binary assembler.

Introducing typed Clojure

Clojure is at its core a typed language because the runtime that Clojure sits upon (the Java JVM) is a type based system. However by a large the programming model offered to Clojure programmers is essentially type agnostic. For the most part, Clojure programmers are free to deal in terms of abstract sequence and map types without real regard for the true Java class used to represent their data at runtime.

This is a Good Thing. It reduces the intellectual load on programmers in so much as programmers are free to leave the implementation details of their algorithm to the Clojure compiler. Unfortunately, types and their implications for data operations are also more or less central to the way that programmers solve problems. In short without some notion of types we are bound hand and foot.

Typed Clojure as a library seeks to implement an optional annotative type system for Clojure. Rather than enforce strict typing, Typed Clojure seeks to make types a tool by which to ease evolving Clojure programs and an aid to program comprehension rather than a set of implicit or comment noted assumptions.

A personal example: This time last year I was tasked with writing an out of order processor simulator for class (gee does that sound familiar). Everything was going great until I discovered that I needed to add metadata to the instructions in my simulator. Lots of metadata. Now at the time, I didn’t know about Clojure.core/meta and Clojure.core/with-meta which make it possible to attach metadata to existing objects. Rather I began the painful process of reworking my simulator days before it was due from using a vector based representation of an instruction to using a map. The results were unfortunate, in so much as Maps it turns out are ISeq. This means that the sequence operators which expected a Vector almost worked on maps making it a serious pain for me to debug these type mismatches. As I said in my first post, the simulator never quite worked.

Typed Clojure helps resolve this class of problem by offering inference backed annotative typing. This means that you describe through annotations what the structure of your intended types are, and then a type inference engine attempts to verify that your program is correct with respect to the described data structures. So in the case of my map vs. vector instruction representation problem, Typed Clojure checking would have made it obvious what code had not yet been rewritten to operate on the instruction map rather than the previous instruction vector structure.

Diving in

As of this writing, Typed Clojure is at version [org.Clojure/core.typed "0.2.17"] and it is against this version that I will be typing my current BatBridge simulator.

Typed Clojure offers a number of built in types: Seqs, Sets, Maps, Integers, the list goes on and on. So just for grins, lets do a simple demo.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
(ns demo
	(:require [Clojure.core.typed :as t :refer [defn>]]))

(t/ann three t/AnyInteger)
(def three 3)

(t/ann ^:no-check Clojure.core/zero? [t/AnyInteger -> boolean])
(t/ann myzero? [t/AnyInteger -> boolean])
(defn myzero? [n]
	(zero? n))

(defn> add-two :- t/AnyInteger
	([x :- t/AnyInteger]
	    (+ x 2))
		
	([x :- t/Anyinteger
      & more :- (t/Seq t/AnyInteger)]
        (+ (apply + x more) 2)))

Type Clojure uses the ann form to annotate a symbol with type(s). Here, I use the ann form to indicate that the symbol three is of type AnyInteger. I use AnyInteger rather than integer or long because Clojure typecasts between long and integer under the covers, and AnyInteger wraps both of them.

I then defined a function, myzero?, which simply wraps Clojure.core/zero? annotating both my function and Clojure.core/zero? with the appropriate types. For functions, the ann notation is [param-type + -> ret-type]. Here, I define both zero functions to be [t/AnyInteger -> boolean]. This works as expected, but when we run the typechecker on this namespace we will see that the annotation of Clojure.core/zero? warns that it isn’t being checked. This is because I used the ^:no-check reader metadata when I annotated the symbol Clojure.core/zero?. When the Typed Clojure type checker encounters ^:no-check flagged symbols no effort is made to process the definition of the annotated symbol and ensure that the annotative type is correct with respect to the runtime type.

Finally I use the new defn> syntax, to achieve the same sort of effect as my previous defn/ann pair. In fact, if you inspect the defn> macro, you will discover that it simply generates the defn/ann pair. However it provides nice syntactic sugar so I elected to demonstrate it.

Checking types

There are three main entry points I use to typed Clojure: Clojure.core.typed/cf, Clojure.core.typed/check-ns and lein-typed.

Clojure.core.typed/cf (short for Check Form) is a function of two arguments, the first being an expression for which some type is claimed and the second being a type. cf engages the Typed Clojure inference engine and attempts to verify that the typed form is indeed of the type claimed. For example, if we attempt to typecheck one of the example forms above…

Introducing Seqs, Maps and Unions

Typed Clojure allows for considerably more variety in annotation than simple types like integers and booleans. Sets and Maps are fully supported, through the use of the forms Clojure.core.typed/Set and Clojure.core.typed/Map.

As one would imagine, Clojure.core.typed/Set is a function of one argument, which creates a type representing a set of elements which all share the argument type.

Clojure.core.typed/Map is similar, in that it is a function of two arguments, one for the key type and one for the value type.

Clojure.core.typed/Seq describes an arbitrary sequence of elements sharing a type, and Clojure.core.type/Vec get the same place but is specific that the sequence involved is a Vector.

However we’re still in the realm of relatively simple types as far as Clojure is concerned. Much of Clojure is about mixing and matching different types in and out to create variable type structures and other kinds of weirdness. For this we need three more forms:

Introducing U, HMap and HVec

U is the Typed Clojure function for expressing the union of several types. That’s all it’s good for, but that makes it immensely powerful. Need to represent a value which could be either Long or Integer? (U Long Integer). Yes, it’s that easy.

While still experimental, HMap is for me where the rubber meets the road in terms of Typed Clojure’s usefulness. Heterogeneous maps, maps where different special keys map to differently typed values, are a core design pattern of the Clojure community. As described over on the core.typed wiki, HMaps can represent required keys, the types thereof, optional keys and the types thereof, and even the absence of certain keys! HVec is the same idea as HMap, except that it’s used to represent Vectors rather than Maps.

Applying this to BatBridge

Okay. Well let’s start defining some of the types that are integral to the BatBridge simulator then…

(t/def-alias Memory "A storage medium within a processor"
  (t/Map t/AnyInteger t/AnyInteger))

(t/def-alias Storage
  (U (Value :registers) (Value :memory)))

(t/def-alias InstructionParam "An instruction parameter"
  (U t/AnyInteger
     (Value :r_PC)
     (Value :r_IMM)
     (Value :r_ZERO)))

okay, cool… this lets us represent all the legal values which the simulator could see as storage locations within the VM, and values that we can interpret to put in those storage locations. Now let’s try and tackle the VM state itself… we’re gonna need an HMap with a bunch of mandatory keys…

(t/def-alias Processor "A processor state"
  (HMap :mandatory {:registers Memory
                    :memory    Memory
                    :fetch   InstructionVec
                    :decode  InstructionMap
                    :execute CommitCommand
                    :halted  boolean}
        :complete? true)) ; indicates that anything else makes the Processor illegal

But what about the instructions themselves? We need three real instruction encodings: the vector instructions which I use for my assembly notation, the map instructions which execute operates on and which decode builds, and the commit commands that writeback uses.

(t/def-alias InstructionVec "A symbolic vector instruction"
  '[t/Keyword InstructionParam InstructionParam InstructionParam t/AnyInteger])

(t/def-alias InstructionMap "A parsed map instruction"
  (HMap :mandatory {:icode t/Keyword
                    :dst   InstructionParam
                    :srca  InstructionParam
                    :srcb  InstructionParam
                    :lit   t/AnyInteger}
        :complete? true))

(t/def-alias CommitCommand
  (U '[(Value :memory)   t/AnyInteger t/AnyInteger]
     '[(Value :registers) t/AnyInteger t/AnyInteger]
     (Value :halt)))

Now, armed with these type annotations, it’s more or less trivial to chase through the simulator I originally presented adding the appropriate annotations. However, all this comes to a head when you try and type the processor’s memory because there is a fundamental type conflict in both of my processors to date: both InstructionVecs and AnyIntegers are legal values within the :memory of a Processor. This means that the type system cannot be sure (and nor can we) that the value fetched from some address is in fact an instruction not a value.

There are two ways to resolve this conflict. The approach which I’ve taken thus far if you’ve read the Typed Clojure code over on github is to sweep this conflict under the rug by creating a pair of ^:no-check functions one of which reads memory returning instructions, the other of which reads memory returning values. However the real fix to this is to introduce a binary (numeric) encoding for instructions. I hope that this constitutes a helpful introduction to Typed Clojure, next time I will at long last define an encoding for BatBridge and present the full architecture complete with a bytecode assembler toolkit.

f : ⊥ x ⊥ → ⊥ | Clojure

Shelving; Building a Datalog for fun! and profit?

Databases?

Relational data

Operating with Tuples

cartesian product

projection (select keys)

selection

Joins

join (R⋈S)

semijoin

Enter Datalog

Extensions

Recursive rules!

Negation!

Incremental queries / differentiability!

Eventual consistency / distributed data storage!

The Yak

95 Theses

Grimoire

Grenada

cljdoc

Prior art

Design goals

Building a Datalog

Storage models!

Schemas!

Writing!

Schema migrations!

Query parsing!

Query planning!

Goodies

API management!

Documentation generation!

More fun still to be had

Negation!

Recursive rules!

More backends!

Transactions!

Ergonomics!

Actually replacing Grimoire…

Docs Sketches

Intermediate Abstraction

Table of Contents

1 Preface

2 Context

3 Complexity

4 Abstraction

4.1 Examples of Abstraction

4.2 Limits of Abstractions

4.3 Inheritance, Composition & Re-use

4.4 Naming

4.5 Achieving Abstraction

4.6 Utility Is Contextual

5 Performance

6 Fin: Purism

Immutable Env Things

Deprecation warnings in Jaunt

Jaunt - A friendly Clojure fork

TL;DR

The Long Version

And Yet I Shave

Waxing Political

A Fork in the Road

Demo

No Promises

Hacking like it's 2288

Strictly Tagged Clojure

Some motivation

The Hack

Disclaimer

What Didn’t (seem) To Work

Inline Unrolling

Next Steps

Toothpick: A theory of bytecode machines

A Theory of Bytecode Machines

An Assembler Generator

Talk is cheap, show me the code

Oxcart going forwards

Oxcart and Clojure