f : ⊥ x ⊥ → ⊥ | Oxlang

牛: How static is enough?

2017-11-02T02:11:13+00:00

Ox has been and will continue to be a long-haul project in part because it’s a dumping ground of ideas for me, and in part because I don’t desperately need it. There’s plenty of bad in the software industry, but I’m able to swing the hammers that already exist. Ox isn’t so much a hammer as it is an exercise in the ongoing search for what a better hammer would be.

Part of this search comes from my judgments of the tools I use on the daily and their failures. The most significant part however is how the decisions I’ve made in what prototypes I have built have shaken out.

It would have been easy enough to crank out an RⁿRS interpreter and call it Ox. It just wouldn’t have added anything to the software ecosystem or solved any problems I actually had. In fact most of the problems I’ve tried in one way or another to approach in Ox are directed by my programming experience.

So lets start with the motivation and what I was trying to get at in the last cut I took at Ox.

Clojure is a nice language which does a good job at trying to provide sane defaults. Its data structures are immutable by default, it features destructuring - macro sugar for binding values in structures to locals, it interoperates nicely with the JVM and it has a namespace system which makes code organization fairly predictable.

Because Clojure uses a singe pass compiler and its compilation unit is a single form rather than a file/module or group thereof, it can be difficult to Clojure whose module structure reflects its logical structure. Cyclic dependencies between modules/namespaces are difficult to achieve and have poor ergonomics. In fact, cyclic dependencies within a single namespace are - if not difficult to achieve - then at least requires an effort which tends to prohibit their use.

This has a couple of consequences, well exemplified by the language implementation itself. The namespace clojure.core is a huge namespace containing many groups of related functions which would do just as well if not better in their own namespaces. Big namespaces which don’t have a unifying concept are difficult to approach as a new user and difficult to document. Furthermore, they create artificial contention over the “good” names, and tend to lead to the use of prefixes.

Perhaps most interesting is the consequences the lack of circular dependency support has for the language’s own implementation. The first several thousand lines of clojure.core serve to bootstrap the language using mainly JVM inter-operation to Java interfaces and the Java implementation of Clojure’s various built-ins. Some of this bootstrap code is unavoidable, but parts of it could be avoided if the implementation was able to make forward use of symbols which weren’t defined yet.

With explicit forward declarations in some cases this could be made possible today within a single namespace such as clojure.core. The most interesting use cases of circular dependencies however require circular dependencies across module boundaries. For instance what if we could have a clojure.core.seqs namespace containing all the operations related to the sequence abstraction, and so-forth. A large number of small namespaces would be more modular, easier to break out of the standard library into library packages where possible.

And after all, Ox is a pie in the sky project so what would it take?

Well - circular dependencies on functions are pretty easy. They don’t have a data dependency on each-other at compile time; okay they do but letrec is easy to write and rewriting modules as letrec to support mutual function recursion is well understood and a traditional Scheme trick.

For instance if I had the nonsensical module

(define (f a b)
  (c a))

(define (g a b)
  (c b))

(define (c x)
  (+ x 1))

This could be implemented instead by re-writing it to the letrec form

(letrec ((f (lambda (a b) (c a)))
         (g (lambda (a b) (c b)))
         (c (lambda (x) (+ x 1))))
  ...)

Implementing circular dependencies Scheme style via letrec works because the top level special forms (define etc.) are well understood and can be re-written into letrec bindings as above.

The idea I had at the time was that REALLY the core problem here is that the evaluator can’t figure out the dependency graph between functions and macros because we have an indeterminable number of un-expanded macros. A solution to this problem is to simply make the dependency graph statically visible.

In scheme, the above letrec conversion is trivial because the top level form define is well understood by the implementation and so can be rewritten into a form where circular references will function as desired. The take on this in #29 is to do so by restricting top level forms to a small set of forms which explicitly state what definitions they provide. For instance def, defn, deftype and soforth all explicitly name the definitions they create.

(ns example)

;; This provides the definition example/foo. `def` is well know, so
;; this form can be understood as a definition at read time without
;; any effort to evaluate the body or docstring just like `define`.
(def foo
  "Some docs"
  1)

;; Same as above, `defn` is well known and can be statically parsed
;; at least to the degree we need to understand that this defines
;; the symbol example/qux for dependency analysis
(defn qux
  "Some docs"
  ([x] "arity 1")
  ([x y] "arity 2")
  ([x y z & more] "arity 3+"))

;; `defmacro` is just `defn` with the macro flag and some tweaks to
;; accept implicit arguments like `&form` and `&env`.
(defmacro deftype
  ;; ...
  )

;; deftype, definterface and friends all nicely fit this pattern to
;; the extent one may choose to implement them as special forms.

This all works pretty well. The problem is what happens when a macro occurs at the top level? Doing macros this way doesn’t work, especially when there are sequential dependencies between the macros and the functions in the module. While the macros can be re-written as letrec bindings, the semantics of macro expansion don’t fit that way because the macros have to be expanded before the functions or (or the other macros!) using them become executable.

Consider something like this…

(deftype Foo "A foo record." [a b c])
;; We now have the functions `->Foo`, `Foo?` and some friends

(defn foo-or-bar [a]
  (or (Foo? a)   ;; Where does `Foo?` come from? Nothing at the top
                 ;; level trivially provides `Foo?`, but we have to
                 ;; find it in order to compile `foo-or-bar`.
      (Bar? a)))

I thought about this for a long time because I think there’s a clear case for supporting mutually recursive definitions with the most generality possible. And the first solution I came up with is the one implemented in ox-lang/ox#29.

By defining a new top-level special form providing, users can just annotate their top level macros with the symbols they’re expected to produce! Problem solved! This lets Ox determine what top level forms need to be evaluated to see of they provide macros and perform lazy loading by doing functionally free dependency analysis before any evaluation.

So for instance

(providing [->Foo, Foo?]
  (deftype Foo "A foo record." [a b c]))

(defn foo-or-bar [a]
  (or (Foo? a) ;; The reader understands `providing` and can determine
               ;; that to get the needed `Foo?` definition the
               ;; previous form needs to be evaluated.
      (Bar? a)))

The problem with this approach - and the reason I call it a mistake - is that it greatly complicates load time and neuters definition writing macros. As noted earlier, there’s plenty of good reason to want to write macros that write definitions. However a good bit of the syntactic power of such macros is lost if you have to provide a list by hand of definitions the macros will create for you.

Frankly this made implementing code loading so complicated that what with trying to bootstrap Ox in Java I couldn’t do it. This decision proved absolutely fatal to that particular run at the Ox project.

There are, now that I’ve seen the failures of this approach and had some hammock time, other ways to achieve this same end. For instance if you don’t allow macros to write macros, you can play games where macro definitions are letrec rewritten as above, and the definitions of “normal” functions or expressions are delayed until forced by a macro expansion. This lets you play games whereby functions and macros may mutually recur so long as you can ground out a macroexpansion to some concrete form or function of forms that can be evaluated without unbounded mutual recursion between functions and macros. Which, honestly, is the best I think you can do.

The advantage of Clojure’s single pass approach here is a huge complexity win which avoids all these considerations, and the fact that the Clojure read/evaluate/print loop has exactly the same structure as code loading.

Anyway. Languages are hard. That’s what makes them fun.

More when I’ve got better ideas.

Immutable Env Things

2016-06-13T00:00:00+00:00

As with some of my other posts, this was originally an email which turned into enough of a screed I thought it would be of general interest and worth posting. Some op/ed material has been removed, but the technical details are unaltered if not clarified.

So here’s the rundown I’ve got so far on a lisp with immutable environments. I’m merely gonna sketch at some stuff, because giving a full treatment would require dusting off and finishing a bunch of stuff I came up with for Ox and put down again.

I also threw out the Haskell sketch I was dinking with because it was just distracting from writing this :P

So lets start from the top.

Clojure is a form-at-a-time language which happens to support reading files of forms (or rather Readers which produce textual representations of forms, where they come from is magical).

Binding is achieved through a global, thread shared, transactional, mutable mapping from Symbols to Namespaces, which are themselves transactional mappings from Symbols to Vars, which are themselves a triple (QualifiedSymbol, Metadata, Binding). Vars happen to also support dynamic binding (pushing and popping), but this is less used. From here on out I’ll treat them simply as a global mostly static binding mechanism, which is their primary use case anyway.

Control and local bindings are achieved entirely using lambda/fn forms compiled to JVM methods, produced by the opaque compiler as opaque IFn/AFn objects. Top level forms are compiled and executed for effect by being wrapped in an anonymous (fn [] <form>) which can be blindly invoked by the compiler to realize whatever effects the form may have.

Circular dependencies are achieved via Var indirection. A Var is first created with no binding, so that it can be referenced. With that environment binding, code which will depend on (call) the Var’s final bound value can be compiled since it just needs the target Var to exist in order to compile. The depended on Var may then be redefined, having both itself and its dependent visible in the compilation context and so the two Var bindings can be mutually recursive.

While traditional and lispy, this approach has a number of problems which I’m sure I don’t need to reiterate here. The goals of an immutable namespace system then should be to

Make Namespaces the compilation unit rather than Forms
Make Namespaces reloadable (change imports/exports/aliases w/o system reboot)
Enable Namespace compilation to fail sanely
Enable refactoring/analysis

The sketch of this I’ve been playing with is as follows:

Lets assume an interpreter. If you can interpret you can compile as an implementation detail of eval and interpretation is easier to implement initially. Assume a reader. So we read a form, then we have to evaluate it. Evaluation is first macroexpansion, then sequential evaluation of each resulting form in the environment. Evaluation at the top level is eval Env Form -> Env (we discard the result) which is fine. Because we aren’t really doing compilation, only interpretation, Clojure’s tactic of having Vars and analyzing against Var bindings works 1:1. Create an unbound symbol, analyze/bind against it, then add a binding later.

The (global) environment structure I had in mind is pretty simple: one unqualified symbol names the current namespace (*ns*), a mapping exists from unqualified symbols to Namespaces.

Namespaces are essentially compilation contexts, and really just serve to immutably store the alias mapping, import mapping, the mapping of the Vars in the namespace to bindings, and a list of the public Vars/exports from the namespace.

Vars are a fully qualified name, metadata and a value. That’s it.

Compilation occurs at the namespace level. All the forms in a namespace are sequentially read and evaluated in interpretation mode. All macros expand normally and type hints are processed they just don’t do anything.

Once the whole namespace has been successfully loaded in interpretation mode, a second pass may be made in which static linking is performed. Because everything in the namespace has been fully realized already it’s possible to statically link even mutually recursive calls within the same namespace. Full type inference within the namespace is also possible here, etc.

Aside: Interestingly because it is single pass, the existing Clojure compiler doesn’t (can’t) handle hinted mutually recursive function calls at all. It relies on Var metadata to store arglists (and hints for primitive invocations), so in order to emit a hinted recursive call the forward declaration of the var has to carry the same ^:arglists metadata (hints and all!) which will be added by defn whenever it is finally defined.

Once a namespace has been fully loaded and compiled (including the second pass) the global environment with the resulting namespace and var mappings is the result of whatever caused loading. If an error occurs during compilation, we just abort and return the environment state before anything else happened. I think there are tricks to be played with generated garbage class names and classloader isolation here so that this works even during the 2nd static compile pass, but it should be possible to fully clean up a failed compile.

So this works really great for module loading, and even for macros which expand into multiple def forms. This model of eval :: env -> form -> (env, result) starts to break down when you want to talk about macros that do evaluation during code loading, and likewise the REPL. Basically your interpreter winds up holding an additional reference, *env* or something, which is intrinsically mutable, but which references an immutable environment and holds shall we say the state continuation of the entire REPL. Consider user code which actually calls eval during execution. Whatever state changes that evaluation creates need to be preserved in the “global” continuation.

In this model when you compile a form at the REPL, the runtime could obviously interpret (this may be appropriate for macroexpansion) and can also compile. This is only tricky when the user recompiles a symbol which was already bound/compiled. In this case, the runtime could either eagerly recompile all the code which links against this function using the same interpret then statically link tactic or could just invalidate any compiled bytecodes and lazily recompile later. The former is probably more predictable.

Once a namespace has been reloaded/altered, all namespaces importing it must also be recompiled in topsort order using the same tactics. That we already have per-symbol dependency information and per-module dependency information helps with building refactoring tools which otherwise have to derive all this themselves. Ideally top level expressions/definitions would also expose local variable tables tables and local variable dataflow graphs so that analysis there is also possible.

Aside: It may be possible to allow cyclic dependencies between namespaces (see submodules in racket for some study on this problem). In the common case it may well be that macros are well behaved and that cyclic dependencies between modules work out just fine. However because macros perform arbitrary computation it’s also pretty easy to cook up pathological cases where macro generated code never reaches a fixed point in repeated interpretive analysis which can be statically compiled. For this reason I’m inclined towards formalizing a ns or module special form and throwing out require, refer, import and friends as legal top level forms altogether. Namespaces should be declarative as in Java/Haskell.

There’s probably a way which I just haven’t seen yet to treat updates to the “global” namespace mapping and updates within a namespace the same way via the same mechanism since they’re both dependent on the same dependency tracking and recompile machinery.

Writing this has made me think about having an ImmutableClassLoader which is derived entirely from a immutable environment and loads classes by lazily (statically) compiling forms.

That’s kinda all I got. ztellman gets credit for the mutable *env* in the repl bit which I spent literally weeks trying to do without last year. Maybe something with monad transformers can get you there but I couldn’t figure it out. mikera has some sketches of all this with types in his KISS project.

I’ve already prototyped an environment system that looks a lot like this in the Ox repo if you want to go dig. Originally there was this ox/lang/environment.clj then I started dinking with a Java implementation of the environment structure ox/lang/environment which also has a couple different stack based contextual binding types for the interpreter.

As written about in the Jaunt essays, I’ve kinda concluded that whatever this immutable ns language winds up looking like is so divorced from what Clojure or at least the way that most people write Clojure that you’ll loose so much value in the changeover it won’t pay off for a very long time. The mount library is a perfect example of this in that it’s deeply imperative and takes advantage of code loading order to achieve start and stop. It’s not the only bit of code I’ve seen which does effects at load time either. Hence Jaunt which tries to be less flawed around namespace reloading/refactoring without going whole hog on ns immutability.

The more I kick this stuff around the more I find that the whole endeavor verges on a Haskell clone in sexprs and the more I decide I’d rather take the time to learn Haskell properly than mess with an immutable lisp or Jaunt. Besides then you get nice stuff like typeclasses and fully typed instance dispatch and equality that actually works and theorems and native bytecode and yeah.

Jaunt - A friendly Clojure fork

2016-02-22T13:00:00+00:00

TL;DR

I’ve started a fork of Clojure which I intend to operate in a community directed manner; Jaunt. Check out the demo script or just keep reading for more.

The Long Version

After several months of dinking with Oxlang on and off I came to a realization. Ox, while an annoyingly persistent daydream was doomed to be one of two things. Either it would be something akin to Haskell in S expressions, or it would be indistinguishable from Clojure with some tweaks under the hood.

I’d gotten Ox’s reader sort of working, and was starting to experiment with notations, explore notions of higher kinded types and generally was having a ball. However even getting that far was a highly educational exercise in how much Clojure already does fairly well. To continue down the road of Ox was to admit that I’d have to reinvent all of Clojure’s datastructures, most of the Clojure compiler and a whole boatload of the standard library. It was also to admit that due to limitations of the JVM as a target, I wouldn’t be able to play some of the many tricks which the GHC folks get to play even were I to go down the road of a pure language.

Why do all this? Because it’s hard? I’m no Heracles seeking labors or yaks for their own sake. I’d rather be learning something new or playing Dota or… a thousand other things.

YOUR FINEST YAK SIR
— tail -f /dev/ardm0 (@arrdem) February 13, 2016

I swear I would.

Besides, even though Michiel Trimpe was kind enough to mention Ox alongside a number of other “conversational” programming languages in his ClojureX talk, I’m not convinced that it’s a good idea.

In looking back at that particular talk, one thing struck me. Sure in a pure functional programming language such as Haskell you could stick an arbitrary AST in a Merkle tree and have a globally distributed version control and distribution system with strong guarantees about tree uniqueness and reuse safety.

Unfortunately, Clojure is no such beast. We have no language level concept of effects (okay we have the io! macro but that’s hardly the same). While you could implement immutable namespaces (and in fact they are backed by immutable maps already) the concept doesn’t make a whole lot of sense to me.

The very notion of a conversational language is that you can reference any definition from any version of the program as if it were the current version. What does this look like at a REPL? Are namespaces (or rather full states) identified with a commit ID and compilation occurs in terms of a fully qualified “commit” and Var? Now we need notation for that. Maybe version/name/namespace or something. Some sort of explorer to browse older commits is in order… the list goes on and on. In short, I think you wind up reinventing a bunch of stuff that git already does pretty well.

This is not to say that Clojure’s existing mutable namespaces are ideal. For me, they work against the ns or file level code loading style I use when developing Clojure code. Old habits from C die hard as it were. A common failure mode I encounter is that I’ll rename a function but not all of its uses. IntelliJ and refactor-nrepl can be of some help here, but I’ve had them break down and I’m not always in a “full ide” configuration. Because namespaces are mutable, even if I were to perform a rename correctly, the old name is still present in the namespace. If I got it wrong, my code reloads, all symbols resolved then I get Really Cool Unexpected Behavior when parts of my code call the new function and parts call the old function. Or worse my code works fine until I restart my repl or the test server and everything explodes. Or the old name just sticks around cluttering up my autocompletion.

This becomes a real UX problem for me, because I rely heavily on refactoring. I start by sketching out what amount to types, or deriving operations from types I already have, and keep working renaming and moving code around until I get something I’m happy with.

@arrdem your problem is all the refactoring. Why isn't the code right from the start?
— Chas Emerick (@cemerick) December 23, 2015

And Yet I Shave

The thing which came to me now a few weeks ago is that there’s an interesting halfway house between these two extremes of mutable and immutable namespaces. What if we gave Namespaces and Vars version numbers? When a Namespace is re-defined (the ns form is evaluated) then the Namespace’s version is incremented. When a def is evaluated, the version of the Var bound by the def is set to be the version of the Namespace in which that Var exists.

This allows traditional REPL style interaction with Namespaces. You may enter a Namespace, add bindings, aliases, imports and so forth as you will. However the version numbers give you important static information about the state of the Var in its Namespace.

If I have some def which I entered at the REPL, when I do something to ns backing file and reload it, that def persists. But, its version is now out of sync with the Namespace it exists within. When we reloaded the file, the Namespace’s version was incremented and then every definition was reevaluated. If my code is still using an old symbol which is not textually present in the file and was was not reloaded, its version doesn’t increase. Now the compiler can detect the version disparity and emit warnings that I’m using old code.

While this solves dependency freshness issues within a single def and a single Namespace, solving them globally requires more leg work. The compiler needs to be adapted to track the use and reach sets of each compiled expression, and to inter that information into the Var binding forms so that it’s possible for a newly compiled form to emit warnings about transitive use of stale vars.

These changes don’t alter the semantics of the Namespace at all outside of reloading. If you evaluate a new def or a single form in a Namespace it works just as before. However if you take this whole file reloading approach, the language can offer added support which papers over a rather painful pitfall.

A related change is that Namespaces can be made to purge their imports and aliases on reload. Right now, Namespaces persist imports and aliases until the VM is restarted. This leads to a nasty set of refactoring problems where you want to change the name of an alias, but can’t reuse a name because aliases are never cleared. If we consider a whole Namespace and its source file(s) as the unit of compilation we can again do better. This is another language level change which would meaningfully improve my day to day use of the language without breaking existing use patterns.

Both of these changes turn out to be quite simple to implement. In the course of only a few days I went from the concept of these changes to a build of Clojure (pr, build) which features Namespace clearing and Var versions. I’m using now daily because it better supports the way that I want to work with the language.

Waxing Political

Rich, Stuart Sierra, Stu Halloway, Alex and the rest of the Clojure/Core crew (see Clojure/Huh?) built Clojure and have shepherded it this far. I’m immensely grateful to them for the work they’ve done to date in building a tool which I’ve had great fun using. But I think our priorities diverge.

From what I’ve been able to gather by talking to people who’ve been around longer and what Rich has written, it should be clear that Clojure is not a community project in the same way Python or Rust is. Clojure is Rich’s personal project, which he is so kind as to share with the rest of us and in in its refinement Rich chooses to accept some amount of input from us as users. I’m somewhat ashamed to admit that it took me two years to realize this.

Open-source development problem in one sentence:
"Users assume they are customers"
— Aleksey Shipilëv (@shipilev) January 20, 2016

Discussion of particulars, and the reasons for the status quo is I think a somewhat useless exercise. Rich has his project and administers it as he sees fit. As Clojure is Rich’s project, the priority of its governance is to conserve Rich’s time and it has repeatedly been made clear that there is no desire to change this. This is an eminently reasonable goal I can find no fault in. Rich doesn’t owe me or anyone else jack, let alone time. However the consequence of this structure and of not delegating is the slow and steady path which frustrates my desires for more rapid iteration.

A Fork in the Road

With loyalty the path of the last 18 months and clearly one of frustration, and voice ineffective due to the (reasonable!) priorities of Clojure/Core, The only course of action left to me is some form of exit.

I think that there’s a lot of possible improvements that can still be made to the language. Currently there doesn’t seem to be a lot of appetite for exploring that in Clojure itself. The namespace reloading stuff sketched above is just the first thing that popped to mind when I started thinking about how to improve my day to day experience and implementing that alone has shaken loose a number of other small but worthy improvements.

The consequence of this opinion is that I’ve started Jaunt.

Jaunt is my personal fork of Clojure which seeks to deal with the various organizational and technical limitations expressed above.

Technically, with Jaunt I want to explore a space of refinements to the language such as the Namespace reloading changes sketched above which add value without breaking compatibility with the ecosystem of libraries and tools like CIDER which I’ve come to rely on every day.

Organizationally, I want Jaunt to be a community driven affair. I don’t claim that my judgment is infallible, nor do I think I’ll have all the good ideas, nor will I have infinite time to consider changes even were these restrictions lifted. I’d love to involve contributors who want to bring improvements to the project within the stated bounds of preserving compatibility at least with the documented parts of Clojure.

Demo

Because otherwise how do you believe that any of this works, here’s a script that’ll build a empty leiningen project in a temporary directory and run a quick demo of some of the stale Var stuff.

No Promises

If you want a stable language. If you’re running code in production. If you don’t want to get down and dirty with the language implementation or the development process, stick with Clojure.

As the EPL states:

THE PROGRAM IS PROVIDED ON AN “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED INCLUDING, WITHOUT LIMITATION, ANY WARRANTIES OR CONDITIONS OF … FITNESS FOR A PARTICULAR PURPOSE.

Jaunt should be a drop in replacement for Clojure. I want to keep it compatible. I foresee no reason to break the documented Clojure API. But some amount of drift is inevitable. I’ve already moved some stuff around inside the Java implementation. If you were depending on those undocumented yet public implementation details, all bets are off. I’ve changed how metadata behaves on vars (I think it’s a better approach but it does impose behavioral differences). I’ve added deprecated warnings. I added clojure.core/*line* and clojure.core/*column* (pr) which the compiler totally had internally and just didn’t expose forcing weirdness. And this list of differences (perhaps improvements) is just going to grow.

Whether Jaunt proves to be something I depend on in the future, whether I wander off into the land of types and leave it to someone else or whether it diverges too much from Clojure and dies as a result of the weight of its mutations, I’ve overheard enough interest in a project such as this that I think this is a worthwhile endeavor and I hope you’ll join me in this little adventure.

Update: Clojarr has been renamed to Jaunt, updated article accordingly.

牛: Some motivation

2015-01-27T05:41:54+00:00

Here is some of my motivation for Oxlang. I’m sorry it isn’t shorter, I gave up on copy editing to turn in. This was originally a personal email and appears almost unaltered.

Clojure is awesome. It is grammatically simple with a classical lispy syntax, and its cheap immutable data structures enable equational reasoning about most problems and solution tools.

My top bugbears & goals:

Contributor culture. The contribution process should be as open, friendly and low friction as possible see Guy Steele’s comments, on growing a better language from one with warts through acceptance of contributions. The GHC team has a “3% rule”, in that if a change improves the language or the type checker and comes at a cost of less than a 3% slowdown it has a high probability of being accepted and acceptance/rejection is publicly debated ala LKML. Commentary on the Clojure process may be inferred.
Formalism. Formalism gives portabity and is the only real rout thereto. Parser spec. Evaluator spec. Kernel standard library spec with a features list. This enables portability of code and data structures. Keeps us out of the JVM centric mess that Clojure itself is close to right now.
Documentation. I built Grimoire (Introduction post) because frankly I think that the existing Clojure documentation is lacking and there is not interest from the Core team in improving it as evidenced by the consistent rejection of patches which change doc strings beyond simple typo fixes. Goes with formalism requirements.
Hide the host. That formalism should include the call stack and exceptions. Use the host under the hood sure, but don’t leak the host for the most part. To paraphrase Paul Phillips, Just because Java or Clojure jumped off a bridge doesn’t mean we can’t have nice things like Data.Ord. This likely means making interop deliberately second class. Possible even explicitly generated from introspection but not the common case. Musings on this may be found here.
Types (and protocols/interfaces) matter. Clojure and much of the Cognitect crew seem to pretend that they don’t. It feels like not a week goes by without someone in IRC or twitter posting a function that checks the run time type of some value against some implementation type of Clojure’s core. Aphyr obligingly posted today’s, clojure.core.incubator/sequable, I have a few. Enable static reasoning rather than pretending it doesn’t matter and that nobody does it. Edit: From the horse’s mouth. Bake in type inference and something akin to prismatic/schema.
Purify the language implementation. Practice what you preach. Oxlang/Kiss’s immutable environments absolutely required. Death to &env, *ns* and mutable namespaces, do reloading by dependency tracking.
Give thought to being self-hosting. It’s not essential off the bat, but it makes porting & compatability easier in the long run. GHC’s bootstrap & rebuild itself process beccons and Toccata plays the same trick.
Give more thought to the module system & versioning culture. I want to be able to depend on “something implementing these functions in the namespace clojure.core then require those and alias them in as such”, not “Dear lein, this exact version of this library, thanks no ranges can’t trust anyone. Okay lets just load crap up”. The language should enforce versioning b/c programmers sure as hell can’t be trusted with it. Note that this goes with my first point to lower the cost(s) of churn in the standard library.
Tooling. I’m with Chas in his comments here. As hinted in 8, the language needs to provide as much support as possible for editors, for REPLs, for packages, for optimizing, for testing. These are the things that make or break the UX of the language and if the UX isn’t good people will just go back to writing node.js or whatever the latest flavor of the week is. Tooling is hard and takes time, but it must be a goal. I also went over this in ‘On Future Languages’.

All of these (save the parser and evaluator spec) are goals towards which work can be accumulated over time once the semantics of the language are agreed upon. Tooling, types, speed, correctness analysis can all be done if changes (growth) is cheap and the goals are agreed upon.

I think that given these changes it would be pretty easy to design embedded and memory restricted targets for “Clojure”, and I’ll admit that the goal of writing an OS with this language is near to my heart but for the most part I want to improve the platform independent experience since most of the code I write isn’t platform bound it’s exploratory scratch work.

牛: the environment model

2014-10-27T08:17:00+00:00

Disclaimer: Oxlang is vaporware. It may exist some day, there is some code, however it is just my thought experiment at polishing out some aspects of Clojure I consider warts by starting from a tabula rasa. The following represents a mostly baked scheme for implementing ns and require more nicely in static (CLJS, Oxcart) rather than dynamic (Clojure) contexts.

Unlike Clojure in which the unit of compilation is a single form, Oxlang’s unit of compilation is that of a “namespace”, or a single file. Oxlang namespaces are roughly equivalent to Haskell modules in that they are a comprised of a “header”, followed by a sequence of declarative body forms.

In Clojure, the ns form serves to imperatively create and initialize a namespace and binding scope. This is done by constructing a new anonymous function, using it as a class loader context to perform cached compilation of depended namespaces. Subsequent forms are compiled as they occur and the results are accumulated as globally visible defs.

Recompiling or reloading a file does exactly that. The ns form is re-executed, incurring more side-effects, and all forms in the file are re-evaluated generating more defs. However this does not discard the old defs from the same file, nor purge the existing aliases and refers in the reloaded namespace. This can lead to interesting bugs where changes in imports and defs create name conflicts with the previous imports and cause reloading to fail. The failure to invalidate deleted defs also creates conditions where for instance during refactorings the old name for a function remains interred and accessible the program run time allowing evaluation of code which depends on the old name to succeed until the entire program is reloaded in a fresh run time at which point the missing name will become evident as a dependency fault.

Furthermore, the Var mechanism serves to enable extremely cheap code reloading because all bindings are dynamically resolved anyway. This means that there is exactly zero recompilation cost to new code beyond compilation of the new code itself since the Var look up operation is performed at invoke time rather than at assemble time.

Unfortunately in my Clojure development experience, the persistence of deleted symbols resulted in more broken builds than I care to admit. Building and maintaining a dependency graph between symbols is computationally inexpensive, is a key part of many language level analyses for program optimization and here critically provides better assurance that REPL development behavior is identical to program behavior in a cold program boot context.

In order to combat these issues, two changes must be made. First, re-evaluating a ns form must yield a “fresh” environment that cannot be tainted by previous imports and bindings. This resolves the import naming conflict issues by making them impossible. By modeling a “namespace” as a concrete “module” value having dependencies, public functions and private functions we can mirror the imperative semantics enabled by Clojure’s defs and Vars simply by accumulating “definitions” into the “module” as they are compiled.

This model isn’t a total gain however due to the second change, that reloading entirely (and deliberately) invalidates the previous definitions of every symbol in the reloaded namespace by swapping out the old namespace definition for the new one. This implies that other namespaces/modules which depend on a reloaded module must themselves be reloaded in topological sort order once the new dependencies are ready requiring dependency tracking and reloading infrastructure far beyond Clojure’s (none). Naively this must take place on a file by file basis as in Scala, however by tracking file change time stamps of source files and the hash codes of individual def forms a reloading environment can prove at little cost that no semantic change has taken place and incur the minimum change cost. I note here the effectiveness of GHCI at enabling interactive development under equivalent per-file reloading conditions as evidence that this model is in fact viable for enabling the interactive work flow that we associate with Clojure development.

With “namespaces” represented as concrete immutable values, we can now define namespace manipulation operations such as require and def in terms of functions which update the “current” namespace as a first class value. A def when evaluated simply takes a namespace and returns a new namespace that “happens” to contain a new def. However the work performed is potentially arbitrary. refer, the linking part of require, can now be implemented as a function which takes some enumeration of the symbols in some other namespace and the “current” environment, then returns a “new” environment representing the “current” environment with the appropriate aliases installed.

This becomes interesting because it means that the return value of load need lot be the eval result of the last form in the target file, it can instead be the namespace value representing the final state of the loaded module. Now, given a caching/memoized load (which is require), we can talk about an “egalitarian” loading system where user defined loading paths are possible because refer only needs the “current” namespace, a “source” namespace and a spec. Any function could generate a “namespace” value, including one which happens to perform loading of an arbitrary file as computed by the user. See technomancy’s egalitarian ns for enabling the hosting of multiple versions of a single lib simultaneously in a single Clojure instance is one possible application of this behavior.

It is my hope that by taking this approach the implementation of namespaces and code loading can be simplified greatly however one advantage of the Var structure is that it enables forwards and out of order declarations which is immensely useful while bootstrapping a language run time ex nihilo, as done here in the Clojure core. ns itself must exist in the “empty” namespace, otherwise as the “empty” namespace is used to analyze the first form in a file (stateless (abstractly) compiler ftw) the ns form itself will not be resolved and no program can be constructed. Ox could follow Clojure’s lead and cheat by not implementing ns in Ox but rather bootstrapping it from Java or Clojure or whatever the implementing language turns out to be but I’d like to do better than that. This is a problem I haven’t solved yet.

牛: a preface

2014-09-10T17:09:39+00:00

牛 (Ox or oxlang) is an experiment, a dream, a thought which I can’t seem to get out of my head. After working primarily in Rich Hicky’s excelent language Clojure for over two years now and spending a summer hacking on Oxcart, an optimizing compiler for Clojure, it is my considered oppinion that Clojure is a language which Rich invented for his own productivity at great personal effort. As such, Clojure is a highly oppinionated Lisp, which makes some design decisions and has priorities which seem to be starkly at odds with mine.

Ultimately I think that the Clojure language is underspecified. There is no formal parse grammar for Clojure’s tokens. There is no spec for the Clojure standard library. Due to underspecification Clojure as a language is bound to the Rich’s implementation for Java as due to leaking implementation details and lack of a spec no other implemetation can ensure compatability. Even ClojureScript, the official effort to implement Clojure atop Javascript is at best a dialect due to the huge implementation differences between the two languages.

These are not to say that Clojure is a bad language. I think that Clojure is an excellent language. If there is an Ox prototype, it will likely be built in Clojure. It’s just that my priorities and Rich’s are at a mismatch. Rich is it seems happy with the existing JVM implementation of Clojure, and I see that there’s a lot of really interesting work that could be done to optimize, type and statically link Clojure or a very Clojure like language and that such work is very unlikely to become part of the Clojure core.

Clojure’s lack of specification makes it futile for me to invest in providing an imperfect implementation so the only thing that makes sense to do is specify my own lang. The result is the Ox language. The joke in the name is twofold: the compiler for Clojure I was working on when I originally had the idea for Ox was Oxcart, named after the A-12/SR-71 program. The rest of it is in the name. Oxen are slow, quiet, tractable beasts of burden used by many cultures. This ultimately characterizes the language which I’m after in the Ox project. There’s also some snarking to be found about the tractability of the Ox compared to that of other languages mascots like the gopher, the camel and the gnu which are much less willing.

The hope of the Ox project is to produce a mostly typed, mostly pure language which is sufficiently well specified that it can support both static and dynamic implementations on a variety of platforms. Ox draws heavily on my experience with Clojure’s various pitfalls, as well as my exposure to Haskell, Shen, Kiss and Ocaml to produce a much more static much more sound Lisp dialect language which specifies a concrete standard library and host interoperation mechanics in a platform abstract way amenable to both static and dynamic implementations on a variety of platforms.

In the forthcomming posts I will attempt to illustrate the model and flavor of the Ox programming language, as well as sketch at some implementation details that differentiate Ox from Clojure and I think make it a compelling project. Note however that as of now Ox is and will remain vaporware. As with the Kernel lisp project, while I may choose to implement it eventually the exercise of writing these posts is really one of exploring the Lisp design space and trying to determine for myself what merit this project has over the existing languages from which it draws inspiration.

On Future Languages

2014-08-26T05:05:56+00:00

As Paul Philips notes at the end of his talk We’re Doing It All Wrong [2013], ultimately a programming language is incidental to building software rather than critical. Ultimately the “software developer” industry is not paid to write microcode. Rather, software as an industry exists to deliver buisness solutions. Similarly computers themselves are by a large incidental. Rather, they are agents for delivering data warehousing, search and transmission solutions. Very few people are employed to improve software and computer technology for the sake of doing so compared to the hordes of industry programmers who ultimately seek to use computers as a magic loom to create value for a business. In this light I think it’s meaningful to investigate recent trends in programming languages and software design as they pertain to solving problems not to writing code.

As the last four or five decades of writing software stand witness, software development is rarely an exercise in first constructing a perfect program, subsequently delivering it and watching pleased as a client makes productive use of it until the heat death of the universe. Rather, we see that software solutions when delivered are discovered to have flaws, or didn’t solve the same problem that the client thought that it would solve, or just didn’t do it quite right, or the problem changed. All of these changes to the environment in which the software exists demand changes to the software itself.

Malleability

In the past, we have attempted to develop software as if we were going to deploy perfect products forged of pure adamantium which will endure forever unchanged. This begot the entire field of software architecture and the top down school of software design. The issue with this model as the history of top down software engineering stands testament is that business requirements change if they are known, and must be discovered and quantified if they are unknown. This is an old problem with no good solution. In the face of incomplete and/or changing requirements all that can be done is to evolve software as rapidly and efficiently as possible to meet changes as Parnas argues.

In the context of expecting change, languages and the other tools used to develop changing software must be efficient to re-architect and change. As Paul Philips says in the above talk, “modification is undesirable, modifiability is paramount”.

Looking at languages which seem to have enjoyed traction in my lifetime, the trend seems to have been that, with the exception of tasks for which the language in which the solution was to be built was a requirement the pendulum both of language design and of language use has been swinging away from statically compiled languages like Java, C & C++ (the Algol family) towards interpreted languages (Perl, Python, Ruby, Javascript) which trade off some performance for interactive development and immediacy of feedback.

Today, that trend would seem to be swinging the other way. Scala, a statically checked language base around extensive type inference with some interactive development support has been making headway. Java and C++ seem to have stagnated but are by no means dead and gone. Google Go, Mozilla Rust, Apple Swift and others have appeared on the scene also fitting into this intermediary range between interactive and statically compiled with varying styles of type inference to achieve static typing while reducing programmer load. Meanwhile the hot frontier in language research seems to be static typing and type inference as the liveliness of the Haskell ecosystem is ample proof.

Just looking at these two trends, I think it’s reasonable to draw the conclusion that interpreted, dynamically checked, dynamically dispatched languages like Python and Ruby succeeded at providing more malleable programming environments than the languages which came before them (the Algol family & co). However while making changes in a dynamically checked language is well supported, maintaining correctness is difficult because there is no compiler or type checker to warn you that you’ve broken something. This limits the utility of malleable environments, because software which crashes or gives garbage results is of no value compared to software which behaves correctly. However, as previously argued, software is not some work of divine inspiration which springs fully formed from the mind onto the screen. Rather the development of software is an evolutionary undertaking involving revision (which is well suited to static type checking) and discovery which may not be.

As a checked program must always be in a legal (correct with respect to the type system) state by definition, this precludes some elements of development by trial and error as the type system will ultimately constrain the flexibility of the program requiring systematic restructuring where isolated change could have sufficed. This is not argued to be a flaw, it is simply a trade off which I note between allowing users to express and experiment with globally “silly” or “erroneous” constructs and the strict requirement that all programs be well formed and typed when with respect to some context or the programmer’s intent the program may in fact be well formed.

As program correctness is an interesting property, and one which static model checking including “type systems” is well suited to assisting with, I do not mean to discount typed languages. Ultimately, a correct program must be well typed with respect to some type system whether that system is formalized or not.

Program testing can be used to show the presence of bugs, but never to show their absence!

~ Dijkstra (1970)

Static model checking on the other hand, can prove the presence of flaws with respect to some system. This property alone makes static model checking an indispensable part of software assurance as it cannot be replaced by any other non-proof based methodology such as assertion contracts or test cases.

Given this apparent trade off between flexibility and correctness, Typed Racket, Typed Clojure and the recent efforts at Typed Python are interesting, because they provide halfway houses between the “wild west” of dynamic dispatch & dynamic checking languages like traditional Python, Ruby and Perl and the eminently valuable model checking of statically typed languages. This is because they enable programmers to evolve a dynamically checked system, passing through states of varying levels of soundness towards a better system and then once it has reached a point of stability solidify it with static typing, property based testing and other static reasoning techniques without translating programs to another language which features stronger static analysis properties.

Utility & Rapidity from Library Support

Related to malleability in terms of ultimately delivering a solution to a problem that gets you paid is the ability to get something done in the first place. Gone (forever I expect) are the days when programs are built without using external libraries. Looking at recent languages, package/artifact management and tooling capable of trivializing leveraging open source software has EXPLODED. Java has mvn, Python has pip, Ruby has gem, Haskell has cabal, Node has npm and the Mozilla Rust team deemed a package manager so critical to the long term success of the language that they built their cargo system long before the first official release or even release candidate of the language.

Why are package managers and library infrastructure critical? Because they enable massive code reuse, especially of vendor code. Building a webapp? Need a datastore? The investment of buying into any proprietary database you may choose has been driven so low by the ease with which $free (and sometimes even free as in freedom) official drivers can be found and slotted into place it’s silly. The same goes for less vendor specific code… regex libraries, logic engine libraries, graphics libraries and many more exist in previously undreamed of abundance (for better or worse) today.

The XKCD stacksort algorithm is a tongue in cheek reference to the sheer volume of free as in freedom forget free as in beer code which can be found and leveraged in developing software today.

This doesn’t just go for “library” support, I’ll also include here FFI support. Java, the JVM family of languages, Haskell, Ocaml and many others gain much broader applicability for having FFI interfaces for leveraging the decades of C and C++ libraries which predate them. Similarly Clojure, Scala and the other “modern” crop of JVM languages gain huge utility and library improvements from being able to reach through to Java and leverage the entire Java ecosystem selectively when appropriate.

While it’s arguably unfair to compare languages on the basis of the quantity of libraries available as this metric neglects functionally immeasurable quality and utility, the presence of any libraries is a legitimate comparison in terms of potential productivity to comparative absence.

What good is a general purpose building material, capable of constructing any manner of machine, when simple machines such as the wheel or the ramp must be reconstructed by every programmer? Not nearly so much utility as a building material providing these things on a reusable basis at little or no effort to the builder regardless of the ease with which one may custom build such tools as needed.

So What’s the Big Deal

I look at this and expect to see two trends coming out of it. The first of which is that languages with limited interoperability and/or limited library bases are dead. Stone cold dead. Scheme, RⁿRS, and Common Lisp were my introductions to the Lisp family of languages. They are arguably elegant and powerful tools, however compared to other tools such as python they seem offer at best equal leverage due to prevailing lack of user let alone newbie friendly library support compared to other available languages.

I have personally written 32KLoC in Clojure. More counting intermediary diffs. That I can find on my laptop. Why? Because Clojure unlike my experiences with Common Lisp and Scheme escapes the proverbial lisp curse simply thanks to tooling which facilitates library and infrastructure sharing at a lower cost than the cost of reinvention. Reinvention still occurs, as it always will, but the marginal cost of improving an existing tool vs writing a new one is in my experience a compelling motivator for maintaining existing tool kits and software. This means that Clojure at least seems to have broken free of the black hole of perpetual reinvention and is consequently liberating to attack real application critical problems rather than distracting programmers into fighting simply to build a suitable environment.

It’s not that Clojure is somehow a better language, arguably it ultimately isn’t since it lacks the inbuilt facilities for many interesting static proof techniques, but that’s not the point. As argued above, the language(s) we use are ultimately incidental to the task of building a solution to some problem. What I really want is leverage from libraries, flexibility in development, optional and/or incremental type systems and good tooling. At this task, Clojure seems to be a superior language.

The Long View

This is not all to say that I think writing languages is pointless, nor that the languages we have today are the best we’ve ever had let alone the best we ever will have at simultaneously providing utility, malleability and safety. Nor is this to say that we’ll be on the JVM forever due to the value of legacy libraries or something equally silly. This is however to say that I look with doubt upon language projects which do not have the benefit of support from a “major player”, a research entity or some other group willing to fund long term development in spite of short term futility simply because the literal price of bootstrapping a “new” language into a state of compelling utility is expensive in terms of man-years.

This conclusion is, arguably, my biggest stumbling block with my Oxlang project. It’s not that the idea of the language is bad, it’s that a tradeoff must be carefully made between novelty and utility. Change too much and Oxlang will be isolated from the rest of the Clojure ecosystem and will loose hugely in terms of libraries as a result. Change to little and it won’t be interesting compared to Clojure. Go far enough, and it will cross the borders of hard static typing, entering the land of Shen, Ocaml and Haskell and as argued above I think sacrifice interesting flexibility for uncertain gains.