Yurt: A Functional Relation Programming Implementation

Have you read Out of the Tar Pit? If you have not, please do so before reading this post so it would make sense. Actually you can just read that paper, there is nothing nearly as useful here. I will just be ranting on my side project that implements the system described in the paper for the next couple hundred words.

It started with a prototype written in Clojure. It seems I have deleted it, but it was June 2015. Once I had a decent idea how to approach the problem, I put the prototype aside and started working on the real implementation. Again, in Clojure. Here is the entire public API:

;; Public interface
;;
;;   (server)
;;
;;   (feed server feeder-id transaction)
;;
;;   (observe server observer-id queries callback)
;;   (forget server observer-id)

For those who did not take my advice about reading the paper; a functional relational programming (FRelP) platform can be thought of a hybrid of database and programming language. The server keeps the state, like a database. Data comes in via feeders and outputs go through observers. And definitions for schema and the queries are expressed using a DSL.

My implementation follows the description in the paper as closely as possible. A major deviation from the paper was my choice of lisp syntax for the DSL. This is not because I chose Clojure as the (initial) implementation language. And the program DSL is not embedded in Clojure. It is a statically typed, expression-based language with lisp syntax.

Nine months and 326 commits later, I decided to switch to Scala. This was not an easy decision. A lot of effort went into that implementation. I had been practising TDD and keeping things clean. But it was getting frail at its integration seams. I found myself worrying too much about the internal validation of data structures. They are used in more contexts and are quite complex compared to web applications.

I find core.typed too difficult to get right, and I knew it couldn’t handle all of Clojure. I started looking into runtime validation libraries. But eventually I had to admit it would at best be a stopgap measure, I needed proper type checking. I tried to put together something in Java first. But extreme verbosity got to me quickly, after Clojure’s extreme expressiveness. I found myself reinventing case classes (and other Scala features) too many times.

I have been working to achieve feature parity in the Scala codebase since April. I would not say Scala is a sluggish language but it sure does not have the development speed you have with Clojure. Clojure is the tersest programming language that I know of. And now spec is distributed with the standard library. It seems spec is designed to address the exact pain I was having. Nevertheless I will continue developing yurt in Scala.

Every person I have mentioned yurt asked if I was planning to open source it. The answer is no. I might open source it in the future, but that is not the plan. Had it been the plan I would have done it much sooner. Often failed projects are open sourced in an effort to salvage at least some reputation to the authors, or something like that. I do not plan on doing that either.

I am guessing another popular question would be can I use X language to use yurt? The answer is yes, as long as X is a JVM language. It has a Scala API naturally. I will add idiomatic Clojure & Java wrappers, and possibly a Kotlin one too.

The project’s website does not have a lot of information for now. I will add more documentation in the next months. For now you can leave your email there if you are interested.