Benchmarking Clojure Validation Libraries

Three weeks ago I have written about my interest in validation libraries. There were some benchmark results in that post that I was not too happy with. I am working on a new, more comprehensive benchmark.

UPDATE: Performance Comparison of Annotate, Herbert & Schema

I wanted to build a better benchmark, one that doesn’t only measure defn variants. To be honest I try to avoid defn variants. Actually any def form that add a new trick to existing ones[1]. Because, like I have written, they do not compose:

Initially I have chosen two libraries that provide annotating functions but later realized a specialized defn can be written for any validation library and these annotations are not unintrusive like core.typed annotations. Also having to use a special defn is not composable. If there is another library that adds, say instrumentation, via a defn like macro, you have to choose between validation and that. So I should admit my sampling was not quite fair and my method was not necessarily best way to approach this. Perhaps lack of a special defn form should be considered as a feature.

Two use cases I have mentioned in the previous post are also worth repeating:

1. Validating external input. Trust, but verify. No, actually just verify. These should be always on.
2. Validating inputs and outputs of code units as well as intermediary values. Assuming valid inputs, these serve to ensure the correctness of our code. We should be able to disable these, especially in production.

We can talk about a third use case that falls somewhere in between; validating functions passed as inputs. This does not fall into the first category because functions are opaque, we cannot be sure of what kind of value they will return until we call them. It does not fall into the second category either because the function in question is external input. Validating the return values of such calls to third party functions allow us to provide helpful error messages to the consumer if the function does not satisfy the contract. Otherwise the error will occur somewhere within our code and all we can say is “here is a stacktrace, open the source now, good luck figuring out what went wrong”. Checks of this third category should be always on, like the first category.

Talk Is Cheap, Show Me the Code

Code is here. You can clone the repository and a lein run will create the following in target directory:

1. results.edn contains raw results. The data is a simplified version of criterium’s output map. See validation-benchmark.core/summarize.
2. chart.png shows mean run times. The tests are grouped. See resources/tests.edn.
3. Criterium output is redirected to criterium.output.

Currently there is a validation-benchmark.core/quick? var that determines whether benchmark or quick-benchmark is used. Ideally it should be a command line parameter. Please make sure you set it as false before you run tests for the time being.

Currently there are few benchmarks and few libraries. For what it’s worth here are the results:

+------------------------------------+-----------+------------+
| Test name                          | Library   | Mean (ns)  |
+------------------------------------+-----------+------------+
| [:nil-allowed :valid]              | :annotate |   1747.448 |
| [:nil-allowed :valid]              | :herbert  |  16809.820 |
| [:nil-allowed :valid]              | :schema   |   5640.546 |
| [:nil-allowed :invalid]            | :annotate |  15178.604 |
| [:nil-allowed :invalid]            | :herbert  |  29018.129 |
| [:nil-allowed :invalid]            | :schema   | 112353.285 |
| [:sequential-collections :valid]   | :annotate |   3539.017 |
| [:sequential-collections :valid]   | :herbert  |  35712.639 |
| [:sequential-collections :valid]   | :schema   |   6065.441 |
| [:sequential-collections :invalid] | :annotate |   2009.663 |
| [:sequential-collections :invalid] | :herbert  |  12614.990 |
| [:sequential-collections :invalid] | :schema   |  33973.530 |
+------------------------------------+-----------+------------+

This is a better benchmark than the last one. It is far from telling us the library with the best performance. Raw results are here Please consider these results work in progress.