Don't worry, it's ...

Probably Fine

Hi! I'm a product developer, currently keeping busy at Unruly. I've an interest in eXtreme Programming, Continuous Delivery, and cutting-edge Agile development techniques - I speak semi-regularly on these topics and co-organise XProLo, a monthly meetup for XP practitioners. I wrote a paper on Mob Programming in LBNIP 2015.

java-dirty - A fast append-only data store

11 Apr 2015

I frequently find myself wanting a light append-only data store for pet projects, so I've recently released and published a small library called java-dirty to accomplish that. Each store is backed onto a file and uses Java's NIO MappedByteBuffers for fast reads and writes.


As an implementation choice, java-dirty only supports primitive fields on objects. MappedByteBuffer provides e.g. .putLong() to directly write out a long into the file.

Creating a store is easy, and takes a Class<?> as a parameter.

Store<Event> eventStore = Store.of(Event.class).from("/path/to/file");

The store is aware of the offset of an object in bytes, which it uses to both write and read objects. In addition to this, it keeps track of an additional integer - the number of entries in the store. This lets a store be re-opened and know the number of objects currently written to it.

To give an example, here is an Event class declaration

class Event {
  public int one, two;

A store of event objects is laid out like the illustration below. 4 bytes are set aside at the start for the size of the store, and 8 bytes per object (4 bytes for each integer field).

On an SSD, java-dirty can store objects at around 4 million insertions/s, and read at a similar speed. I will be publishing the benchmarking code for these results shortly alongside the project.

Source is available on Github and the artifact is published on Maven Central.

You can include it with the following Maven coordinates


comments powered by Disqus