Don't worry, it's ...

Probably Fine

Hi! I'm a full-stack developer, keeping busy at Unruly. I've also got a particular interest in eXtreme Programming and Continuous Delivery - I speak semi-regularly on both topics and co-organise XProLo, a monthly meetup for XP practitioners. I wrote a paper on Mob Programming in LBNIP 2015.

java-dirty - A fast append-only data store

11 Apr 2015

I frequently find myself wanting a light append-only data store for pet projects, so I've recently released and published a small library called java-dirty to accomplish that. Each store is backed onto a file and uses Java's NIO MappedByteBuffers for fast reads and writes.

Implementation

As an implementation choice, java-dirty only supports primitive fields on objects. MappedByteBuffer provides e.g. .putLong() to directly write out a long into the file.

Creating a store is easy, and takes a Class<?> as a parameter.

Store<Event> eventStore = Store.of(Event.class).from("/path/to/file");

The store is aware of the offset of an object in bytes, which it uses to both write and read objects. In addition to this, it keeps track of an additional integer - the number of entries in the store. This lets a store be re-opened and know the number of objects currently written to it.

To give an example, here is an Event class declaration

class Event {
  public int one, two;
}

A store of event objects is laid out like the illustration below. 4 bytes are set aside at the start for the size of the store, and 8 bytes per object (4 bytes for each integer field).

On an SSD, java-dirty can store objects at around 4 million insertions/s, and read at a similar speed. I will be publishing the benchmarking code for these results shortly alongside the project.

Source is available on Github and the artifact is published on Maven Central.

You can include it with the following Maven coordinates

<dependency>
    <groupId>uk.co.probablyfine</groupId>
    <artifactId>java-dirty</artifactId>
    <version>1.4</version>
</dependency>

comments powered by Disqus