Module Irmin

module Irmin: sig .. end
Irmin public API.

Irmin is a library to design and use persistent stores with built-in snapshot, branching and reverting mechanisms. Irmin uses concepts similar to Git but it exposes them as a high level library instead of a complex command-line frontend. It features a bidirectional Git backend, where an application can read and persist its state using the Git format, fully-compatible with the usual Git tools and workflows.

Irmin is designed to use a large variety of backends. It is written in pure OCaml and does not depend on external C stubs; it is thus very portable and aims to run everywhere, from Linux to browser and MirageOS unikernels.

Consult the basics and Examples of use for a quick start. See also the documentation for the unix backends.

Release 1.1.0 - %%HOMEPAGE%%


val version : string
The version of the library.

Preliminaries


module Type: sig .. end
Dynamic types for Irmin values.
module Info: sig .. end
Commit info are used to keep track of the origin of write operations in the stores.
module Merge: sig .. end
Merge provides functions to build custom 3-way merge operators for various user-defined contents.
module Diff: sig .. end
Differences between values.

Stores


type config 
The type for backend-specific configuration values.

Every backend has different configuration options, which are kept abstract to the user.

type 'a diff = 'a Diff.t 
The type for representing differences betwen values.

An Irmin store is automatically built from a number of lower-level stores, implementing fewer operations, such as append-only and read-write stores. These low-level stores are provided by various backends.
module type RO = sig .. end
Read-only backend stores.
module type AO = sig .. end
Append-only backend store.
module type LINK = sig .. end
Immutable Link store.
module type RW = sig .. end
Read-write stores.

User-Defined Contents


module Path: sig .. end
Store paths.
module Hash: sig .. end
Hashing functions.
module Contents: sig .. end
Contents specifies how user-defined contents need to be serializable and mergeable.
module Branch: sig .. end
User-defined branches.
module Metadata: sig .. end
Metadata defines metadata that is attached to contents but stored in nodes.
module Private: sig .. end
Private defines functions only useful for creating new backends.

High-level Stores

An Irmin store is a branch-consistent store where keys are lists of steps.

An example is a Git repository where keys are filenames, i.e. list of '/'-separated strings. More complex examples are structured values, where steps might contain first-class field accessors and array offsets.

Irmin provides the following features:


module type S = sig .. end
Irmin stores.
module type S_MAKER = functor (M : Metadata.S-> functor (C : Contents.S-> functor (P : Path.S-> functor (B : Branch.S-> functor (H : Hash.S-> S  with type key = P.t
       and type step = P.step
       and type metadata = M.t
       and type contents = C.t
       and type branch = B.t
       and type Commit.Hash.t = H.t
       and type Tree.Hash.t = H.t
       and type Contents.Hash.t = H.t
S_MAKER is the signature exposed by any backend providing Irmin.S implementations.
module type KV = S  with type key = string list
     and type step = string
     and type branch = string
KV is similar to Irmin.S but choose sensible implementations for path and branch.
module type KV_MAKER = functor (C : Contents.S-> KV  with type contents = C.t
KV_MAKER is like Irmin.S_MAKER but where everything except the contents is replaced by sensible default implementations.

Synchronization


type remote 
The type for remote stores.
val remote_uri : string -> remote
remote_uri s is the remote store located at uri. Use the optimized native synchronization protocol when available for the given backend.

Examples

These examples are in the examples directory of the distribution.

Synchronization

A simple synchronization example, using the Git backend and the Irmin.Sync helpers. The code clones a fresh repository if the repository does not exist locally, otherwise it performs a fetch: in this case, only the missing contents is downloaded.

open Lwt.Infix

module S = Irmin_unix.Git.FS.KV(Irmin.Contents.String)
module Sync = Irmin.Sync(S)
let config = Irmin_git.config ~root:"/tmp/test" ()

let upstream =
  if Array.length Sys.argv = 2 then (Irmin.remote_uri Sys.argv.(1))
  else (Printf.eprintf "Usage: sync [uri]\n%!"; exit 1)

let test () =
  S.Repo.v config >>= S.master
  >>= fun t  -> Sync.pull_exn t upstream `Set
  >>= fun () -> S.get t ["README.md"]
  >|= fun r  -> Printf.printf "%s\n%!" r

let () = Lwt_main.run (test ())

Mergeable logs

We will demonstrate the use of custom merge operators by defining mergeable debug log files. We first define a log entry as a pair of a timestamp and a message, using the combinator exposed by mirage-tc:

  module Entry = sig
    include Irmin.Contents.Conv
    val v: string -> t
    val compare: t -> t -> int
    val timestamp: t -> int
  end = struct

    type t = { timestamp: int; message : string; }

    let compare x y = compare x.timestamp y.timestamp

    let v message =
      incr time;
      { timestamp = !time; message }

    let t =
      let open Irmin.Type in
      record "entry" (fun timestamp message -> { timestamp; message })
      |+ field "timestamp" int    (fun t -> t.timestamp)
      |+ field "message"   string (fun t -> t.message)
      |> sealr

    let timestamp t = t.timestamp

    let pp ppf { timestamp; message } =
      Fmt.pf ppf  "%04d: %s\n" timestamp message

    let of_string str =
      match String.cut ~sep:": " str with
      | None -> Error (`Msg ("invalid entry: " ^ str))
      | Some (x, message) ->
        try Ok { timestamp = int_of_string x; message }
        with Failure e -> Error (`Msg e)
  end

A log file is a list of entries (one per line), ordered by decreasing order of timestamps. The 3-way merge operator for log files concatenates and sorts the new entries and prepend them to the common ancestor's ones.

(* A log file *)
module Log: sig
  include Irmin.Contents.S
  val add: t -> Entry.t -> t
  val empty: t
end = struct

  type t = Entry.t list
  let t = Irmin.Type.(list Entry.t)

  let empty = []

  let pp ppf l = List.iter (Fmt.pf ppf "%a\n" Entry.pp ) (List.rev l)

  let of_string str =
    let lines = String.cuts ~sep:"\n" str in
    try
      List.fold_left (fun acc l ->
          match Entry.of_string l with
          | Ok x           -> x :: acc
          | Error (`Msg e) -> failwith e
        ) [] lines
      |> fun l -> Ok l
    with Failure e ->
      Error (`Msg e)

  let timestamp = function
    | [] -> 0
    | e :: _ -> Entry.timestamp e

  let newer_than timestamp file =
    let rec aux acc = function
      | [] -> List.rev acc
      | h:: _ when Entry.timestamp h <= timestamp -> List.rev acc
      | h::t -> aux (h::acc) t
    in
    aux [] file

  let merge ~old t1 t2 =
    let open Irmin.Merge.Infix in
    old () >>=* fun old ->
    let old = match old with None -> [] | Some o -> o in
    let ts = timestamp old in
    let t1 = newer_than ts t1 in
    let t2 = newer_than ts t2 in
    let t3 = List.sort Entry.compare (List.rev_append t1 t2) in
    Irmin.Merge.ok (List.rev_append t3 old)

  let merge = Irmin.Merge.(option (v t merge))

  let add t e = e :: t

end 

Note: The serialisation primitives used in that example are not very efficient in this case as they parse the file every-time. For real usage, you would write buffered versions of Log.pp and Log.of_string.

To persist the log file on disk, we need to choose a backend. We show here how to use the on-disk Git backend on Unix.

  (* Build an Irmin store containing log files. *)
  module S = Irmin_unix.Git.FS.KV(Log)

  (* Set-up the local configuration of the Git repository. *)
  let config = Irmin_git.config ~root:"/tmp/irmin/test" ~bare:true ()

  (* Set-up the commit info function *)
  let info fmt = Irmin_unix.info ~author:"logger" fmt

We can now define a toy example to use our mergeable log files.

  open Lwt.Infix

  (* Name of the log file. *)
  let file = [ "local"; "debug" ]

  (* Read the entire log file. *)
  let read_file t =
    S.find t file >|= function
    | None   -> []
    | Some l -> l

  (* Persist a new entry in the log. *)
  let log t fmt =
    Fmt.kstrf (fun message ->
        read_file t >>= fun logs ->
        let logs = Log.add logs (Entry.v message) in
        S.set t (info "Adding a new entry") file logs
      ) fmt

  let () =
    Lwt_main.run begin
      S.Repo.v config >>= S.master
      >>= fun t  -> log t "Adding a new log entry"
      >>= fun () -> Irmin.clone_force ~src:t ~dst:"x"
      >>= fun x  -> log x "Adding new stuff to x"
      >>= fun () -> log x "Adding more stuff to x"
      >>= fun () -> log x "More. Stuff. To x."
      >>= fun () -> log t "I can add stuff on t also"
      >>= fun () -> log t "Yes. On t!"
      >>= fun () -> S.merge (info "Merging x into t") x ~into:t
      >|= function Ok () -> () | Errror _ -> failwith "merge conflict!"
    end


Helpers


val remote_store : (module Irmin.S with type t = 'a) -> 'a -> remote
remote_store t is the remote corresponding to the local store t. Synchronization is done by importing and exporting store slices, so this is usually much slower than native synchronization using Irmin.remote_uri but it works for all backends.
module type SYNC = sig .. end
SYNC provides functions to synchronization an Irmin store with local and remote Irmin stores.
module Sync: 
functor (S : S-> SYNC with type db = S.t and type commit = S.commit
The default Sync implementation.
module Dot: 
functor (S : S-> sig .. end
Dot provides functions to export a store to the Graphviz `dot` format.

Backends

API to create new Irmin backends. A backend is an implementation exposing either a concrete implementation of Irmin.S or a functor providing Irmin.S once applied.

There are two ways to create a concrete Irmin.S implementation:


module type AO_MAKER = functor (K : Hash.S-> functor (V : Contents.Raw-> sig .. end
AO_MAKER is the signature exposed by append-only store backends.
module type LINK_MAKER = functor (K : Hash.S-> sig .. end
LINK_MAKER is the signature exposed by store which enable adding relation between keys.
module type RW_MAKER = functor (K : Contents.Conv-> functor (V : Contents.Conv-> sig .. end
RW_MAKER is the signature exposed by read-write store backends.
module Make: 
functor (AO : AO_MAKER-> 
functor (RW : RW_MAKER-> S_MAKER
Simple store creator.
module Make_ext: 
functor (P : Private.S-> S with type key = P.Node.Path.t and type contents = P.Contents.value and type branch = P.Branch.key and type Commit.Hash.t = P.Commit.key and type Tree.Hash.t = P.Node.key and type Contents.Hash.t = P.Contents.key and type step = P.Node.Path.step and type metadata = P.Node.Val.metadata and type Key.step = P.Node.Path.step and type repo = P.Repo.t
Advanced store creator.