Module Irmin

Irmin public API.

Irmin is a library to design and use persistent stores with built-in snapshot, branching and reverting mechanisms. Irmin uses concepts similar to Git but it exposes them as a high level library instead of a complex command-line frontend. It features a bidirectional Git backend, where an application can read and persist its state using the Git format, fully-compatible with the usual Git tools and workflows.

Irmin is designed to use a large variety of backends. It is written in pure OCaml and does not depend on external C stubs; it is thus very portable and aims to run everywhere, from Linux to browser and MirageOS unikernels.

Consult the Examples of use for a quick start. See also the documentation for the unix backends.

Release 3.4.1 - %%HOMEPAGE%%

val version : string

The version of the library.

Preliminaries

module Type = Repr

Dynamic types for Irmin values, supplied by Repr. These values can be derived from type definitions via [@@deriving irmin] (see the documentation for ppx_irmin)

module Metrics : sig ... end

Type agnostics mechanisms to manipulate metrics.

module Info : sig ... end

Commit info are used to keep track of the origin of write operations in the stores. Info models the metadata associated with commit objects in Git.

module Merge : sig ... end

Merge provides functions to build custom 3-way merge operators for various user-defined contents.

module Diff : sig ... end

Differences between values.

type 'a diff = 'a Diff.t

The type for representing differences betwen values.

module Perms : sig ... end

Types representing permissions 'perms for performing operations on a certain type 'perms t.

Low-level Stores

An Irmin store is automatically built from a number of lower-level stores, each implementing fewer operations, such as content-addressable and atomic-write stores. These low-level stores are provided by various backends.

module Read_only : sig ... end

Read-only backend backends.

module Append_only : sig ... end

Append-only backend backends.

module Indexable : sig ... end

Indexable backend backends.

module Content_addressable : sig ... end

Content-addressable backends.

module Atomic_write : sig ... end

Atomic-write stores.

User-Defined Contents

module Path : sig ... end

Store paths.

module Hash : sig ... end

Hashing functions.

module Metadata : sig ... end

Metadata defines metadata that is attached to contents but stored in nodes. For instance, the Git backend uses this to indicate the type of file (normal, executable or symlink).

module Contents : sig ... end

Contents specifies how user-defined contents need to be serializable and mergeable.

module Branch : sig ... end

User-defined branches.

module Node : sig ... end

Node provides functions to describe the graph-like structured values.

module Commit : sig ... end

Commit values represent the store history.

module Key : sig ... end

Module types for keys into an arbitrary store.

type remote = ..

The type for remote stores.

type config

The type for backend-specific configuration values.

Every backend has different configuration options, which are kept abstract to the user.

module Backend : sig ... end

Backend defines functions only useful for creating new backends. If you are just using the library (and not developing a new backend), you should not use this module.

module Storage : sig ... end

Storage provides Storage.Make for defining a custom storage layer that can be used to create Irmin stores. Unlike Backend.S, an implementation of Storage.Make is only concerned with storing and retrieving keys and values. It can be used to create stores for Backend.S through something like Storage.Content_addressable or, primarily, with Of_storage to automatically construct an Irmin store.

High-level Stores

An Irmin store is a branch-consistent store where keys are lists of steps.

An example is a Git repository where keys are filenames, i.e. lists of '/'-separated strings. More complex examples are structured values, where steps might contain first-class field accessors and array offsets.

Irmin provides the following features:

exception Closed

The exception raised when any operation is attempted on a closed store, except for S.close, which is idempotent.

module type S = sig ... end

Irmin stores.

module type KV = sig ... end

KV is similar to S but chooses sensible implementations for path and branch.

module Json_tree : sig ... end
module Schema : sig ... end

Store schemas

module type Maker = sig ... end

Maker is the signature exposed by any backend providing S implementations. Maker.Make is parameterised by Schema.S. It does not use any native synchronization primitives.

module type KV_maker = sig ... end

KV_maker is like Maker but where everything except the contents is replaced by sensible default implementations. KV_maker.Make is parameterised by Contents.S

module Generic_key : sig ... end

"Generic key" stores are Irmin stores in which the backend may not be keyed directly by the hashes of stored values. See Key for more details.

Synchronization

val remote_store : (module Generic_key.S with type t = 'a) -> 'a -> remote

remote_store t is the remote corresponding to the local store t. Synchronization is done by importing and exporting store slices, so this is usually much slower than native synchronization using Store.remote but it works for all backends.

module Sync : sig ... end

Remote synchronisation.

Examples

These examples are in the examples directory of the distribution.

Syncing with a remote

A simple synchronization example, using the Git backend and the Sync helpers. The code clones a fresh repository if the repository does not exist locally, otherwise it performs a fetch: in this case, only the missing contents are downloaded.

open Lwt.Infix
module S = Irmin_unix.Git.FS.KV (Irmin.Contents.String)
module Sync = Irmin.Sync (S)

let config = Irmin_git.config "/tmp/test"

let upstream =
  if Array.length Sys.argv = 2 then
    Uri.of_string (Store.remote Sys.argv.(1))
  else (
    Printf.eprintf "Usage: sync [uri]\n%!";
    exit 1)

let test () =
  S.Repo.v config >>= S.main >>= fun t ->
  Sync.pull_exn t upstream `Set >>= fun () ->
  S.get t [ "README.md" ] >|= fun r -> Printf.printf "%s\n%!" r

let () = Lwt_main.run (test ())

Mergeable logs

The complete code for the following can be found in examples/custom_merge.ml.

We will demonstrate the use of custom merge operators by defining mergeable debug log files. We first define a log entry as a pair of a timestamp and a message, using the combinator exposed by Irmin.Type:

open Lwt.Infix
open Astring

let time = ref 0L
let failure fmt = Fmt.kstr failwith fmt

(* A log entry *)
module Entry : sig
  include Irmin.Type.S

  val v : string -> t
  val timestamp : t -> int64
end = struct
  type t = { timestamp : int64; message : string } [@@deriving irmin]

  let compare x y = Int64.compare x.timestamp y.timestamp

  let v message =
    time := Int64.add 1L !time;
    { timestamp = !time; message }

  let timestamp t = t.timestamp

  let pp ppf { timestamp; message } =
    Fmt.pf ppf "%04Ld: %s" timestamp message

  let of_string str =
    match String.cut ~sep:": " str with
    | None -> Error (`Msg ("invalid entry: " ^ str))
    | Some (x, message) -> (
        try Ok { timestamp = Int64.of_string x; message }
        with Failure e -> Error (`Msg e))

  let t = Irmin.Type.like ~pp ~of_string ~compare t
end

A log file is a list of entries (one per line), ordered by decreasing order of timestamps. The 3-way merge operator for log files concatenates and sorts the new entries and prepend them to the common ancestor's ones.

(* A log file *)
module Log : sig
  include Irmin.Contents.S

  val add : t -> Entry.t -> t
  val empty : t
end = struct
  type t = Entry.t list [@@deriving irmin]

  let empty = []
  let pp_entry = Irmin.Type.pp Entry.t
  let lines ppf l = List.iter (Fmt.pf ppf "%a\n" pp_entry) (List.rev l)

  let of_string str =
    let lines = String.cuts ~empty:false ~sep:"\n" str in
    try
      List.fold_left
        (fun acc l ->
          match Irmin.Type.of_string Entry.t l with
          | Ok x -> x :: acc
          | Error (`Msg e) -> failwith e)
        [] lines
      |> fun l -> Ok l
    with Failure e -> Error (`Msg e)

  let t = Irmin.Type.like ~pp:lines ~of_string t
  let timestamp = function [] -> 0L | e :: _ -> Entry.timestamp e

  let newer_than timestamp file =
    let rec aux acc = function
      | [] -> List.rev acc
      | h :: _ when Entry.timestamp h <= timestamp -> List.rev acc
      | h :: t -> aux (h :: acc) t
    in
    aux [] file

  let merge ~old t1 t2 =
    let open Irmin.Merge.Infix in
    old () >>=* fun old ->
    let old = match old with None -> [] | Some o -> o in
    let ts = timestamp old in
    let t1 = newer_than ts t1 in
    let t2 = newer_than ts t2 in
    let t3 =
      List.sort (Irmin.Type.compare Entry.t) (List.rev_append t1 t2)
    in
    Irmin.Merge.ok (List.rev_append t3 old)

  let merge = Irmin.Merge.(option (v t merge))
  let add t e = e :: t
end

Note: The serialisation primitives used in that example are not very efficient in this case as they parse the file every time. For real usage, you would write buffered versions of Log.pp and Log.of_string.

To persist the log file on disk, we need to choose a backend. We show here how to use the on-disk Git backend on Unix.

(* Build an Irmin store containing log files. *)
module Store = Irmin_unix.Git.FS.KV (Log)

(* Set-up the local configuration of the Git repository. *)
let config = Irmin_git.config ~bare:true Config.root

(* Convenient alias for the info function for commit messages *)
let info = Irmin_unix.info

We can now define a toy example to use our mergeable log files.

let log_file = [ "local"; "debug" ]

let all_logs t =
  Store.find t log_file >|= function None -> Log.empty | Some l -> l

(** Persist a new entry in the log. Pretty inefficient as it reads/writes
    the whole file every time. *)
let log t fmt =
  Printf.ksprintf
    (fun message ->
      all_logs t >>= fun logs ->
      let logs = Log.add logs (Entry.v message) in
      Store.set_exn t ~info:(info "Adding a new entry") log_file logs)
    fmt

let print_logs name t =
  all_logs t >|= fun logs ->
  Fmt.pr "-----------\n%s:\n-----------\n%a%!" name (Irmin.Type.pp Log.t)
    logs

let main () =
  Config.init ();
  Store.Repo.v config >>= fun repo ->
  Store.main repo >>= fun t ->
  (* populate the log with some random messages *)
  Lwt_list.iter_s
    (fun msg -> log t "This is my %s " msg)
    [ "first"; "second"; "third" ]
  >>= fun () ->
  Printf.printf "%s\n\n" what;
  print_logs "lca" t >>= fun () ->
  Store.clone ~src:t ~dst:"test" >>= fun x ->
  log x "Adding new stuff to x" >>= fun () ->
  log x "Adding more stuff to x" >>= fun () ->
  log x "More. Stuff. To x." >>= fun () ->
  print_logs "branch 1" x >>= fun () ->
  log t "I can add stuff on t also" >>= fun () ->
  log t "Yes. On t!" >>= fun () ->
  print_logs "branch 2" t >>= fun () ->
  Store.merge_into ~info:(info "Merging x into t") x ~into:t >>= function
  | Ok () -> print_logs "merge" t
  | Error _ -> failwith "conflict!"

let () = Lwt_main.run (main ())

Helpers

module Dot (S : Generic_key.S) : Dot.S with type db = S.t

Dot provides functions to export a store to the Graphviz `dot` format.

Backends

API to create new Irmin backends. A backend is an implementation exposing either a concrete implementation of S or a functor providing S once applied.

Ways to create a concrete Irmin.S implementation:

Maker uses the same type for all internal keys and store all the values in the same store.

module KV_maker (CA : Content_addressable.Maker) (AW : Atomic_write.Maker) : KV_maker with type endpoint = unit and type metadata = unit

KV_maker is like Maker but uses sensible default implementations for everything except the contents type.

module Of_storage (M : Storage.Make) (H : Hash.S) (V : Contents.S) : KV with type hash = H.t and module Schema.Contents = V

Of_storage uses a custom storage layer and chosen hash and contents type to create a key-value store.

module Of_backend (B : Backend.S) : Generic_key.S with module Schema = B.Schema and type repo = B.Repo.t and type slice = B.Slice.t and type contents_key = B.Contents.Key.t and type node_key = B.Node.Key.t and type commit_key = B.Commit.Key.t and module Backend = B

Of_backend gives full control over store creation through definining a Backend.S.

module Export_for_backends : sig ... end

Helper module containing useful top-level types for defining Irmin backends. This module is relatively unstable.