Module Internal.File_manager

Abstraction that governs the lifetime of the various files that are part of a pack store (except the branch store).

The file manager (FM) handles the files one by one and makes explicit all the interactions between them (except the index which is handled from a high level API). This allows to gain confidence on SWMR consistency and crash consistency.

Two types of guarantees

Irmin_pack_unix supports the SWMR access scheme. It means that it is undefined for the files to be opened twice in RW mode by 2 FMs. It also means that it is undefined for a FM in RW mode to be used simultaneously from 2 different fibers.

Irmin_pack_unix aims to be (crash) consistent (in the ACID sense). In case of a system failure (e.g. power outage), the files should be left in a consistent state for later uses.

Ensuring SWMR consistency is easier than ensuring crash consistency because the of the OS's shared page cache.

Files mutations

Here are all the moments where the files managed may be mutated:

1. 2. and 6. don't support SWMR and leave the store in an undefined state in case of crash.

4. operates on files private to the worker. It doesn't necessitate to worry about crash concistency and SWMR.

3. and 5. are highly critical.

module Io = Io
module Control : Control_file.Upper with module Io = Io
module Dict : Dict.S with module Io = Io
module Suffix : Chunked_suffix.S with module Io = Io

Abstraction for a chunked suffix. It is functionally equivalent to Append_only_file but with a chunked implementation that is parameterized by

module Index = Index
module Errs = Errs
module Sparse : Sparse_file.S with module Io = Io
module Lower : Lower.S with module Io = Io
type t

A series of getters to the underlying files managed.

val control : t -> Control.t
val dict : t -> Dict.t
val suffix : t -> Suffix.t
val index : t -> Index.t
val prefix : t -> Sparse.t option
val lower : t -> Lower.t option
type create_error := [
  1. | Io.create_error
  2. | Io.write_error
  3. | Io.open_error
  4. | Io.mkdir_error
  5. | `Corrupted_mapping_file of string
  6. | `Corrupted_control_file of string
  7. | `Double_close
  8. | `Unknown_major_pack_version of string
  9. | `Volume_missing of string
  10. | `Not_a_directory of string
  11. | `Multiple_empty_volumes
  12. | `Index_failure of string
  13. | `Sys_error of string
  14. | `No_tmp_path_provided
]
val create_rw : overwrite:bool -> Irmin.Backend.Conf.t -> (t, [> create_error ]) Stdlib.result

Create a rw instance of t by creating the files.

Note on SWMR consistency: It is undefined for a reader to attempt an opening before create_rw is over.

Note on crash consistency: Crashing during create_rw leaves the storage in an undefined state.

Note on errors: If create_rw returns an error, the storage is left in an undefined state and some file descriptors might not be closed.

type open_rw_error := [
  1. | `Corrupted_control_file of string
  2. | `Corrupted_mapping_file of string
  3. | `Double_close
  4. | `Closed
  5. | `File_exists of string
  6. | `Index_failure of string
  7. | `Invalid_argument
  8. | `Invalid_layout
  9. | `Io_misc of Io.misc_error
  10. | `Migration_needed
  11. | `Migration_to_lower_not_allowed
  12. | `No_such_file_or_directory of string
  13. | `Not_a_directory of string
  14. | `Not_a_file
  15. | `No_tmp_path_provided
  16. | `Read_out_of_bounds
  17. | `Ro_not_allowed
  18. | `Sys_error of string
  19. | `V3_store_from_the_future
  20. | `Only_minimal_indexing_strategy_allowed
  21. | `Unknown_major_pack_version of string
  22. | `Index_failure of string
  23. | `Sys_error of string
  24. | `Inconsistent_store
  25. | `Volume_missing of string
  26. | `Multiple_empty_volumes
  27. | `Invalid_parent_directory
  28. | `Pending_flush
]
val open_rw : Irmin.Backend.Conf.t -> (t, [> open_rw_error ]) Stdlib.result

Create a rw instance of t by opening existing files.

If the pack store has already been garbage collected, opening with a non-minimal indexing strategy will return an error.

If no_migrate = false in the config, the store will undergo a major version upgrade if necessary.

Note on SWMR consistency: It is undefined for a reader to attempt an opening during an open_rw, because of major version upgrades.

Note on crash consistency: If open_rw crashes during a major version upgrade, the storage is left in an undefined state. Otherwise the storage is unaffected.

Note on errors: If open_rw returns an error during a major version upgrade, the storage is left in an undefined state. Otherwise the storage is unaffected. Anyhow, some file descriptors might not be closed.

type open_ro_error := [
  1. | `Corrupted_control_file of string
  2. | `Corrupted_mapping_file of string
  3. | `Io_misc of Io.misc_error
  4. | `Migration_needed
  5. | `No_such_file_or_directory of string
  6. | `Not_a_file
  7. | `Double_close
  8. | `Closed
  9. | `V3_store_from_the_future
  10. | `Index_failure of string
  11. | `Unknown_major_pack_version of string
  12. | `Inconsistent_store
  13. | `Invalid_argument
  14. | `Read_out_of_bounds
  15. | `Invalid_layout
  16. | `Volume_missing of string
]
val open_ro : Irmin.Backend.Conf.t -> (t, [> open_ro_error ]) Stdlib.result

Create a ro instance of t by opening existing files.

Note on SWMR consistency: open_ro is supposed to work whichever the state of the pack store and the writer, with 2 exceptions: 1. the files must exist and create_rw must be over and 2. during a major version upgrade of the files (which occurs during a open_rw).

Note on crash consistency: Crashes during open_ro cause no issues because it doesn't mutate the storage.

Note on errors: The storage is never mutated. Some file descriptors might not be closed.

type close_error := [
  1. | `Double_close
  2. | `Index_failure of string
  3. | `Io_misc of Io.misc_error
  4. | `Pending_flush
  5. | `Ro_not_allowed
]
val close : t -> (unit, [> close_error ]) Stdlib.result

Close all the files.

This call fails if the append buffers are not in a flushed stated. This situation will most likely never occur because the append buffers will contain data only during the scope of a batch function.

type flush_error := [
  1. | `Index_failure of string
  2. | `Io_misc of Io.misc_error
  3. | `Ro_not_allowed
  4. | `Closed
  5. | `Double_close
  6. | `File_exists of string
  7. | `Sys_error of string
  8. | `No_tmp_path_provided
]
type flush_stages := [
  1. | `After_dict
  2. | `After_suffix
]
type 'a hook := 'a -> unit
val flush : ?hook:flush_stages hook -> t -> (unit, [> flush_error ]) Stdlib.result

Execute the flush routine. Note that this routine may be automatically triggered when buffers are filled.

type fsync_error := flush_error
val fsync : t -> (unit, [> fsync_error ]) Stdlib.result

fsync executes an fsync for all files of the file manager.

Note: This function exists primarily for operations like snapshot imports. If fsync is enabled for the store (see Irmin_pack.Config.use_fsync), calls to flush will also call fsync and therefore there is little need to call this function directly.

type reload_stages := [
  1. | `After_index
  2. | `After_control
  3. | `After_suffix
]
val reload : ?hook:reload_stages hook -> t -> (unit, [> Errs.t ]) Stdlib.result

Execute the reload routine.

Is a no-op if the control file did not change.

val register_prefix_consumer : t -> after_reload:(unit -> (unit, Errs.t) Stdlib.result) -> unit
val register_suffix_consumer : t -> after_flush:(unit -> unit) -> unit
type version_error := [
  1. | `Corrupted_control_file of string
  2. | `Corrupted_legacy_file
  3. | `Invalid_layout
  4. | `Io_misc of Io.misc_error
  5. | `No_such_file_or_directory of string
  6. | `Not_a_directory of string
  7. | `Unknown_major_pack_version of string
]
val version : root:string -> (Irmin_pack.Version.t, [> version_error ]) Stdlib.result

version ~root is the version of the pack stores at root.

val cleanup : t -> (unit, [> `Sys_error of string ]) Stdlib.result

cleanup t performs cleanup operations for files related to GC.

val swap : t -> generation:int -> mapping_size:Optint.Int63.t -> suffix_start_offset:Optint.Int63.t -> chunk_start_idx:int -> chunk_num:int -> suffix_dead_bytes:Optint.Int63.t -> latest_gc_target_offset:Optint.Int63.t -> volume:Lower.volume_identifier option -> (unit, [> Errs.t ]) Stdlib.result

Swaps to using files from the GC generation. The values suffix_start_offset, chunk_start_idx, chunk_num, and suffix_dead_bytes are used to properly load and read the suffix after a GC. The value volume is used to reload the lower if it was modified by the GC. The control file is also updated on disk.

val readonly : t -> bool
val generation : t -> int
val gc_allowed : t -> bool
val split : t -> (unit, [> Errs.t ]) Stdlib.result
val add_volume : t -> (unit, [> Errs.t ]) Stdlib.result
val gc_behaviour : t -> [ `Delete | `Archive ]

Decides if the GC will delete or archive the garbage data, depending on the presence of a lower layer.

val gc_destination : t -> [ `Delete | `Archive of Lower.t ]

Returns where data discarded by the GC will end up. (see gc_behaviour).

val create_one_commit_store : t -> Irmin.Backend.Conf.t -> Control_file.Payload.Upper.Latest.gced -> Index.key Pack_key.t -> (unit, [> open_rw_error | close_error ]) Stdlib.result

create_one_commit_store t conf generation new_store_root key is called when creating a new store at new_store_root from the existing one, containing only one commit, specified by the key. Ths new store will use configuration options from conf and set to generation.