How to load additional files at runtime¶

There are many ways for applications to load files at runtime and Dune provides a well tested, key-in-hand portable system for doing so. The Dune model works by defining sites where files will be installed and looked up at runtime. At runtime, each site is associated to a list of directories which contain the files added in the site.

WARNING: This feature remains experimental and is subject to breaking changes without warning. It must be explicitly enabled in the dune-project file with (using dune_site 0.1)

Sites¶

Defining a site¶

A site is defined in a package package in the dune-project file. It consists of a name and a section (e.g lib, share, etc) where the site will be installed as a sub-directory.

(lang dune 2.9)
(using dune_site 0.1)
(name mygui)

(package
 (name mygui)
 (sites (share themes)))

Adding files to a site¶

Here the package mygui defines a site named themes that will be located in the section share. This package can add files to this sites using the install stanza:

(install
 (section (site mygui themes))
 (files
  (layout.css as default/layout.css)
  (ok.png  as default/ok.png)
  (ko.png  as default/ko.png)))

Another package mygui_material_theme can install files inside mygui directory for adding a new theme. Inside the scope of mygui_material_theme the dune file contains:

(install
 (section (site mygui themes))
 (files
  (layout.css as material/layout.css)
  (ok.png  as material/ok.png)
  (ko.png  as material/ko.png)))

The package mygui must be present in the workspace or installed.

Warning

Two files should not be installed by different packages at the same destination.

Getting the locations of a site at runtime¶

The executable mygui will be able to get the locations of the themes site using the generate sites module stanza

(executable
 (name mygui)
 (modules mygui mysites)
 (libraries dune-site))

(generate_sites_module
 (name mysites)
 (sites mygui))

The generated module mysites depends on the library dune-site provided by Dune.

Then inside mygui.ml module the locations can be recovered and used:

(** Locations of the site for the themes *)
let themes_locations : string list = Mysites.Sites.themes

(** Merge the content of the directories in [dirs] *)
let rec readdirs dirs =
  List.concat
    (List.map
       (fun dir -> Array.to_list (Sys.readdir dir))
       (List.filter Sys.file_exists dirs))

(** Get the lists of the available themes  *)
let find_available_themes () : string list = lookup_dirs themes_locations

(** Lookup a file in the directories *)
let rec lookup_file filename = function
  | [] -> raise Not_found
  | dir::dirs ->
     let filename' = Filename.concat dir filename in
     if Sys.file_exists filename' then filename'
     else lookup_file filename dirs

(** [lookup_theme_file theme file] get the [file] of the [theme] *)
let lookup_theme_file file theme =
  lookup_file (Filename.concat theme file) themes_locations

let get_layout_css = lookup_theme_file "layout.css"
let get_ok_ico = lookup_theme_file "ok.png"
let get_ko_ico = lookup_theme_file "ko.png"

Tests¶

During tests the files are copied into the sites through the dependency (package mygui) and (package mygui_material_theme) as for other files in install stanza.

Installation¶

Installation is done simply with dune install, however if one want to install this tool such that it is relocatable, one can use dune install --relocatable --prefix $dir. The files will be copied to the directory $dir but the binary $dir/bin/mygui will find the site location relative to its location. So even if the directory $dir is moved, themes_locations will be correct.

For installation trough opam, dune install must be invoked with the option --create-install-files which creates an install file <pkg>.install and copy the file that needs subtistution to an intermediary directory. The <pkg>.opam file generated by dune generate_opam_files does the right invocation.

Implementation details¶

The main difficulty for sites is that their directories are found at different locations at different times:

When the package is available locally, the location is inside _build
When the package is installed, the location is inside the install prefix
If a local package wants to install files to the site of another installed package the location is at the same time in _build and in the install prefix of the second package.

With the last example we see that the location of a site is not always a single directory, but can consist of a sequence of directories: ["dir1" ; "dir2"]. So a lookup must first look into dir1, then into dir2.

Plugins and dynamic loading of packages¶

Dune allows to define and load plugins without having to deal with specific compilation, installation directories, dependencies, or the Dynlink module.

To define a plugin:

The package defining the plugin interface must define a site where the plugins must live. Traditionally, this is in (lib plugins), but it is just a convention.
Define a library that each plugin must use to register itself (or otherwise provide its functionality).
Define the plugin in another package using the plugin stanza.
Generate a module that may load all available plugins using the generated_module stanza.

Example¶

We demonstrate an example of the scheme above. The example consists of the following components:

Inside package c,

A package c, containing the executable c, that we intend to extend with plugins.
A library c.register which defines the plugin registration interface.
A generated module Sites which can load available plugins at runtime.
An executable c that will use the module Sites to load all the plugins.

Inside package b, we declare plugin using the c.register api and the plugin stanza.

Main executable (C)¶

dune-project file:

(lang dune 2.9)
(using dune_site 0.1)
(name c)
(package
 (name c)
 (sites (lib plugins)))

dune file:

(executable
 (public_name c)
 (modules sites c)
 (libraries c.register dune-site dune-site.plugins))

(library
 (public_name c.register)
 (name c_register)
 (modules c_register))

(generate_sites_module
 (module sites)
 (plugins (c plugins)))

The generated module sites depends here also on the library dune-site.plugins because the plugins optional field is requested.

The module c_register.ml of the library c.register:

let todo = Queue.create ()

The code of the executable c.ml:

(* load all the available plugins *)
let () = Sites.Plugins.Plugins.load_all ()
(* Execute the code registered by the plugins *)
let () = Queue.iter (fun f -> f ()) !C_register.todo

One plugin (B)¶

dune-project file:

(lang dune 2.9)
(using dune_site 0.1)
(name b)

dune file:

(library
 (public_name b)
 (libraries c.register))

(plugin
 (name b)
 (libraries b)
 (site (c plugins)))

The code of the plugin b.ml:

let () =
  Queue.add (fun () -> print_endline "B is doing something") C_register.todo