How to Load Additional Files at Runtime¶

There are many ways for applications to load files at runtime and Dune provides a well-tested, key-in-hand portable system for doing so. The Dune model works by defining sites where files will be installed and looked up at runtime. At runtime, each site is associated to a list of directories which contain the files added in the site.

WARNING: This feature remains experimental and is subject to breaking changes without warning. It must be explicitly enabled in the dune-project file with (using dune_site 0.1)

Sites¶

Defining a Site¶

A site is defined in a package package in the dune-project file. It consists of a name and a section (e.g lib, share, etc) where the site will be installed as a sub-directory.

(lang dune 3.8)
(using dune_site 0.1)
(name mygui)

(package
 (name mygui)
 (sites (share themes)))

Adding Files to a Site¶

Here the package mygui defines a site named themes that will be located in the section share. This package can add files to this site using the install stanza:

(install
 (section (site (mygui themes)))
 (files
  (layout.css as default/layout.css)
  (ok.png  as default/ok.png)
  (ko.png  as default/ko.png)))

Another package mygui_material_theme can install files inside mygui directory for adding a new theme. Inside the scope of mygui_material_theme the dune file contains:

(install
 (section (site mygui themes))
 (files
  (layout.css as material/layout.css)
  (ok.png  as material/ok.png)
  (ko.png  as material/ko.png)))

The package mygui must be present in the workspace or installed.

Warning

Two files should not be installed by different packages at the same destination.

Getting the Locations of a Site at Runtime¶

The executable mygui will be able to get the locations of the themes site using the generate sites module stanza

(executable
 (name mygui)
 (modules mygui mysites)
 (libraries dune-site))

(generate_sites_module
 (module mysites)
 (sites mygui))

The generated module mysites depends on the library dune-site provided by Dune.

Then inside mygui.ml module the locations can be recovered and used:

(** Locations of the site for the themes *)
let themes_locations : string list = Mysites.Sites.themes

(** Merge the contents of the directories in [dirs] *)
let lookup_dirs dirs =
  List.filter Sys.file_exists dirs
  |> List.map (fun dir -> Array.to_list (Sys.readdir dir))
  |> List.concat

(** Get the available themes *)
let find_available_themes () = lookup_dirs themes_locations

(** [lookup_file name dirs] finds the first file called [name] in [dirs] *)
let lookup_file filename dirs =
  List.find_map
    (fun dir ->
      let filename' = Filename.concat dir filename in
      if Sys.file_exists filename' then Some filename' else None)
    dirs

(** [lookup_theme_file theme file] get the [file] of the [theme] *)
let lookup_theme_file file theme =
  lookup_file (Filename.concat theme file) themes_locations

let get_layout_css = lookup_theme_file "layout.css"
let get_ok_ico = lookup_theme_file "ok.png"
let get_ko_ico = lookup_theme_file "ko.png"

Tests¶

During tests, the files are copied into the sites through the dependency (package mygui) and (package mygui_material_theme) as for other files in install stanza.

Installation¶

Installation is done simply with dune install; however, if one wants to install this tool to make it relocatable, one can use dune install --relocatable --prefix $dir. The files will be copied to the directory $dir but the binary $dir/bin/mygui will find the site location relative to its location. So even if the directory $dir is moved, themes_locations will be correct.

For installation through opam, dune install must be invoked with the option --create-install-files which creates an install file <pkg>.install and copy the file that needs substitution to an intermediary directory. The <pkg>.opam file generated by Dune generate_opam_files does the right invocation.

Implementation Details¶

The main difficulty for sites is that their directories are found at different locations at different times:

When the package is available locally, the location is inside _build
When the package is installed, the location is inside the install prefix
If a local package wants to install files to the site of another installed package the location is at the same time in _build and in the install prefix of the second package.

With the last example, we see that the location of a site is not always a single directory, but rather it can consist of a sequence of directories: ["dir1" ; "dir2"]. So a lookup must first look into dir1, then into dir2.

Plugins and Dynamic Loading of Packages¶

Dune allows you to define and load plugins without having to deal with specific compilation, installation directories, dependencies, or the Dynlink_ module.

To define a plugin:

The package defining the plugin interface must define a site where the plugins must live. Traditionally, this is in (lib plugins), but it’s just a convention.
Define a library that each plugin must use to register itself (or otherwise provide its functionality).
Define the plugin in another package using the plugin stanza.
Generate a module that may load all available plugins using the generated_module stanza.

Example¶

We demonstrate an example of the scheme above. The example consists of the following components:

Inside package app:

An executable app, that we intend to extend with plugins
A library app.registration which defines the plugin registration interface
A generated module Sites which can load available plugins at runtime
An executable app that will use the module Sites to load all the plugins

Inside package Plugin1, we declare a plugin using the app.registration api and the plugin stanza.

Directory structure¶

.
├── app.ml
├── dune
├── dune-project
├── plugin
│   ├── dune
│   ├── dune-project
│   └── plugin1_impl.ml
└── registration.ml

Main Executable (C)¶

The dune-project file:

(lang dune 3.8)
(using dune_site 0.1)
(name app)

(package
  (name app)
  (sites (lib plugins)))

The dune file:

(executable
  (public_name app)
  (modules sites app)
  (libraries app.register dune-site dune-site.plugins))

(library
  (public_name app.register)
  (name registration)
  (modules registration))

(generate_sites_module
(module sites)
(plugins (app plugins)))

The generated module sites depends here also on the library dune-site.plugins because the plugins optional field is requested.

If the executable being created is an OCaml toplevel, then the libraries stanza needs to also include the dune-site.toplevel library. This causes the loading to use the toplevel’s normal loading mechanism rather than Dynload.loadfile (which is not allowed in toplevels).

The module registration.ml of the library app.registration:

let todo : (unit -> unit) Queue.t = Queue.create ()

The code of the executable app.ml:

(* load all the available plugins *)
let () = Sites.Plugins.Plugins.load_all ()

let () = print_endline "Main app starts..."
(* Execute the code registered by the plugins *)
let () = Queue.iter (fun f -> f ()) Registration.todo

The Plugin “plugin1”¶

The plugin/dune-project file:

(lang dune 3.8)
(using dune_site 0.1)

(generate_opam_files true)

(package
  (name plugin1))

The plugin/dune file:

(library
  (public_name plugin1.plugin1_impl)
  (name plugin1_impl)
  (modules plugin1_impl)
  (libraries app.register))

(plugin
  (name plugin1)
  (libraries plugin1.plugin1_impl)
  (site (app plugins)))

The code of the plugin plugin/plugin1_impl.ml:

let () =
  print_endline "Registration of Plugin1";
  Queue.add (fun () -> print_endline "Plugin1 is doing something...") Registration.todo

Running the Example¶

$ dune build @install && dune exec ./app.exe
Registration of Plugin1
Main app starts...
Plugin1 is doing something...