Working on the Dune codebase

This section gives guidelines for working on Dune itself. Many of these are general guidelines that are specific to Dune. However, given that Dune is a large project developed by many different people, it is important to follow these guidelines when working on Dune in order to keep the project in a good state and pleasant to work on for everybody.

Writing tests

Most of our tests are written as expectation style tests. While writing such tests, the developer write some code and then let the system insert the output produced during the execution of this code right next to the code in the source file.

Once a test is written and committed, the system will check that the captured output is still the one produced by a fresh execution of the code. When the two don’t match, the test is considered as failing and the system displays a diff between what was expected and what the code produced.

Both our unit tests and integration tests are written this way. For unit tests, we use the ppx_expect framework where tests are introduced via let%expect_test and expectation are capture in [%expect ...] nodes:

let%expect_test "<test name>" =
   print_string "Hello, world!";
   [%expect {|
     Hello, world!
   |}]

For integration tests, we use a system similar to cram tests for testing shell commands and their behavior:

$ echo 'Hello, world!'
Hello, world!

$ false
[1]

$ cat <<EOF
> multi
> line
> EOF
multi
line

Guidelines

As with any long running software project, code written by one person will always eventually end up being maintained by another. Just like normal code, it is important to document tests. Especially since test suites are most often composed of many individual tests that must be understood on their own.

A well written test case should be easy to understand. A reader should be able to quickly understand what property the test is checking, how it is doing it and how to convince one-self that the test outcome is the right one. A well written test will make it easy for future maintainers to understand the test and react when the test breaks. Most often, the code will need to be adapted to preserve the existing behavior, however in some rare cases the test expectation will need to be updated.

It is crucial that each test cases makes it purpose and logic crystal clear so that future maintainers know how to deal with it.

When writing a test, we generally have a good idea of what we want to test. Sometimes, we want to test that a new feature we developed is behaving as we expect. Sometimes, we want to add a reproduction case for a bug reported by a user to make sure future changes won’t re-introduce the faulty behaviour. Just like when programming, we turn such an idea into code, which is a formal language that a computer can understand. While another person reading this code might be able to follow and understand what the code is doing step by step, it is not clear that they will be able to reconstruct the original idea the developer had in their mind when they originally wrote the code. What is worse, they might understand the code in a completely different way which would lead them to update it the wrong way.

Adding Stanzas

Adding new stanzas is the most natural way to extend dune with new features. Therefore we try to make this as easy as possible. The minimal amount of steps to add a new stanza is:

  • Extend Stanza.t with a new constructor to represent the new stanza
  • Modify Dune_file to parse the dune language into this constructor
  • Modify the rules to interpret this stanza into rules. This is usually done in Gen_rules`

Versioning

Dune is incredibly strict with versioning of new features, modifications that are visible to the user, and changes to existing rules. This means that any added stanza must be guarded behind the version of the dune language in which it was introduced. For example:

; ( "cram"
  , let+ () = Dune_lang.Syntax.since Stanza.syntax (2, 7)
    and+ t = Cram_stanza.decode in
    [ Cram t ] )

Here the cram stanza was introduced in dune 2.7, so the user must enable (lang dune 2.7) in their dune-project file to use it.

since isn’t the only primitive for making sure that versions are respected. See Dune_lang.Syntax for other commonly used functions.

Experimental & Independent Extensions

Sometimes, dune’s versioning policy is too strict. For example, it does not work in the following situations:

  • Mostly independent extensions of dune that only exist inside dune for development convenience. For example, build rules for coq. Such extensions would like to impose their own versioning policy.
  • Experimental features that cannot yet guarantee dune’s strict backwards compatibility. Such features may dropped or modified at any time.

To handle both of these use cases, dune allows to define new languages (with the same syntax). These languages have their own versioning scheme and their own stanzas (or fields). In dune itself, such languages are represented with Syntax.t Here’s an example of how the coq syntax is defined:

let coq_syntax =
  Dune_lang.Syntax.create ~name:"coq" ~desc:"the coq extension (experimental)"
   [ ((0, 1), `Since (1, 9)); ((0, 2), `Since (2, 5)) ]

The list provides which versions of the syntax are provided, and in which version of dune they were introduced.

Such languages must be enabled in the dune-project separately:

(lang dune 2.9)
(using coq 0.2)

If such extensions are experimental, it’s recommended that they pass ~experimental:true, and that their versions are below 1.0.

It’s also recommended that such extensions introduce stanzas or fields of the form ext_name.stanza_name or ext_name.field_name to make it clear to the user which extensions is providing a certain feature.

Dune Rules

Creating Rules

A dune rule consists of 3 components:

  • Dependencies that the rule may read when executed (files, aliases, ..) This is described by 'a Build.t values
  • Targets the rule produces (files) Targets, in addition to dependencies is described by 'a Build.With_targets.t'
  • Action that dune must execute (external programs, redirects, etc.) Actions are represented by Action.t

Combined, one needs to produce a Action.t Build.With_targets.t value to create a rule. The rule may then be added by Super_context.add_rule, or a related function.

To make this maximally convenient, there’s a Command module to make it easier to create actions that run external commands and describe their targets & dependencies simultaneously.

Loading Rules

Dune rules are loaded lazily to improve performance. Here’s a sketch of the algorithm that tries to load the rule that generates some target file t.

  • Get the directory that of t. Call it d.
  • Load all rules in d into a map from targets in that directory to rules that produce it.
  • Look up the rule for t in this map.

To adhere to this loading scheme, our rules must therefore be generated as part of the callback that generates targets in that directory. See the Gen_rules module for how this callback is constructed.