Using Rule Generation¶
Sometimes it can be useful to generate Dune rules that depend on the file system layout or on the content of configuration files. This often happens for integration tests.
In this document, we will see two ways to encode this behavior.
We suppose that we are testing an executable named tool
. There are some
input files named *.input
, output files named *.output
, and we want to
ensure that when running tool
on x.input
, the standard output
corresponds to x.output
.
The Generate-Include-Commit Pattern¶
Note
This is the most common way to do this. It has a couple drawbacks listed below, but you should start with this pattern.
What we are going to do is:
generate a
dune.inc
file;include it in our main
dune
file;commit the generated code in the source repository.
This creates a loop: a program (the “generator”) looks at the file system and
creates a dune.inc
file. Changes to the file system (for example, if a test
is added) mean that a change in dune.inc
will be promoted. These generated
rules are included in the main dune
file, so dune runtest
will run the
tests. Finally, the generated file is part of the source repository, so it is
not necessary to run several commands to run the test suite.
Let’s expand a bit on how to achieve this.
Generating a dune.inc
File¶
Create a gen
subdirectory and create a gen.ml
file in it:
gen/gen.ml
let generate_rules base = Printf.printf {| (rule (with-stdout-to %s.gen (run %%{bin:tool} %s.input))) (rule (alias runtest) (action (diff %s.output %s.gen))) |} base base base base let () = Sys.readdir "." |> Array.to_list |> List.sort String.compare |> List.filter_map (Filename.chop_suffix_opt ~suffix:".input") |> List.iter generate_rules
Create a dune
file in that directory:
gen/dune
(executable (name gen))
This defines an executable that lists *.input
files in the current
directory and outputs rules on its standard output.
Note
It is important to sort the input files to ensure that the output is
independent from the order in which Sys.readdir
returns the files .
For each input file, we output two rules:
The first one creates a
x.gen
file that corresponds to the actual output.The second uses a diff action to compare the actual output to the expected output. If it is different,
dune runtest
will display the difference, which can be accepted bydune promote
.
Note
It is possible to have more complicated logic here. For example, to pass
different arguments to tool
depending on the presence of a *.args
file. To do that, check if *.args
exists in generate_rules
and emit
a different (run ...)
action.
Including it in the Main dune
File¶
Our main test dune
file contains the following:
dune
(include dune.inc) (rule (deps (source_tree .)) (with-stdout-to dune.inc.gen (run gen/gen.exe))) (rule (alias runtest) (action (diff dune.inc dune.inc.gen)))
In addition to including the contents of dune.inc
, we use the same pattern
as before: dune.inc.gen
is the actual output of the generator, and
dune.inc
is the expected output. At runtime, the generator will read the
contents of the current directory (where the *.input
and *.output
files
are located), so we record (source_tree .)
as a dependency to make it run
again if a file is created, for example.
Commit the Generated Code In The Source Repository¶
To make this work, we have a final step to do. We have to add the generated file to our source tree. But since it is generated, we will have to first create an empty file, run the test, and promote the result.
$ touch dune.inc
$ dune runtest
+ (rule
+ (with-stdout-to a.gen
+ (run %{bin:tool} a.input)))
+
+ (rule
+ (alias runtest)
+ (action
+ (diff a.output a.gen)))
$ dune promote dune.inc
$ git add dune.inc
Now, running dune runtest
will run the test suite.
Notes¶
This pattern is “correct”: it will execute all tests and make sure the
list of tests is up to date. But when adding a test, it is necessary to first
run dune runtest
, promote the result, and then re-run dune runtest
to
actually run the test (and possibly promote the result of the test itself).
There is a variant of this pattern which will promote the output automatically instead of using a manual promotion step. This variant can be used either for the test list or for the individual tests.
To use it in the test list, replace the dune
file by this version:
dune
(alternative version)(include dune.inc) (rule (mode promote) (alias runtest) (deps (source_tree .)) (with-stdout-to dune.inc (run gen/gen.exe)))
Using this version, dune runtest
will directly replace dune.inc
with an
updated version.
Another caveat of this approach is that the generator needs to emit the same
output on all systems. For example, if some tests should be skipped on Linux,
the generator can not just filter the corresponding tests depending on
Sys.os_type
. It has to consistently emit a (enabled_if)
field for the
rules.
Using (dynamic_include)
¶
New in version 3.14.
This technique relies on dynamic_include, which is
more flexible than include. The difference is that
the intermediate dune.inc
file does not need to be part of the source tree.
It will only be generated by a rule and be present in the _build
directory.
At first it looks like it would be possible to reuse the same pattern as above:
change include
to dynamic_include
and delete the dune.inc
file.
However, it is not possible. The reason is that rules are loaded per directory,
and there needs to be a strict order (no cycles) between directories for this
to work.
So, instead we are going to:
generate
dune.inc
in a subdirectory namedgenerate
, andinclude these rules in a subdirectory named
run
.
These subdirectories do not need to be actual directories. They can be emulated through subdir.
To do this, we can create the following dune
file in the same directory as
the *.input
and *.output
files.
dune
(executable (name gen)) (subdir run (dynamic_include ../generate/dune.inc)) (subdir generate (rule (deps (glob_files ../*.input)) (action (with-stdout-to dune.inc (run ../gen.exe)))))
Then create the following gen.ml
file. Note that here we can define it in
the same directory.
gen.ml
let generate_rules base = Printf.printf {| (rule (with-stdout-to %s.gen (run %%{bin:tool} ../%s.input))) (rule (alias runtest) (action (diff ../%s.output %s.gen))) |} base base base base let () = Sys.readdir ".." |> Array.to_list |> List.sort String.compare |> List.filter_map (Filename.chop_suffix_opt ~suffix:".input") |> List.iter generate_rules
There are a few differences from the generator above because this one
is going to be invoked from subdirectories, so it is necessary to refer to the
..
directory both in the input (which files to read) and in the output (how
the rules are executed).
These two files are enough. dune runtest
is going to generate the rules and
interpret them in a single command.
Notes¶
This approach is shorter, but it might be more difficult to debug because changes to the generated rules will not be visible. Also, it works in that case, but it is not possible to generate all kinds of stanzas with that pattern. See dynamic_include for more information about the limitations.