Compile ocamlbuild using Dune

[dra27]

This PR replaces the GNU make build system with Dune. The old Makefile is kept for convenience targets only.

Persuading ocamlbuild to compile itself using Dune "wrapped" libraries instead of packs would have required me to consume more prosecco than would then allow me to operate a computer but, honestly, having upgraded the project to build with Dune, why would you want to bootstrap it with ocamlbuild?

Testsuite and dependent modules all pass (well, WFM, anyway), and you can even build the man page. Welcome to Dune!

[rgrinberg]

Date aside, this is quite a simplification compared to what ocamlbuild had before 💯

[whitequark]

I think actually merging this isn't even a terrible idea; given the amount of projects already using jbuilder it's about as likely to be installed as ocamlbuild itself. That said, given the lack of active development of ocamlbuild there isn't much benefit either.

[gasche]

Sounds amusing and potentially useful to have this capability, but there are two points that I would like clarification on:

• Why must we lose the ability to bootstrap (build ocamlbuild using ocamlbuild itself)? Couldn't we keep the .mllib, .mlpack files (possibly only in the bootstrap sub-directory in some way?), or have a dune rule to generate them? The bootstrap is the most advanced test we do during ocamlbuild's day-to-day development (not that it happens that often), so I'm a bit worried about the testing implication of losing it.
• The configuration system has changed in a non-trivial way and we need someone to review the fact that it will not break other people's stuff. This should be easy to check (and must be checked) for opam-using scenario, and I would be reassured to learn that someone would have equivalent to hand-define configuration values, in not-from-opam scenarios, with as much flexibility as before.

[gasche]

Small ping. @dra27, do you think that just keeping the ocamlbuild-specific .ml files in-repo would allow the bootstrap to keep working?

[dra27]

That's weird - I most definitely authored a long comment complete a check-list of required changes several days ago... which somewhat spectacularly appears not to have been saved by GitHub...! I'll reconstruct what I wrote in the morning

[gasche]

I'm preparing another ocamlbuild release today, and looking at this issue again. My conclusion is the same: why not use Dune as ocamlbuild build system, but I really would like to keep the ability to bootstrap ocamlbuild using itself, because it is one of the most effective way to do quality control.

@dra27 wrote, to describe the difficulty:

Persuading ocamlbuild to compile itself using Dune "wrapped" libraries instead of packs

I'm not sure why that would be required. The bootstrap happens in a separate subdirectory, it doesn't need to do about the dune build outputs, and it can rebuild the packs on its own. Wouldn't keeping the bootstrap/ files suffice to make the bootstrap work?

[dra27]

I'm just updating this. Can I check I understand the bootstrap process properly:

1. You build and install ocamlbuild using the "normal" procedure (i.e. you make an existing ocamlbuild)
2. From a clean source tree, make -C bootstrap should then generate an ocamlbuild built using the previously installed ocamlbuild?

Is there any way to have the second step use an ocamlbuild which is still in the source tree? i.e. make all && make -C bootstrap?

[dra27]

OK, somewhat violating my "don't do work just before a concert" rule, I have just figured out a small patch which allows the bootstrap to work. I'll tidy it up and push it after the concert - it needs a bit of polish, which I shall aim to do later!

[gasche]

Apologies for not clarifying this earlier. My actual workflow is rather like like

make all allopt
cd bootstrap
OCAMLBUILD="../ocamlbuild.native" make

in particular, installing is not required. (This impression is given by a dubious choice of default value for the OCAMLBUILD variable in the script.)

Note that there is no hurry for this.

Have a great concert!

[dra27]

Hmm, I tried that before - it doesn't seem to work?

dra@bionic:~/ocamlbuild-master/bootstrap$ OCAMLBUILD=../ocamlbuild.native make
make -C .. clean
make[1]: Entering directory '/home/dra/ocamlbuild-master'
rm -f src/lexers.ml
rm -f src/glob_lexer.ml
make -f configure.make clean
make[2]: Entering directory '/home/dra/ocamlbuild-master'
make[2]: Nothing to be done for 'clean'.
make[2]: Leaving directory '/home/dra/ocamlbuild-master'
rm -f man/ocamlbuild.options.1
rm -f man/options_man.cm*
rm -f man/options_man.byte
rm -f man/options_man.o
rm -rf testsuite/_test_*
rm -rf tmp/
rm -f src/*.cm* *.cm*
rm -f src/*.o *.o
rm -f src/*.a *.a
rm -f test/test2/vivi.ml
make[1]: Leaving directory '/home/dra/ocamlbuild-master'
make -C .. src/ocamlbuild_config.ml
make[1]: Entering directory '/home/dra/ocamlbuild-master'
make[1]: 'src/ocamlbuild_config.ml' is up to date.
make[1]: Leaving directory '/home/dra/ocamlbuild-master'
ln -sf ../src
mkdir -p bin
BOOTSTRAP (1): building local ocamlbuild from '../ocamlbuild.native'
Warning: Won't be able to compile a native plugin
Failure: Cannot find "ocamlbuild.cmo" in ocamlbuild -where directory.
Compilation unsuccessful after building 0 targets (0 cached) in 00:00:00.
Makefile:12: recipe for target 'stage1' failed
make: *** [stage1] Error 2

[gasche]

Hm, it does look like you need to install first indeed...

[XVilka]

A good time for reviving this by the way, with the dune-2.0 release