mymake

Overview

Mymake is a tool to build C/C++-programs that requires close to no configuration.

For simple programs, it is enough to do mm main.cpp, where main.cpp is the file containing the main function of the program. Mymake will then find all other source files that need to be compiled and will compile these. Like make, mymake also only re-compiles the files that have been changed (keeping track of included headers) automatically, without configuration.

To be able to do this, mymake makes two assumptions about your code. Firstly: if you include a header, say foo.h, the file foo.cpp, foo.cxx, foo.c... contains the implementation for what is declared in the header. So if you include the header, the source file needs to be linked into your program as well. Secondly: mymake assumes that all source files that are not reachable this way are not needed by your program. This is true as long as your are not declaring things in another file than the header file that has the same name as an implementation file. If you break this assumption, mymake fails to detect your dependencies automatically and needs some help figuring this out.

Generally, these assumptions are met as long as you follow the structure usually found in C/C++-programs, so usually you do not have to worry. If your codebase does not follow these criteria, it is usually easy to make the codebase meed the criteria by adding a few includes and/or a few mostly empty header files.

Terminology

Here, the terminology used in mymake is explained.

• target: a directory containing a set of source files that together will provide one final output file.
• project: a directory containing a set of related targets. These targets can depend on each other.
• option: a string (either specified on the command line or in a project) that modifies the behaviour of the build.

Note: I am using the command mm to execute mymake through this document.

Installation

Linux/Unix (or MinGW)

The first time you install mymake, clone the repository somewhere and then run install.sh. This will compile mymake using g++, and then ask you if you want to set up an alias in your shell. If you set up an alias, you need to type source ~/.bashrc in your shell, or restart your shell, to be able to use mymake.

If you do not want an alias, you can copy the binary mymake to somewhere in your path.

After this, run mm --config to generate a global .mymake-file which contains your system specific compilation settings.

Windows (using Visual Studio 2008 or later)

Make sure that you have Visual Studio 2008 or later installed. Then, clone the repository somewhere and run setup.exe in the root directory (either by clicking it, or from the terminal, cd:ing to the proper directory first). This will locate the Visual Studio installation (automatically, most of the time), and use it to compile mymake. After compilation, it will ask if you want to place mm.exe somewhere in your path (either an existing location, or modifying your path to point to the repository location).

After installation succeeded, open a terminal and generate a global .mymake-file with the command mm --config. This will once again locate the Visual Studio installation, identify the capabilities of the compiler and generate a proper configuration. This means that mm will work outside the Visual Studio Command Prompt.

If you have multiple Visual Studio installations, you will be prompted which to use. Note that it is not necessary to use the same compiler for compiling mymake and in the global configuration.

If mymake fails to locate the installation, you can help it by providing a path to the command (e.g. setup.exe C:\Path\To\Vs or mm --config C:\Path\To\Vs), and if that fails, locate the file vsvarsall.bat and provde the path to that file.

Update

Linux/Unix

Run the update.sh script, and it will download the latest version from git and re-compile it. If you have set up an alias when you installed mymake, you do not need to do anything more. Otherwise you need to copy mymake to your path once more.

Windows

Check out the latest version of mymake using the git client you are using. Then re-compile mymake using setup.exe again, making sure to specify the same installation path as last time.

Usage

To compile things with mymake, run mm <file> where <file> is the source file containing the main function of your program. In most simple programs, this is enough, but sometimes some configuration is required. Mymake will then compile your program and finally run your program if the compilation succeeded.

If you need to configure your build process, run mm --target to generate a sample .mymake-file that shows all available options. At startup, mymake tries to find a .mymake or .myproject-file in the current directory or any parent directories (except for your home directory). If one is found, mymake uses that directory as the root directory when trying to compile.

To pass parameters to the compiled program, use -- or the -a parameter.

Parallel builds

Mymake supports parallel builds by default. When you run mm --config for the first time, mymake asks for the number of threads you want to use when compiling by default. This is then written into your global .mymake-file. As it resides in the global .mymake-file, it can be overwritten in local configurations and even from the command line using -j 3 or -j 1 to disable parallel builds temporarily.

When building projects in parallel, mymake analyzes the dependency graph of your targets and finds projects which can be compiled in parallel without breaking any dependencies between them. This can be disabled on a project-by-project basis by setting parallel to no.

When building single targets, mymake assumes that all source files (except any precompiled headers) can be compiled in parallel. If this is not the case, disable parallel builds for those targets by setting parallel to no in that .mymake-file, or in a global section of your .myproject-file. Pre- and post build steps are never run in parallel.

To limit how many compilation processes are spawned during compilation, mymake examines maxThreads. Mymake spawns maximum that many processes globally, even if two or more targets are compiled in parallel.

When building in parallel, mymake automatically adds a string like 1> or p1: in front of all output done in parallel. Each target gets a unique number, so that it is easy to see which target each error message originates from. The output 1> is similar to what is used in Visual Studio, which works in Emacs when using the Visual Studio compiler. On GCC, mymake adds a string like p1:, which Emacs correctly recognizes. However, if it causes trouble, set usePrefix=no either in your project file, or in your global .mymake-file.

Configuration files

Configuration in mymake is done by assigning values to variables. Each variable is an array of strings.

Mymake knows two different configuration files: .mymake and .myproject. Both follows the same basic syntax. A configuration file contains a number of sections, each of which contains a number of assignments.

In this context, a name is a string only containing characters and numbers.

Lines starting with # are comments.

A section starts with zero or more comma separated names enclosed in square brackets ([]). The names indicate which options need to be specified on the command line for the section to be evaluated. For example: [windows,lib] indicates that the options windows and lib needs to be active for that section to be evaluated. You can also negate the options required by prepending a !, eg. [lib,!windows]. A special case: [] is always evaluated. If nothing is specified before an assignment, it is assumed to be contained inside an empty section (ie. []).

Each section contains zero or more assignments. An assignment has the form name=value or name+=value. The first form, means that the variable on the left hand side is replaced with the value on the right hand side. The second form, however, means that the value is appended to the variable.

Each of these assignments are evaluated from top to bottom, so a variable can be overwritten later in a configuration file. The global .mymake-file is evaluated before any local files. Any variables originating from the command-line are applied last.

By default, mymake defines either the option windows or unix so that files can detect what kind of platform are being used easily.

Pre-defined options

The following options are pre-defined by mymake or the default configuration:

• release: When this option is present, mymake generates a release version of your program. See the default .mymake-file for how this is done for your compiler.
• lib: build a static library.
• sharedlib: build a shared library.
• project: defined when a project file is evaluated in the project-context.
• windows: Defined when running on a windows system, expected to be using cl.exe as the compiler.
• unix: Defined when running on an unix system, or when running in MinGW.
• reproducible: Enables reproducible builds on Linux by turning of relative paths, and adding the -ffile-prefix-map flag to the compile commands. This may not be enough if you have added custom compilation commands. This generally worsens debugging experience a bit as full paths are not stored in the debug information.

Wildcard patterns in mymake

Wildcard patterns (containing * and ?) work as expected in mymake, with one exception: the character / matches both / and \, to ease compatibility between systems. Hence, always write your patterns as if / is the path delimiter.

Variables used by mymake

These variables are used by mymake to understand what should be done:

• ext: array of the file extensions you want to compile. Whenever mymake realizes you have included x.h, looks for all extensions in ext and tries them to find the corresponding implementation file.
• execExt: file extension of executable files.
• intermediateExt: file extension of intermediate files.
• buildDir: string containing the directory used to store all temporary files when building your program. Relative to the root directory of the target (ie. where the .mymake file is).
• execDir: string containing the directory used to store the final output (the executable) of all targets. Relative to the root directory of the target.
• ignore: array of patterns (like in the shell) that determines if a certain file should be ignored. Useful when working with templates sometimes.
• noIncludes: array of patterns (like in the shell) that determines if a certain path should not be scanned for headers. Useful when you want to parts of the code that is not C/C++, where it is not meaningful to look for #include.
• input: array of file names to use as roots when looking for files that needs to be compiled. Anything that is not an option that is specified on the command line is appended to this variable. The special value * can be used to indicate that all files with an extension in the ext variable should be compiled. This is usually what you want when you are compiling a library of some kind.
• output: string specifying the name of the output file. If not specified, the name of the first input file is used instead.
• appendExt: append the original extension of the original source file to the intermediate file when compiling. This allows mymake to compile projects where there are multiple files with the same name, eg. foo.cpp and foo.c without both trying to create foo.o and thereby causing compilation to fail. Mymake warns you if you might need to add use this option.
• include: array of paths that should be added to the include path of the compilation.
• includeCl: flag to prepend all elements in include.
• library: array of system libraries that should be linked to your executable.
• libraryCl: flag to prepend all elements in library.
• localLibrary: array of local libraries that should be linked to your executable (usually used in a project).
• localLibraryCl: flag to prepend all elements in localLibrary.
• define: preprocessor defines.
• defineCl: preprocessor define flag.
• exceute: yes or no, telling if mymake should execute the program after a successful compilation. This can be overridden on the command line using -e or -ne.
• showTime: yes or no, telling if mymake should show the total compilation time when done (not implemented).
• pch: the precompiled header file name that should be used. If you are using the default configuration, you only need to set this variable to use precompiled headers. If you are using #pragma once in gcc, you will sadly get a warning that seems impossible to disable (it is not a problem when precompiling headers).
• pchFile: the name of the compiled version of the file in pch.
• pchCompile: command line for compiling the precompiled header file.
• pchCompileCombined: if set to yes, pchCompile is expected to generate both the pch-file and compile a .cpp-file.
• preBuild: array of command-lines that should be executed before the build is started. Expands variables.
• preBuildCreates: array of files created by the pre-build step which should also be included in the compilation. These are expected not to introduce any additional dependencies into the project, as they are not available at the point where mymake resolves dependencies between files and targets.
• postBuild: array of command-lines that should be executed after the build is completed. Expands variables.
• compile: array of command lines to use when compiling files. Each command line starts with a pattern (ending in :) that is matched against the file name to be compiled. The command line added last is checked first, and the first matching command-line is used. Therefore it is useful to first add the general command-line (starting with *:), and then add more specific ones. Here, you can use <file> for the input file and <output>.
• link: command line used when linking the intermediate files. Use <files> for all input files and <output> for the output file-name.
• linkOutput: link the output of one target to any target that are dependent on that target. See projects for more information.
• forwardDeps: forward any of this target's dependencies to any target that is dependent on this target.
• env: set environment variables. Each of the elements in env are expected to be of the form: variable=value or variable<=value or variable=>value. The first form replaces the environment variable variable with value, the second form prepends value to variable using the system's separator (: on unix and ; on windows), the third form appends value to variable. The second and third forms are convenient when working with PATH for example.
• explicitTargets: In projects: ignore any potential targets that do not have their own .mymake-file.
• parallel: In projects, this indicates if projects that have all dependencies satisfied may be built in parallel. The default value is yes, so projects not tolerating parallel builds may set it to no. In targets, this indicates if files in targets may be built in parallel. If so, all input files, except precompiled headers, are built in parallel using up to maxThreads threads globally. If specific targets do not tolerate this, set parallel to no, and mymake will build those targets in serial.
• maxThreads: Limits the global number of threads (actually processes) used to build the project/target globally.
• usePrefix: When building in parallel, add a prefix to the output corresponding to different targets. Defaults to either vc or gnu (depending on your system). If you set it to no, no prefix is added. vc adds n> before output, gnu adds pn: before output. This is so that Emacs recognizes the error messages from the vc and the gnu compiler, respectively.
• absolutePath: send absolute paths to the compiler, this helps emacs find proper source files in projects with multiple targets.
• implicitDeps: (defaults to yes), if set, mymake tries to figure out dependencies between targets by looking at includes. Sometimes, this results in unneeded circular dependencies, causing compilation to fail, so sometimes it is neccessary to set this to no.
• projectRoot: The full path of the root of the project (either for the .myproject file if it exists or for the .mymake file). Ends with a slash.
• targetRoot: The full path of the current target. Differs from projectRoot if a project contains multiple targets, otherwise the same path as projectRoot.
• env:X: A special variable, that evaluates to the environment variable of X. The benefit of using this syntax rather than the specific syntax of the shell on the platform is that <env:X> is general, and that it lets mymake keep track of changes to the environment variables.

Variables in strings

In indicated variables, you can embed any variable (even ones not known by mymake) that will be expanded into that string. The syntax for this is <variable>. This means that the value of variable is inserted into the string at that location. If variable is an array with multiple values, these values are concatenated into one space separated string.

It is also possible to append a string before each element in another variable by using <prefix*variable>. This means that the string in the variable prefix is appended to each element in variable and then concatenated into a space separated string.

It is also possible to apply an operation to each element in an array by using <op|variable>, where op is the operation to apply. Currently, it is not possible to specify operations in configuration files, you can only use the ones built into mymake. Operations currently supported are:

• title: treat the element as a path and extract the file or directory name (eg. src/foo.txt gives foo.txt).
• titleNoExt: same as title, but the file extension is removed as well.
• noExt: remove the file type from a path.
• path: format the element as a path for the current operating system. For example: src/foo.txt evaluates to src\foo.txt on Windows.
• buildpath: make the element into a path inside the build path.
• execpath: make the element int a path inside the bin path.
• parent: evaluates to the parent directory of the path. If no parent is given (eg. only a file-name), the element is removed from the array.
• if: make all elements empty. This can be used to test if a variable contains a value and then include something.

It is also possible to combine these two operations, like: <prefix*path|files>. Stars have priority over pipes, and expressions are evaluated left-to-right.

There are two variables that mymake generate automatically:

• includes: equivalent to <includeCl*path|include>, ie. all include paths required by the project prefixed by any flags needed.
• libs: equivalent to <libraryCl*path|library> <localLibraryCl*path|library>, ie. all libraries required by the project.

Targets

The goal of a target is to produce the file <execDir>/<output>.<outputExt>. To do this, mymake starts by looking at all files in input (which, among others are the files specified on the command line) and adding them to the set of files to compile. For each file in the set, mymake then finds all included files. Both directly included files and indirectly included files are found. After that, mymake finds corresponding implementation files (of any type in <ext>) and adds them to the set of files to compile as well. When this yields no more files, mymake compiles them one by one if necessary (just like make would do). Finally, it link all files together to the output file.

To do this, mymake considers the global ~/.mymake-file as well as any local .mymake-file, in that order. Therefore, any assignments in the local file can overwrite global settings.

Projects

A project is a collection of targets that depend on each other in some way. In mymake, a project is represented by a .myproject-file in one directory and optional .mymake-files in each subdirectory that is representing another project. The project configuration follows the same syntax as the target configurations, but the project configuration has two special sections: build and deps.

The build section contains information about which options should be passed to each project. When building a project, any options passed on the command line are not automatically passed on to the targets. Instead you have to specify which options are needed by assigning a variable with the same name as the target any options needed like so:

[build]
main+=lib
main+=debug

There is also a special target named all. All options in this target are used for all targets.

The deps section contains information about target dependencies. By default, mymake tries to figure out which targets depend on each other by looking at the includes. Sometimes this is not enough, since a project may need the result of another project as a pre-processing step or similar. Therefore, mymake allows you to specify any dependencies explicitly.

If one target results in a library, it is convenient to set the variable linkOutput to yes for that target. Mymake will then add the output of the library target to the library variable of any targets that depend on it. See the testproj directory for an example.

Of course, the build and deps sections are nothing but regular options, and can therefore be combined with other options as well:

[build,windows]
main+=foo

Skipping output

On most command lines (in particular, compile and link) it is possible to prepend the string <n>! where <n> is an integer to cause mymake to skip that many lines of stdout of the started process. This can be used to ignore banners and the likes from the compiler. For example, cl.exe outputs the name of the compiled file as its first line. This can be omitted by prepending 1! to the compile command line.