Native Windows support (#277)

* Compile project on Windows

* Fix compilation and run-time issues

* Fix runtime errors

The problem was with reading files: file paths are different on Unix and Windows.

* Set library extension to .pyd

* Add platform specific instructions. Build engine using pip

Use variable length arrays on platforms other than Windows

Build engine using pip on Windows

Add more platform-specific instructions

* Prepare for PR: add development install, update guides

Improve finding Boost Python on Linux. Prepare for PR.

Include source files with gfootball_engine

Add installation instructions for Windows

Add CMakeModules with data_files

Use MANIFEST.in to include additional files

Remove unnecessary call to Shutdown SDL: it is being called in blunted.cpp on Exit

Use platform-specific logic for copying fonts and building

Allow development (-e) install on Windows

Fix development (-e) install. Update README

* * Add GitHub workflow to publish wheels to PyPI
* Use vcpkg manifests to install C++ dependencies
* Fix tests to run on Windows
* Delete Simple Vertex Buffer during Exit (not in the destructor)
* Update MANIFEST.in to include necessary files
* Add requirements.txt
* Update compile engine doc

* Install psutil with a separate command in Docker

* Update CHANGELOG and docs

* Finalize PR changes

.dockerignore

@@ -1,3 +1,6 @@
-third_party/gfootball_engine/build
+third_party/gfootball_engine/build*/
 build/
 gfootball.egg-info
+football-env*
+.vs
+.idea

.github/workflows/publish-wheels.yml

@@ -0,0 +1,111 @@
+name: Build and Publish Python Wheels
+
+on:
+  release:
+    types: [published]
+
+jobs:
+  build-windows:
+    runs-on: windows-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: [ 3.6, 3.7, 3.8, 3.9 ]
+        architecture: ['x86', 'x64']
+
+    env:
+      VCPKG_ROOT: "${{ github.workspace }}/vcpkg"
+
+    steps:
+    - name: Check out football repository
+      uses: actions/checkout@v2
+      with:
+        path: football
+
+    - name: Set up Python
+      uses: actions/setup-python@v2
+      with:
+        python-version: ${{ matrix.python-version }}
+        architecture: ${{ matrix.architecture }}
+
+    - name: Upgrade pip and essential dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install -U build setuptools psutil wheel
+        pip install -r requirements.txt
+      working-directory: football
+
+    - name: Checkout vcpkg
+      uses: actions/checkout@v2
+      with:
+        repository: microsoft/vcpkg
+        ref: 025e564979cc01d0fbc5c920aa8a36635efb01bb
+        path: vcpkg
+        fetch-depth: 0
+
+     # Uncomment to speed up building by caching C++ dependencies. Requires `python -m build --wheel` in the next step.
+#    - name: Restore vcpkg_installed cache
+#      uses: actions/cache@v2
+#      with:
+#        path: ${{ github.workspace }}/football/third_party/gfootball_engine/build_win/vcpkg_installed
+#        key: vcpkg-${{ runner.os }}-py${{ matrix.python-version }}-${{ matrix.architecture }}-${{ hashFiles('football/third_party/gfootball_engine/vcpkg_manifests/py3*/vcpkg.json') }}
+
+    - name: Build package
+      # `python -m build` without any arguments is used to check if the wheel can be built from source (sdist):
+      #  the tool copies source code to the random temp directory, and proceeds with building.
+      #  Doing so insures that MANIFEST.in contains all the required files to build engine from source.
+      #  The downside is that we can't cache vcpkg_installed packages, since they'll be located in the random temp directory.
+      #  It increases workflow running time from ~15 minutes to ~40 minutes. On the other hand, cache is valid only for 7 days,
+      #  and releases are coming not so often anyway.
+      # To build in-place and take advantage of vcpkg_installed cache use `python -m build --wheel` instead.
+      run: python -m build
+      working-directory: football
+
+    - name: Install the package using the wheel
+      run: python -m pip install temp-gfootball --no-index --find-links=dist
+      working-directory: football
+
+    - name: Run the tests
+      run: FOR %%f IN (gfootball\env\*test.py) DO ( call python %%f & if errorlevel 1 exit /B 1 )
+      env:
+        UNITTEST_IN_DOCKER: 1
+      shell: cmd
+      working-directory: football
+
+    - uses: actions/upload-artifact@v2
+      with:
+        name: windows-wheel-py-${{ matrix.python-version }}-${{ matrix.architecture }}
+        path: ${{ github.workspace }}/football/dist/*.whl
+        retention-days: 7
+
+  upload-wheels:
+    needs: [build-windows]
+    runs-on: ubuntu-latest
+    steps:
+    - name: Check out football repository
+      uses: actions/checkout@v2
+
+    - name: Set up Python
+      uses: actions/setup-python@v2
+      with:
+        python-version: '3.x'
+
+    - name: Install dependencies
+      run: python -m pip install --upgrade pip setuptools wheel psutil build
+
+    - name: Build sdist
+      run: python -m build --sdist
+
+    - name: Download artifacts
+      uses: actions/download-artifact@v2
+      with:
+        path: ${{ github.workspace }}/artifacts
+
+    - name: Move wheels to dist directory
+      run:  mv artifacts/**/*.whl dist
+
+    - name: Publish package
+      uses: pypa/gh-action-pypi-publish@27b31702a0e7fc50959f5ad993c78deac1bdfc29
+      with:
+        user: __token__
+        password: ${{ secrets.PYPI_API_TOKEN }}

.gitignore

@@ -1,17 +1,18 @@
-third_party/gfootball_engine/build
+third_party/gfootball_engine/build*
 build/
 gfootball.egg-info
 __pycache__
 *.a
 *.so
 *.dylib
-CmakeCache.txt
+C[M,m]akeCache.txt
 third_party/gfootball_engine/CMakeFiles/
 third_party/gfootball_engine/Makefile
 third_party/gfootball_engine/cmake_install.cmake
 third_party/gfootball_engine/gameplayfootball
 footballenv/
-football-env/
+football-env*/
 cmake-build-debug/
 .idea
 .vs
+vcpkg_installed/

CHANGELOG

@@ -5,6 +5,10 @@ should not change, as modifications made to the environment are either
 new features or backward compatible bug fixes. We will maintain vX branches
 pointing at the most recent vX.Y.
 
+v2.10
+- Windows support.
+- Development mode (pip install -e .)
+
 v2.9
 - Drop hard dependency on Tensorflow.
 - Make installation easier, support for Ubuntu 20.04.

Dockerfile

@@ -6,9 +6,10 @@ RUN apt-get update && apt-get --no-install-recommends install -yq git cmake buil
   libgl1-mesa-dev libsdl2-dev \
   libsdl2-image-dev libsdl2-ttf-dev libsdl2-gfx-dev libboost-all-dev \
   libdirectfb-dev libst-dev mesa-utils xvfb x11vnc \
-  libsdl-sge-dev python3-pip
+  python3-pip
 
-RUN python3 -m pip install --upgrade pip setuptools psutil
+RUN python3 -m pip install --upgrade pip setuptools wheel
+RUN python3 -m pip install psutil
 
 COPY . /gfootball
 RUN cd /gfootball && python3 -m pip install .

Dockerfile_examples

@@ -7,9 +7,9 @@ RUN apt-get update && apt-get --no-install-recommends install -yq git cmake buil
   libgl1-mesa-dev libsdl2-dev \
   libsdl2-image-dev libsdl2-ttf-dev libsdl2-gfx-dev libboost-all-dev \
   libdirectfb-dev libst-dev mesa-utils xvfb x11vnc \
-  libsdl-sge-dev python3-pip
+  python3-pip
 
-RUN python3 -m pip install --upgrade pip setuptools
+RUN python3 -m pip install --upgrade pip setuptools wheel
 RUN pip3 install --no-cache-dir psutil dm-sonnet==1.*
 RUN pip list | grep 'tensorflow ' || pip3 install --no-cache-dir tensorflow==1.15.*
 

MANIFEST.in

@@ -1,3 +1,16 @@
-include gfootball/build_game_engine.sh
-recursive-include third_party/gfootball_engine/ *
-recursive-include third_party/fonts/ *
+include gfootball/build_game_engine.*
+recursive-include gfootball/colabs *
+recursive-include gfootball/doc *
+recursive-include gfootball/eval_server/proto *
+recursive-include gfootball/examples *
+
+include Dockerfile .dockerignore CHANGELOG CONTRIBUTING.md run_docker_test.sh
+
+recursive-include third_party/fonts *
+recursive-include third_party/gfootball_engine *
+recursive-exclude third_party/gfootball_engine *.a libgame.so cmake_install.cmake Makefile
+prune third_party/gfootball_engine/CMakeFiles
+prune third_party/gfootball_engine/cmake-build-debug
+prune third_party/gfootball_engine/build*
+prune third_party/gfootball_engine/.vs
+prune third_party/gfootball_engine/.idea

README.md

@@ -36,50 +36,63 @@ Instructions are available [here](gfootball/doc/docker.md).
 
 #### 1. Install required packages
 #### Linux
-```
+```shell
 sudo apt-get install git cmake build-essential libgl1-mesa-dev libsdl2-dev \
 libsdl2-image-dev libsdl2-ttf-dev libsdl2-gfx-dev libboost-all-dev \
-libdirectfb-dev libst-dev mesa-utils xvfb x11vnc libsdl-sge-dev python3-pip
+libdirectfb-dev libst-dev mesa-utils xvfb x11vnc python3-pip
 
-python3 -m pip install --upgrade pip setuptools psutil
+python3 -m pip install --upgrade pip setuptools psutil wheel
 ```
 
-#### Mac OS X
+#### macOS
 First install [brew](https://brew.sh/). It should automatically install Command Line Tools.
 Next install required packages:
 
-```
+```shell
 brew install git python3 cmake sdl2 sdl2_image sdl2_ttf sdl2_gfx boost boost-python3
+
+python3 -m pip install --upgrade pip setuptools psutil wheel
 ```
 
-#### 2a. From PyPi package
+#### Windows
+Install [Git](https://git-scm.com/download/win) and [Python 3](https://www.python.org/downloads/).
+Update pip in the Command Line (here and for the **next steps** type `python` instead of `python3`)
+```commandline
+python -m pip install --upgrade pip setuptools psutil wheel
 ```
+
+#### 2. Install GFootball
+#### 2a. From PyPi package
+```shell
 python3 -m pip install gfootball
 ```
 
 #### 2b. Installing from sources using GitHub repository
 
-```
+```shell
 git clone https://github.com/google-research/football.git
 cd football
 ```
 
 Optionally you can use [virtual environment](https://docs.python.org/3/tutorial/venv.html):
 
-```
+```shell
 python3 -m venv football-env
 source football-env/bin/activate
 ```
 
-The last step is to build the environment:
+Next, build the game engine and install dependencies:
 
-```
+```shell
 python3 -m pip install .
 ```
 This command can run for a couple of minutes, as it compiles the C++ environment in the background.
 
+To compile the engine on Windows you have to install additional tools and set environment variables. See 
+[Compiling Engine](gfootball/doc/compile_engine.md#windows) for detailed instructions.
+
 #### 3. Time to play!
-```
+```shell
 python3 -m gfootball.play_game --action_set=full
 ```
 Make sure to check out the [keyboard mappings](#keyboard-mappings).
@@ -106,11 +119,11 @@ To quit the game press Ctrl+C in the terminal.
 ### Run training
 In order to run TF training, you need to install additional dependencies
 
-- Update PIP, so that tensorflow 1.15 is available: `python3 -m pip install --upgrade pip setuptools`
+- Update PIP, so that tensorflow 1.15 is available: `python3 -m pip install --upgrade pip setuptools wheel`
 - TensorFlow: `python3 -m pip install tensorflow==1.15.*` or
   `python3 -m pip install tensorflow-gpu==1.15.*`, depending on whether you want CPU or
   GPU version;
-- Sonnet: `python3 -m pip install dm-sonnet==1.*`;
+- Sonnet and psutil: `python3 -m pip install dm-sonnet==1.* psutil`;
 - OpenAI Baselines:
   `python3 -m pip install git+https://github.com/openai/baselines.git@master`.
 
@@ -131,7 +144,8 @@ In order to reproduce PPO results from the paper, please refer to:
 
 ## Playing the game
 
-Please note that playing the game is implemented through an environment, so human-controlled players use the same interface as the agents. One important implication is that there is a single action per 100 ms reported to the environment, which might cause a lag effect when playing.
+Please note that playing the game is implemented through an environment, so human-controlled players use the same interface as the agents.
+One important implication is that there is a single action per 100 ms reported to the environment, which might cause a lag effect when playing.
 
 
 ### Keyboard mappings
@@ -143,8 +157,8 @@ The game defines following keyboard mapping (for the `keyboard` player type):
 * `ARROW RIGHT` - run to the right.
 * `S` - short pass in the attack mode, pressure in the defense mode.
 * `A` - high pass in the attack mode, sliding in the defense mode.
-* `D` - shot in the the attack mode, team pressure in the defense mode.
-* `W` - long pass in the the attack mode, goalkeeper pressure in the defense mode.
+* `D` - shot in the attack mode, team pressure in the defense mode.
+* `W` - long pass in the attack mode, goalkeeper pressure in the defense mode.
 * `Q` - switch the active player in the defense mode.
 * `C` - dribble in the attack mode.
 * `E` - sprint.

gfootball/build_game_engine.bat

@@ -0,0 +1,29 @@
+@echo off
+rem This batch file compiles gfootball engine on Windows
+
+if "%VCPKG_ROOT%"=="" (
+    echo VCPKG_ROOT environment variable is missing
+    echo Execute: SET "VCPKG_ROOT=PATH_TO_VCPKG" e.g., SET "VCPKG_ROOT=C:\src\vcpkg\"
+    exit /b 1
+)
+if "%GENERATOR_PLATFORM%"=="" (
+    echo GENERATOR_PLATFORM environment variable is missing
+    echo Execute: SET GENERATOR_PLATFORM={x64 or Win32}, depending on the Python Interpreter
+    exit /b 1
+)
+if "%BUILD_CONFIGURATION%"=="" set BUILD_CONFIGURATION=Release
+
+pushd third_party\gfootball_engine
+
+if not exist build_win mkdir build_win
+if exist build_win\CMakeCache.txt del build_win\CMakeCache.txt
+if exist build_win\%BUILD_CONFIGURATION% rmdir /s /q build_win\%BUILD_CONFIGURATION%
+
+pushd build_win
+cmake .. -A%GENERATOR_PLATFORM%
+if errorlevel 1 exit /B 1
+
+cmake --build . --parallel --config %BUILD_CONFIGURATION%
+if errorlevel 1 exit /B 1
+
+popd && popd