Merge branch 'master' into pramodk/ion-var-as-constant

.github/ISSUE_TEMPLATE/release-patch.md

@@ -0,0 +1,95 @@
+---
+name: NEURON Patch Release
+about: Create a NEURON Patch release for an existing branch.
+title: 'NEURON [x.y.z] patch release'
+labels: 'release'
+assignees: ''
+
+---
+
+Action items
+============
+
+Pre-release
+---
+- [ ] Create a cherrypicks branch where all commits go into new release and open a PR against `release/x.y` branch
+- [ ] Create CoreNEURON tag on the `release/x.y` branch after cherrypicking required commits, update semantic version in its `CMakeLists.txt`, tag it & update submodule in cherrypicks PR
+- [ ] Look out for ModelDB regressions by manually submitting and analyzing [nrn-modeldb-ci](https://github.com/neuronsimulator/nrn-modeldb-ci/actions/workflows/nrn-modeldb-ci.yaml?query=event%3Aschedule++) for the cherrypicks branch vs previous version
+- [ ] Update cherrypicks PR:
+  - [ ] Update semantic version in `CMakeLists.txt`
+  - [ ] Update changelog below and agree on it with everyone; then commit it to `docs/changelog` in the cherrypicks PR (copy structure as-is)
+  - [ ] Update `docs/index.rst` accordingly with the new `.pkg` and `.exe` links for `PKG installer` and `Windows Installer`
+
+Sanity checks
+---
+- [ ] After cherrypicks PR is merged, make sure GitHub, Azure and CircleCI builds pass for `release/x.y` branch
+- [ ] Run [nrn-build-ci](https://github.com/neuronsimulator/nrn-build-ci/actions/workflows/build-neuron.yml) for the `release/x.y` branch; see [nrn-build-ci guide](https://github.com/neuronsimulator/nrn-build-ci#azure-wheels-testing---manual-workflow) 
+- [ ] Activate ReadTheDocs build for `release/x.y` & make it hidden. Check docs are fine after build is done.
+- [ ] Run BBP Simulation Stack, CoreNEURON CI & other relevant tests
+- [ ] Build release wheels but WITHOUT upload ([see details](https://nrn.readthedocs.io/en/latest/install/python_wheels.html#publishing-the-wheels-on-pypi-via-azure))
+
+
+Releasing
+---
+- [ ] Create new release+tag on GitHub via [release workflow](https://github.com/neuronsimulator/nrn/actions/workflows/release.yml?query=workflow%3A%22NEURON+Release%22). Note that the GitHub release will be marked as pre-release.
+- [ ] Create, test and upload manual artifacts 
+  - [ ] MacOS package installer (manual task, ask Michael)
+  - [ ] arm64 wheels (manual task, check with Alex or Pramod)
+  - [ ] aarch64 wheels (use existing `release/x.y-aarch64` branch for this, see [guide](https://nrn.readthedocs.io/en/latest/install/python_wheels.html#publishing-the-wheels-on-pypi-via-circleci))
+- [ ] Publish the `x.y.z` wheels on PyPI; see [wheel publishing instructions](https://nrn.readthedocs.io/en/latest/install/python_wheels.html#publishing-the-wheels-on-pypi-via-azure)
+- [ ] Once wheels are published, activate the `x.y.z` tag on ReadTheDocs
+- [ ] Upload Windows installer from the wheels publishing Azure run (to get correct tag)
+- [ ] Publish release on GitHub (edit https://github.com/neuronsimulator/nrn/releases/tag/x.y.z and un-tick the pre-release checkbox) 
+
+
+Post-release
+---
+- [ ] Deactivate ReadTheDocs build for `release/x.y`
+- [ ] Let people know :rocket:
+- [ ] Cherrypick changelog and installer links to `master`
+
+
+Changelog
+======
+
+# NEURON X.Y
+
+## [x.y.z]
+_Release Date_ : DD-MM-YYYY
+
+
+### What's New
+* [List new features/support added]
+* .....
+
+
+### Bug Fixes
+* [List the important bug fixes]
+* ...
+
+
+### Improvements /  Other Changes
+* [List the improvements made in the new release and any other changes]
+* ...
+
+
+For the complete list of commits check  [GitHub Issue #[GH_no.]](https://github.com/neuronsimulator/nrn/issues/#[GH_no.])
+
+ReadTheDocs sneak peek
+======================
+* https://nrn.readthedocs.io/en/release-x.y
+
+Commits going into x.y.z
+========================
+
+[given `a.b.c` is the last release:]
+
+Since [a.b.c], with:
+```bash
+git log --pretty=format:"%h : %s" a.b.c..release/x.y
+```
+we get:
+
+- [ ] commit 1
+- [ ] commit 2
+- [ ] ...

.github/ISSUE_TEMPLATE/release.md

@@ -1,5 +1,5 @@
 ---
-name: NEURON Release
+name: NEURON Major/Minor Release
 about: Create a NEURON release for a new branch from master.
 title: 'NEURON [x.y.z] release'
 labels: 'release'
@@ -13,7 +13,7 @@ Action items
 Pre-release
 ---
 - [ ] Look out for ModelDB regressions by analyzing [nrn-modeldb-ci last version vs nightly reports](https://github.com/neuronsimulator/nrn-modeldb-ci/actions/workflows/nrn-modeldb-ci.yaml?query=event%3Aschedule++)
-- [ ] Create CoreNEURON release & update submodule
+- [ ] Create CoreNEURON release branch, update semantic version in `CMakeLists.txt`, tag it & update submodule in NEURON
 
 Sanity checks
 ---
@@ -26,23 +26,29 @@ Sanity checks
 
 Releasing
 ---
+- [ ] Update semantic version in `CMakeLists.txt`
 - [ ] Update changelog below and agree on it with everyone; then commit it to `docs/changelog` (copy structure as-is)
 - [ ] Update `docs/index.rst` accordingly with the new `.pkg` and `.exe` links for `PKG installer` and `Windows Installer`
 - [ ] Run the ReadTheDocs build again for `release-x.y`, make sure the build passes and inspect the Changelog page.
-- [ ] Create new release+tag on GitHub via [release workflow](https://github.com/neuronsimulator/nrn/actions/workflows/release.yml?query=workflow%3A%22NEURON+Release%22)
-- [ ] Create, test and upload manual artifacts (MacOS package installers, arm64/aarch64 wheels, ...)
+- [ ] Create new release+tag on GitHub via [release workflow](https://github.com/neuronsimulator/nrn/actions/workflows/release.yml?query=workflow%3A%22NEURON+Release%22). Note that the GitHub release will be marked as pre-release.
+- [ ] Create, test and upload manual artifacts 
+  - [ ] MacOS package installer (manual task, ask Michael)
+  - [ ] arm64 wheels (manual task, check with Alex or Pramod)
+  - [ ] aarch64 wheels (create a `release/x.y-aarch64` branch for this, see [guide](https://nrn.readthedocs.io/en/latest/install/python_wheels.html#publishing-the-wheels-on-pypi-via-circleci))
 - [ ] Publish the `x.y.z` wheels on Pypi; see [wheel publishing instructions](https://nrn.readthedocs.io/en/latest/install/python_wheels.html#publishing-the-wheels-on-pypi-via-azure)
 - [ ] Once wheels are published, activate the `x.y.z` tag on ReadTheDocs
 - [ ] Upload Windows installer from the wheels publishing Azure run (to get correct tag)
-- [ ] Publish release on GitHub (edit https://github.com/neuronsimulator/nrn/releases/tag/x.y.z)
+- [ ] Publish release on GitHub (edit https://github.com/neuronsimulator/nrn/releases/tag/x.y.z and un-tick the pre-release checkbox)
 
 
 Post-release
 ---
-- [ ] Tag `master` with `x.(y+1).dev` to mark the start of a new development cycle
-- [ ] Deactivate ReadTheDocs build for release/x.y
+- [ ] To mark the start of a new development cycle, tag `master` as follows:
+  - minor version: `x.(y+1).dev` 
+  - major version: `(x+1).0.dev`
+- [ ] Deactivate ReadTheDocs build for `release/x.y`
 - [ ] Let people know :rocket:
-
+- [ ] Cherrypick changelog and installer links to `master`
 
 Changelog
 ======

CMakeLists.txt

@@ -577,22 +577,29 @@ if(MINGW)
   add_subdirectory(src/mswin)
 endif()
 
-# =============================================================================
 # Collect the environment variables that are needed to execute NEURON from the build directory. This
-# is used when configuring tests, and when building documentation targets. TODO: be more careful
-# about trailing colons?
-# =============================================================================
-set(NRN_RUN_FROM_BUILD_DIR_ENV
-    "LD_LIBRARY_PATH=${PROJECT_BINARY_DIR}/lib:$ENV{LD_LIBRARY_PATH}"
-    "NEURONHOME=${PROJECT_BINARY_DIR}/share/nrn" "NRNHOME=${PROJECT_BINARY_DIR}"
-    "PATH=${PROJECT_BINARY_DIR}/bin:$ENV{PATH}")
+# is used when configuring tests, and when building documentation targets.
+function(prepend_to_var var new_path)
+  set(old_list "${NRN_RUN_FROM_BUILD_DIR_ENV}")
+  set(new_val "${var}=${new_path}")
+  if(DEFINED ENV{${var}} AND NOT "$ENV{${var}}" STREQUAL "")
+    string(APPEND new_val ":$ENV{${var}}")
+  endif()
+  list(APPEND old_list "${new_val}")
+  set(NRN_RUN_FROM_BUILD_DIR_ENV
+      "${old_list}"
+      PARENT_SCOPE)
+endfunction()
+set(NRN_RUN_FROM_BUILD_DIR_ENV "NEURONHOME=${PROJECT_BINARY_DIR}/share/nrn"
+                               "NRNHOME=${PROJECT_BINARY_DIR}")
+prepend_to_var(PATH "${PROJECT_BINARY_DIR}/bin")
+prepend_to_var(LD_LIBRARY_PATH "${PROJECT_BINARY_DIR}/lib")
 if(NRN_ENABLE_CORENEURON AND NOT coreneuron_FOUND)
   # CoreNEURON is enabled *and* is being built internally
   list(APPEND NRN_RUN_FROM_BUILD_DIR_ENV "CORENRNHOME=${PROJECT_BINARY_DIR}")
 endif()
 if(NRN_ENABLE_PYTHON)
-  list(APPEND NRN_RUN_FROM_BUILD_DIR_ENV
-       PYTHONPATH=${PROJECT_BINARY_DIR}/lib/python:${PROJECT_SOURCE_DIR}/test/rxd:$ENV{PYTHONPATH})
+  prepend_to_var(PYTHONPATH "${PROJECT_BINARY_DIR}/lib/python:${PROJECT_SOURCE_DIR}/test/rxd")
 endif()
 
 if(NRN_ENABLE_DOCS)

azure-pipelines.yml

@@ -186,10 +186,11 @@ stages:
         inputs:
           secureFile: 'readline7.0.tar.gz'
 
-      # 10.14 is required for full C++17 support, see:
-      # https://cibuildwheel.readthedocs.io/en/stable/cpp_standards
+      # 10.14 is required for full C++17 support according to
+      # https://cibuildwheel.readthedocs.io/en/stable/cpp_standards, but it
+      # seems that 10.15 is actually needed for std::filesystem::path.
       - script: |
-          export MACOSX_DEPLOYMENT_TARGET=10.14
+          export MACOSX_DEPLOYMENT_TARGET=10.15
           export PATH=/usr/local/opt/flex/bin:/usr/local/opt/bison/bin:$PATH
           export SDKROOT=$(xcrun --sdk macosx --show-sdk-path)
           export NRN_BUILD_FOR_UPLOAD=1

cmake/ConfigFileSetting.cmake

@@ -60,11 +60,16 @@ endif()
 # separated 'option=value' where value differs from the default value.
 # ~~~
 set(neuron_config_args "cmake option default differences:")
+set(neuron_config_args_all)
 foreach(_name ${NRN_OPTION_NAME_LIST})
   if(NOT ("${${_name}}" STREQUAL "${${_name}_DEFAULT}"))
     string(APPEND neuron_config_args " '${_name}=${${_name}}'")
   endif()
+  string(REPLACE ";" "\\;" escaped_value "${${_name}}")
+  string(REPLACE "\"" "\\\"" escaped_value "${escaped_value}")
+  list(APPEND neuron_config_args_all "{\"${_name}\", \"${escaped_value}\"}")
 endforeach()
+string(JOIN ",\n  " neuron_config_args_all ${neuron_config_args_all})
 
 # =============================================================================
 # Platform specific options (get expanded to comments)

cmake_nrnconf.h.in

@@ -365,6 +365,12 @@
  */
 #if defined(__cplusplus)
 #include <ivstream.h>
+#include <string_view>
+namespace neuron::config {
+   constexpr std::string_view shared_library_prefix{"@CMAKE_SHARED_LIBRARY_PREFIX@"};
+   constexpr std::string_view shared_library_suffix{"@CMAKE_SHARED_LIBRARY_SUFFIX@"};
+   constexpr std::string_view system_processor{"@CMAKE_SYSTEM_PROCESSOR@"};
+}
 #endif
 
 #ifdef MINGW

setup.py

@@ -483,19 +483,39 @@ def mac_osx_setenv():
     log.info("Setting SDKROOT=%s", sdk_root)
     os.environ["SDKROOT"] = sdk_root
 
-    # Match Python OSX framework
+    # Extract the macOS version targeted by the Python framework
     py_osx_framework = extract_macosx_min_system_version(sys.executable)
-    if py_osx_framework is None:
-        py_osx_framework = [10, 9]
-    if py_osx_framework[1] > 9:
-        log.warn(
-            "[ WARNING ] You are building a wheel with a Python built"
-            " for a recent MACOS version (from brew?). Your wheel won't be portable."
-            " Consider using an official Python build from python.org"
+
+    def fmt(version):
+        return ".".join(str(x) for x in version)
+
+    if "MACOSX_DEPLOYMENT_TARGET" in os.environ:
+        # Don't override the value if it is set explicitly, but try and print a
+        # helpful message
+        explicit_target = tuple(
+            int(x) for x in os.environ["MACOSX_DEPLOYMENT_TARGET"].split(".")
         )
-    macos_target = "%d.%d" % tuple(py_osx_framework[:2])
-    log.info("Setting MACOSX_DEPLOYMENT_TARGET=%s", macos_target)
-    os.environ["MACOSX_DEPLOYMENT_TARGET"] = macos_target
+        if py_osx_framework is not None and explicit_target > py_osx_framework:
+            log.warn(
+                "You are building wheels for macOS >={}; this is more "
+                "restrictive than your Python framework, which supports "
+                ">={}".format(fmt(explicit_target), fmt(py_osx_framework))
+            )
+    else:
+        # Target not set explicitly, set MACOSX_DEPLOYMENT_TARGET to match the
+        # Python framework, or 10.9 if the version targeted by the framework
+        # cannot be determined
+        if py_osx_framework is None:
+            py_osx_framework = (10, 9)
+        if py_osx_framework > (10, 9):
+            log.warn(
+                "You are building a wheel with a Python built for macOS >={}. "
+                "Your wheel won't run on older versions, consider using an "
+                "official Python build from python.org".format(fmt(py_osx_framework))
+            )
+        macos_target = "%d.%d" % tuple(py_osx_framework[:2])
+        log.warn("Setting MACOSX_DEPLOYMENT_TARGET=%s", macos_target)
+        os.environ["MACOSX_DEPLOYMENT_TARGET"] = macos_target
 
 
 if __name__ == "__main__":

share/lib/python/neuron/__init__.py

@@ -142,6 +142,12 @@
 h = hoc.HocObject()
 version = h.nrnversion(5)
 __version__ = version
+
+# Initialise neuron.config.arguments
+from neuron import config
+
+config._parse_arguments(h)
+
 _original_hoc_file = None
 if not hasattr(hoc, "__file__"):
     # first try is to derive from neuron.__file__

share/lib/python/neuron/config.py

@@ -0,0 +1,35 @@
+def _convert_value(key, value):
+    """Convert a string representing a CMake variable value into a Python value.
+
+    This does some basic conversion of values that CMake interprets as boolean
+    values into Python's True/False, and includes some special cases for
+    variables that are known to represent lists. See also:
+    https://cmake.org/cmake/help/latest/command/if.html#basic-expressions.
+    """
+    if key.upper() in {"NRN_ENABLE_MODEL_TESTS"}:
+        return tuple(value.split(";"))
+    elif value.upper() in {"ON", "YES", "TRUE", "Y"}:
+        return True
+    elif value.upper() in {"OFF", "NO", "FALSE", "N"}:
+        return False
+    try:
+        return int(value)
+    except ValueError:
+        return value
+
+
+def _parse_arguments(h):
+    """Map the C++ structure neuron::config::arguments into Python.
+
+    The Python version is accessible as neuron.config.arguments.
+    """
+    global arguments
+    arguments = {}
+    num_keys_double = h.nrn_num_config_keys()
+    num_keys = int(num_keys_double)
+    assert float(num_keys) == num_keys_double
+    for i in range(num_keys):
+        key = h.nrn_get_config_key(i)
+        val = h.nrn_get_config_val(i)
+        assert key not in arguments
+        arguments[key] = _convert_value(key, val)

share/lib/python/neuron/rxd/region.py

@@ -108,11 +108,9 @@ def add_species(self, species_set):
             if isinstance(s, ParameterOnRegion):
                 if s._species() and s._species not in self._react_params:
                     self._react_params.append(s._species)
-            elif (
-                isinstance(s._species(), Parameter)
-                and s._species not in self._react_params
-            ):
-                self._react_params.append(weakref.ref(s))
+            elif isinstance(s, Parameter):
+                if s._species not in self._react_params:
+                    self._react_params.append(s._species)
             elif isinstance(s, SpeciesOnRegion):
                 if s._species() and s._species not in self._react_species:
                     self._react_species.append(s._species)

share/lib/python/neuron/rxd/rxd.py

@@ -1358,6 +1358,12 @@ def localize_index(creg, rate):
                 r = rptr()
                 if isinstance(r, rate.Rate):
                     s = r._species()
+                    if s._id in creg._params_ids:
+                        warn(
+                            "Parameters values are fixed, %r will not change the value of %r"
+                            % (r, s)
+                        )
+                        continue
                     species_id = creg._species_ids[s._id]
                     for reg in creg._react_regions[rptr]:
                         if reg() in r._rate:
@@ -1459,7 +1465,7 @@ def localize_index(creg, rate):
                         summed_mults = collections.defaultdict(lambda: 0)
                         for (mult, sp) in zip(r._mult, r._sources + r._dests):
                             summed_mults[creg._species_ids.get(sp()._id)] += mult
-                        for idx in sorted(summed_mults.keys()):
+                        for idx in sorted([k for k in summed_mults if k is not None]):
                             operator = "+=" if species_ids_used[idx][region_id] else "="
                             species_ids_used[idx][region_id] = True
                             fxn_string += "\n\trhs[%d][%d] %s (%g) * rate;" % (