Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No executor #1112

Merged
merged 70 commits into from
Nov 5, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
217f9ff
move more away from executor
Christian-B Aug 3, 2021
8bd3d8d
_execute_extra_load_algorithms
Christian-B Aug 4, 2021
3c948d7
fix spelling
Christian-B Aug 5, 2021
418e8bf
use FecExecutor
Christian-B Aug 20, 2021
ddf0d0f
cfg compressor moved to fec
Christian-B Aug 20, 2021
715a9bc
flake8
Christian-B Aug 23, 2021
d887cb7
merged in master
Christian-B Aug 23, 2021
b9be05c
fixes found by pylint
Christian-B Aug 23, 2021
1260948
do load now check virtual each execute
Christian-B Aug 23, 2021
f1f65ad
_execute_graph_data_specification_writer
Christian-B Aug 23, 2021
06cae9f
loading_algorithms now compressor
Christian-B Aug 25, 2021
416ff8e
remove unused import
Christian-B Aug 25, 2021
072b8cc
_do_extra_mapping_algorithms
Christian-B Aug 25, 2021
8958db4
reworking do_mapping in progress
Christian-B Aug 25, 2021
fe0a9a1
reworking do_mapping in progress
Christian-B Aug 26, 2021
d7219bd
merged in master
Christian-B Aug 26, 2021
28d889a
reworking do_mapping
Christian-B Aug 31, 2021
be0c532
new cfg settings
Christian-B Aug 31, 2021
e6240d7
call the algorithm you cliam you will
Christian-B Sep 1, 2021
501b0d0
dont need same method twice
Christian-B Sep 1, 2021
0647016
remove unused cfg options
Christian-B Sep 1, 2021
7efb7a1
new cfg algorithm format
Christian-B Sep 1, 2021
c2e945b
cfg add_delay_supports
Christian-B Sep 1, 2021
b4aa24c
cdg delay_support_adder so it can be disabled
Christian-B Sep 1, 2021
3c61ef0
merged in master
Christian-B Sep 1, 2021
4732057
renamed to FecTimer
Christian-B Sep 2, 2021
cb8b270
simulator a global FecTimer variable
Christian-B Sep 2, 2021
5aedc86
call fix
Christian-B Sep 2, 2021
08a0e88
correct log message
Christian-B Sep 13, 2021
336cf4f
tidy up
Christian-B Sep 14, 2021
15b5212
fix _execute_splitter_partitioner
Christian-B Sep 14, 2021
6d4ba13
remove unused get_projections_data method
Christian-B Sep 14, 2021
74fe19a
remove abilty to add algotihms via sim.setup
Christian-B Sep 15, 2021
0740670
dsg_algorithm now via method not value
Christian-B Sep 15, 2021
bfa368e
see https://github.com/SpiNNakerManchester/sPyNNaker/issues/1081
Christian-B Sep 15, 2021
e058d56
alogithm timing provenance
Christian-B Sep 15, 2021
deb50de
fix refactor error!
Christian-B Sep 15, 2021
98929c7
remove unused imports
Christian-B Sep 16, 2021
2408dad
add preallocators
Christian-B Sep 20, 2021
c4c704b
RedundantPacketCountReport
Christian-B Sep 22, 2021
ee283a1
docs
Christian-B Sep 22, 2021
41a3eb1
remove the setUp which hides the BaseTestCase setup
Christian-B Sep 30, 2021
d975c26
test depends on monitor being enabled
Christian-B Sep 30, 2021
22860f9
remove unused import
Christian-B Sep 30, 2021
b4b1f0a
Merge branch 'testfix' into no_executor
Christian-B Sep 30, 2021
20e809f
merged in master
Christian-B Oct 4, 2021
eb798cb
Merge branch 'extract_iobuf_during_run' into no_executor
Christian-B Oct 5, 2021
cc0c893
merged in master
Christian-B Oct 5, 2021
0476608
merged in master
Christian-B Oct 5, 2021
c2570e7
merged in master
Christian-B Oct 5, 2021
8dd7788
worry about Energy reports later
Christian-B Oct 5, 2021
7c67b65
merged in master
Christian-B Oct 6, 2021
4c2656e
flake8
Christian-B Oct 6, 2021
abd7807
add delayed compression
Christian-B Oct 11, 2021
c57fce1
skip for https://github.com/SpiNNakerManchester/SpiNNFrontEndCommon/i…
Christian-B Oct 12, 2021
daa48be
DuckType in another algorithm
Christian-B Oct 12, 2021
c972c1c
flake8
Christian-B Oct 12, 2021
c667675
deal compressed but bot loaded yet
Christian-B Oct 12, 2021
3e22dde
merged in master
Christian-B Oct 18, 2021
7797e17
merged in master
Christian-B Oct 25, 2021
e05b460
remove old algorithm provenance
Christian-B Oct 25, 2021
2a5a2cf
reactivated timer provenance
Christian-B Oct 25, 2021
9f61500
better provenance and energy report
Christian-B Oct 26, 2021
6d0a0c5
merged in master
Christian-B Nov 1, 2021
b33dcf2
moved algorithm should never run choices out of FecTimer
Christian-B Nov 2, 2021
a6a92ed
flake8
Christian-B Nov 2, 2021
796c9fd
merged in master
Christian-B Nov 2, 2021
60d047a
hostname is local board ipaddress any
Christian-B Nov 3, 2021
ad14c85
merged in master
Christian-B Nov 3, 2021
a363093
oops forgot to change a test
Christian-B Nov 3, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
304 changes: 169 additions & 135 deletions spynnaker/pyNN/abstract_spinnaker_common.py

Large diffs are not rendered by default.

11 changes: 2 additions & 9 deletions spynnaker/pyNN/spynnaker.cfg
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
[Reports]
# If reportsEnabled is false, no text reports are written.
writeSynapticReport = False
# Note: graphviz is required to draw the graph
write_network_graph = False
# Unless specified write_network_graph is ignored for large graphs
Expand Down Expand Up @@ -38,13 +36,8 @@ drop_late_spikes = True
transfer_overhead_clocks = 200

[Mapping]
# Algorithms below - format is <algorithm_name>,<>

application_to_machine_graph_algorithms = SplitterReset,DelaySupportAdder,SpynnakerSplitterSelector,SpYNNakerSplitterPartitioner
machine_graph_to_machine_algorithms = EdgeToNKeysMapper,SpreaderPlacer,NerRouteTrafficAware,BasicTagAllocator,ProcessPartitionConstraints,ZonedRoutingInfoAllocator,BasicRoutingTableGenerator,RouterCollisionPotentialReport
machine_graph_to_virtual_machine_algorithms = EdgeToNKeysMapper,SpreaderPlacer,NerRouteTrafficAware,BasicTagAllocator,ProcessPartitionConstraints,ZonedRoutingInfoAllocator,BasicRoutingTableGenerator,PairCompressor
loading_algorithms = PairOnChipRouterCompression
#loading_algorithms = SpynnakerMachineBitFieldPairRouterCompressor
# Setting delay_support_adder to None will skip the adder
delay_support_adder = DelaySupportAdder

[Buffers]
# Host and port on which to receive buffer requests
Expand Down
40 changes: 2 additions & 38 deletions spynnaker8/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,7 @@
'external_devices', 'extra_models',
# Stuff that we define
'end', 'setup', 'run', 'run_until', 'run_for', 'num_processes', 'rank',
'reset', 'set_number_of_neurons_per_core', 'get_projections_data',
'reset', 'set_number_of_neurons_per_core',
'Projection',
'get_current_time', 'create', 'connect', 'get_time_step', 'get_min_delay',
'get_max_delay', 'initialize', 'list_standard_models', 'name',
Expand Down Expand Up @@ -255,26 +255,11 @@ def distance(src, tgt, mask=None, scale_factor=1.0, offset=0.0,
src, tgt, mask, scale_factor, offset, periodic_boundaries)


def get_projections_data(projection_data):
"""
:param projection_data: the projection to attributes mapping
:type projection_data:
dict(~.Projection, list(int) or tuple(int) or None)
:return: a extracted data object with get method for getting the data
:rtype: ExtractedData
"""
return globals_variables.get_simulator().get_projections_data(
projection_data)


def setup(timestep=_pynn_control.DEFAULT_TIMESTEP,
min_delay=_pynn_control.DEFAULT_MIN_DELAY,
max_delay=None,
graph_label=None,
database_socket_addresses=None, extra_algorithm_xml_paths=None,
extra_mapping_inputs=None, extra_mapping_algorithms=None,
extra_pre_run_algorithms=None, extra_post_run_algorithms=None,
extra_load_algorithms=None, time_scale_factor=None,
database_socket_addresses=None, time_scale_factor=None,
n_chips_required=None, n_boards_required=None, **extra_params):
""" The main method needed to be called to make the PyNN 0.8 setup. Needs\
to be called before any other function
Expand All @@ -294,21 +279,6 @@ def setup(timestep=_pynn_control.DEFAULT_TIMESTEP,
for the database notification protocol
:type database_socket_addresses:
iterable(~spinn_utilities.socket_address.SocketAddress)
:param extra_algorithm_xml_paths:
list of paths to where other XML are located
:type extra_algorithm_xml_paths: list(str) or None
:param extra_mapping_inputs: other inputs used by the mapping process
:type extra_mapping_inputs: dict(str, Any) or None
:param extra_mapping_algorithms:
other algorithms to be used by the mapping process
:type extra_mapping_algorithms: list(str) or None
:param extra_pre_run_algorithms: extra algorithms to use before a run
:type extra_pre_run_algorithms: list(str) or None
:param extra_post_run_algorithms: extra algorithms to use after a run
:type extra_post_run_algorithms: list(str) or None
:param extra_load_algorithms:
extra algorithms to use within the loading phase
:type extra_load_algorithms: list(str) or None
:param time_scale_factor: multiplicative factor to the machine time step
(does not affect the neuron models accuracy)
:type time_scale_factor: int or None
Expand Down Expand Up @@ -358,12 +328,6 @@ def setup(timestep=_pynn_control.DEFAULT_TIMESTEP,
# create the main object for all stuff related software
SpiNNaker(
database_socket_addresses=database_socket_addresses,
extra_algorithm_xml_paths=extra_algorithm_xml_paths,
extra_mapping_inputs=extra_mapping_inputs,
extra_mapping_algorithms=extra_mapping_algorithms,
extra_pre_run_algorithms=extra_pre_run_algorithms,
extra_post_run_algorithms=extra_post_run_algorithms,
extra_load_algorithms=extra_load_algorithms,
time_scale_factor=time_scale_factor, timestep=timestep,
min_delay=min_delay, graph_label=graph_label,
n_chips_required=n_chips_required,
Expand Down
24 changes: 0 additions & 24 deletions spynnaker8/spinnaker.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,9 +35,6 @@ class SpiNNaker(AbstractSpiNNakerCommon, pynn_control.BaseState):

def __init__(
self, database_socket_addresses,
extra_algorithm_xml_paths, extra_mapping_inputs,
extra_mapping_algorithms, extra_pre_run_algorithms,
extra_post_run_algorithms, extra_load_algorithms,
time_scale_factor, min_delay, graph_label,
n_chips_required=None, n_boards_required=None, timestep=0.1,
hostname=None):
Expand All @@ -58,21 +55,6 @@ def __init__(
# main pynn interface inheritance
pynn_control.BaseState.__init__(self)

# handle the extra load algorithms and the built in ones
built_in_extra_load_algorithms = list()
if extra_load_algorithms is not None:
built_in_extra_load_algorithms.extend(extra_load_algorithms)

# handle extra xmls and the ones needed by default
built_in_extra_xml_paths = list()
if extra_algorithm_xml_paths is not None:
built_in_extra_xml_paths.extend(extra_algorithm_xml_paths)

# handle the extra mapping inputs and the built in ones
built_in_extra_mapping_inputs = dict()
if extra_mapping_inputs is not None:
built_in_extra_mapping_inputs.update(extra_mapping_inputs)

front_end_versions = [("sPyNNaker8_version", _version.__version__)]
front_end_versions.append(("pyNN_version", pynn_version))
front_end_versions.append(("quantities_version", quantities_version))
Expand All @@ -82,12 +64,6 @@ def __init__(
# SpiNNaker setup
super(SpiNNaker, self).__init__(
database_socket_addresses=database_socket_addresses,
user_extra_algorithm_xml_path=built_in_extra_xml_paths,
user_extra_mapping_inputs=built_in_extra_mapping_inputs,
extra_mapping_algorithms=extra_mapping_algorithms,
user_extra_algorithms_pre_run=extra_pre_run_algorithms,
extra_post_run_algorithms=extra_post_run_algorithms,
extra_load_algorithms=built_in_extra_load_algorithms,
graph_label=graph_label, n_chips_required=n_chips_required,
n_boards_required=n_boards_required,
hostname=hostname, min_delay=min_delay,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,9 @@
# 1 atom of IF_curr_exp recording for 1s
max_sdram_allowed_per_chip = 507106

enable_advanced_monitor_support = True
enable_reinjection = True

[Buffers]
# Same as defaults but set here to show difference
use_auto_pause_and_resume = True
Expand Down
8 changes: 4 additions & 4 deletions spynnaker_integration_tests/test_debug_mode/check_debug.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,8 @@
from spinn_front_end_common.utilities.report_functions.\
routing_table_from_machine_report import _FOLDER_NAME as \
routing_tables_from_machine_report
from spinn_front_end_common.utilities.report_functions.energy_report \
import EnergyReport
# from spinn_front_end_common.utilities.report_functions.energy_report \
# import EnergyReport
from spinn_front_end_common.utilities.report_functions.board_chip_report \
import BoardChipReport
from spinn_front_end_common.utility_models import \
Expand All @@ -44,8 +44,8 @@ def debug(self, run_zero):
# pylint: disable=protected-access
reports = [
# write_energy_report
EnergyReport._DETAILED_FILENAME,
EnergyReport._SUMMARY_FILENAME,
# EnergyReport._DETAILED_FILENAME,
# EnergyReport._SUMMARY_FILENAME,
# write_text_specs = False
"data_spec_text_files",
# write_router_reports
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,6 @@ extract_iobuf_from_binary_types = IF_curr_exp.aplx,reverse_iptag_multicast_sourc
clear_iobuf_during_run = True

[Mapping]
machine_graph_to_machine_algorithms = OneToOnePlacer,NerRoute,BasicTagAllocator,EdgeToNKeysMapper,ProcessPartitionConstraints,MallocBasedRoutingInfoAllocator,BasicRoutingTableGenerator,RouterCollisionPotentialReport
placer= OneToOnePlacer
router = NerRoute
info_allocator= MallocBasedRoutingInfoAllocator
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,6 @@ extract_iobuf = True
extract_iobuf_from_cores = 0,0,1:0,0,3:1,1,1

[Mapping]
machine_graph_to_machine_algorithms = OneToOnePlacer,NerRoute,BasicTagAllocator,EdgeToNKeysMapper,ProcessPartitionConstraints,MallocBasedRoutingInfoAllocator,BasicRoutingTableGenerator,RouterCollisionPotentialReport
placer= OneToOnePlacer
router = NerRoute
info_allocator= MallocBasedRoutingInfoAllocator
4 changes: 3 additions & 1 deletion spynnaker_integration_tests/test_master_pop/spynnaker.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,6 @@ extract_iobuf_from_binary_types = IF_curr_exp.aplx,reverse_iptag_multicast_sourc
clear_iobuf_during_run = True

[Mapping]
machine_graph_to_machine_algorithms = KeyConstraintAdder,OneToOnePlacer,NerRoute,BasicTagAllocator,EdgeToNKeysMapper,ProcessPartitionConstraints,MallocBasedRoutingInfoAllocator,BasicRoutingTableGenerator,RouterCollisionPotentialReport
placer= OneToOnePlacer
router = NerRoute
info_allocator= MallocBasedRoutingInfoAllocator
Original file line number Diff line number Diff line change
Expand Up @@ -12,21 +12,28 @@
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
import os

from spinnaker_testbase import BaseTestCase
import spynnaker8 as sim
from spinn_front_end_common.utilities import FecTimer
from spinn_front_end_common.utilities.globals_variables import get_simulator
from spynnaker_integration_tests import test_master_pop
from .key_constraint_adder import KeyConstraintAdder


def fancy_do_load(graph_changed):
with FecTimer("WEIRD", "KeyConstraintAdder"):
simulator = get_simulator()
adder = KeyConstraintAdder()
adder(simulator.machine_graph)
simulator.do_load_normal(graph_changed)


def do_run():
sim.setup(1.0)
simulator = get_simulator()
simulator._xml_paths.append(
os.path.join(
os.path.dirname(test_master_pop.__file__),
"algorithms.xml"))
simulator.do_load_normal = simulator._do_load
simulator._do_load = fancy_do_load

# Break up the pre population as that is where delays happen
sim.set_number_of_neurons_per_core(sim.SpikeSourceArray, 50)
pop1 = sim.Population(100, sim.SpikeSourceArray([1]), label="pop1")
Expand Down
Original file line number Diff line number Diff line change
@@ -1,2 +1,5 @@
[Reports]
write_bit_field_compressor_report = True

[Mapping]
loading_algorithms = HostBasedBitFieldRouterCompressor,BitFieldCompressorReport
compressor = HostBasedBitFieldRouterCompressor
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
[Mapping]
loading_algorithms = OrderedCoveringOnChipRouterCompression
compressor = OrderedCoveringOnChipRouterCompression
Original file line number Diff line number Diff line change
@@ -1,2 +1,5 @@
[Reports]
write_bit_field_compressor_report = True

[Mapping]
loading_algorithms = SpynnakerMachineBitFieldOrderedCoveringCompressor,BitFieldCompressorReport
compressor = SpynnakerMachineBitFieldOrderedCoveringCompressor
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
[Mapping]
loading_algorithms = PairOnChipRouterCompression
compressor = PairOnChipRouterCompression
Original file line number Diff line number Diff line change
@@ -1,2 +1,5 @@
[Reports]
write_bit_field_compressor_report = True

[Mapping]
loading_algorithms = SpynnakerMachineBitFieldPairRouterCompressor,BitFieldCompressorReport
compressor = SpynnakerMachineBitFieldPairRouterCompressor
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
[Reports]
write_bit_field_compressor_report = True

[Mapping]
loading_algorithms = SpynnakerMachineBitFieldPairRouterCompressor,BitFieldCompressorReport
compressor = SpynnakerMachineBitFieldPairRouterCompressor

router_table_compression_with_bit_field_retry_count = 4
router_table_compression_with_bit_field_acceptance_threshold = 10
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@

import os
import numpy
import unittest
import spynnaker8 as p
from spinnaker_testbase import BaseTestCase
from spynnaker_integration_tests.scripts import SynfireRunner
Expand Down Expand Up @@ -64,5 +65,8 @@ def do_run(self):
num_chips = row["the_value"]
self.assertIsNotNone(num_chips, "power provenance was not written")

@unittest.skip(
"https://github.com/SpiNNakerManchester/"
"SpiNNFrontEndCommon/issues/866")
def test_power_monitoring(self):
self.runsafe(self.do_run)
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[Mapping]
# Algorithms below - format is <algorithm_name>,<>

machine_graph_to_machine_algorithms = EdgeToNKeysMapper,OneToOnePlacer,NerRoute,BasicTagAllocator,ProcessPartitionConstraints,MallocBasedRoutingInfoAllocator,BasicRoutingTableGenerator,RouterCollisionPotentialReport
placer= OneToOnePlacer
router = NerRoute
info_allocator= MallocBasedRoutingInfoAllocator
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
[Mapping]
# Algorithms below - format is <algorithm_name>,<>

machine_graph_to_machine_algorithms = EdgeToNKeysMapper,OneToOnePlacer,NerRoute,BasicTagAllocator,ProcessPartitionConstraints,ZonedRoutingInfoAllocator,BasicRoutingTableGenerator,RouterCollisionPotentialReport
placer= OneToOnePlacer
router = NerRoute
8 changes: 1 addition & 7 deletions unittests/test_cfg_checker.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,11 +30,5 @@ def test_config_checks(self):
spynnaker = os.path.join(parent, "spynnaker")
spy8 = os.path.join(parent, "spynnaker8")
spynnaker_it = os.path.join(parent, "spynnaker_integration_tests")
repeaters = [
"application_to_machine_graph_algorithms",
"machine_graph_to_machine_algorithms",
"machine_graph_to_virtual_machine_algorithms",
"loading_algorithms"]
run_config_checks(
directories=[spynnaker, spy8, spynnaker_it, unittests],
repeaters=repeaters)
directories=[spynnaker, spy8, spynnaker_it, unittests])
Original file line number Diff line number Diff line change
Expand Up @@ -15,17 +15,13 @@

import spynnaker8 as sim
from spinnaker_testbase import BaseTestCase
from spynnaker.pyNN.config_setup import unittest_setup

WEIGHT = 5
DELAY = 2


class TestIndexBasedProbabilityConnector(BaseTestCase):

def setUp(self):
unittest_setup()

def check_weights(self, projection, n, expression, allow_self_connections):
weights = projection.get(["weight"], "list")
pairs = [(s, d) for (s, d, _) in weights]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,5 @@ version = None
time_scale_factor = None

[Mapping]
machine_graph_to_virtual_machine_algorithms = RadialPlacer,NerRoute,BasicTagAllocator,EdgeToNKeysMapper,ProcessPartitionConstraints,MallocBasedRoutingInfoAllocator,BasicRoutingTableGenerator,MundyRouterCompressor
machine_graph_to_machine_algorithms = RadialPlacer,nerRoute,BasicTagAllocator,EdgeToNKeysMapper,ProcessPartitionConstraints,MallocBasedRoutingInfoAllocator,BasicRoutingTableGenerator
placer = RadialPlacer