Wire between Orchestrator - Provider #101

pepoviola · 2023-09-19T23:30:57Z

NOTE: The base branch is a merge of refactor/provider-trait and feat/network-spec.

This pr is still very WIP but is a good step forward and now we can spawn networks (relaychain only at the moment), some parts of the code still have some values hardcoded that we need to replace from config/spec.

A brief walkthrough

The orchestrator can be created by passing a filesystem (FS: Filesystem) and a provider (P: Provider) and have a public method spawn (that receive a NetworkConfig) that in the future will return the Network instance.
So, basically you can use it like this

    let fs = LocalFileSystem;
    let provider = NativeProvider::new(fs.clone());
    let orchestrator = Orchestrator::new(fs, provider);
    let _network = orchestrator.spawn(config).await?;

Inside the provider we compute the NwtworkSpec from the provided NetworkConfig, for this we have a few generators that acts as helper methods to create stuffs like chain-spec, node-identities, keystore, compute the command to run the node and other stuffs :)

Ones we have the network spec computed, we start building the needed artifacts (at this point only the chain-spec for the relaychain is implemented but in the future in this step we need to create the state/wasm and chain-spec for each parachain (and if is required register the para in the genesis of the relaychain).

Then we calculate the bootnodes (to spawn first and get the addr to use in others) and continue with the process of spawning the nodes (at the moment I using a simple try_join_all in the future tasks but we should have a method to control the concurrency as we have in the current version)

For each node at the moment we are creating a NetworkNode instance with the DynNode stored as inner and the spec of the node also stored in the same instance. Is not ideal to hold all that info for each node but I think is something to improve ones we have a complete working version.

And that all for now, there is much work to do until we have a real working version and the code is full of //TODO: ... to think through/implement. But to mention some things I'm sure we need to rethink:

IP/Ports, we should implement a view_point as we have in the current version. Since is not the same to require the ip/port from an external runner (like our test-runner running from our laptops) than from a local one (like our CI).
I had implemented a scoped filesystem to encapsulate all the fs operations inside of the isolated (a.k.a fake root) directory of the ns, this is not ideal and would be good to implement some erasure technique to just add the logic as blanket impl. inside the trait. (Note: the fs trait is not object-safe)
The config shared types and support are all over the places, we should just consolidate a common crate (or move all to support` to easily re-use them.
Logger, I will add the env_logger as part of a follow-up commit.
Naming, we should agree on some convention. I briefly talk this with @l0r1s today. I personally don't have some predilection but I do think we should follow a pattern. In example TransferedFile vs TransferFile vs FileToTransfer.

As I said at the begging, this is very much wip but will be good to have your insights :)

Thanks!

To run the example:

cargo run --package examples --example small_network_with_default

…r at the moment

…d mirror implementation of tokio::fs with tests

…rate

… related DTO

…ider struct threadsafe using RwLock and Arc, added implementation logic of resume/restart/pause, destroy and helpers for logs

Refactor/provider trait l0r1s

…ions bits, added implementations and tests

…, moved error next to provider traits

…om_node in NativeProvider, removed unused comments

crates/orchestrator/src/network_spec/node.rs

pepoviola · 2023-09-27T20:49:33Z

Sounds good, those are internals of the network_spec and not used outside, but we can work on the naming and expose a more clear name from the module. Not sure if we need to address this now but is a nice refactor if we have time. Thx!

…

On Wed, 27 Sep 2023 at 17:44 Loris Moulin ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In crates/orchestrator/src/network_spec/node.rs <#101 (comment)> : > + // otherwise use the default_args (can be empty). + let args: Vec<Arg> = if node_config.args().is_empty() { + chain_context + .default_args + .iter() + .map(|x| x.to_owned().clone()) + .collect() + } else { + node_config.args().into_iter().cloned().collect() + }; + + let (key, peer_id) = generators::identity::generate_for_node(node_config.name())?; + + let mut name = node_config.name().to_string(); + let seed = format!("//{}{name}", name.remove(0).to_uppercase()); + let accounts = generators::key::generate_for_node(&seed)?; Same here — Reply to this email directly, view it on GitHub <#101 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACY3B5LHWS5UP4TUIYF7TDX4SFZVANCNFSM6AAAAAA47A6JNQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Parity Technologies is a limited company registered in England and Wales with registered number 09760015 and registered office at c/o Ignition Law, 1 Sans Walk, London, England, EC1R 0LT. This message is intended solely for the addressee(s) and may contain confidential information. If you have received this message in error, please notify us, and immediately and permanently delete it. Do not use, copy or disclose the information contained in this message or in any attachment. For information about how we process data and monitor communications please see our Privacy policy (https://www.parity.io/privacy/ <https://www.parity.io/privacy/>)and for terms of use please see our Terms of Use policy (https://www.parity.io/terms/ <https://www.parity.io/terms/>).

crates/orchestrator/src/network_spec/relaychain.rs

Co-authored-by: Loris Moulin <45130584+l0r1s@users.noreply.github.com>

github-actions · 2023-09-27T23:13:20Z

Coverage after merging feat/glue/network-spec_provider into main

77.96%

Coverage Report

File	Stmts	Branches	Funcs	Lines	Uncovered Lines
crates/configuration/src
global_settings.rs	99.12%	100%	96.43%	99.50%
hrmp_channel.rs	95.88%	100%	89.47%	97.44%
network.rs	99.34%	100%	97.78%	99.49%	111, 116, 124, 80
parachain.rs	98.82%	100%	97.06%	99.02%	51–53, 67–70
relaychain.rs	98.99%	100%	95.38%	99.44%
utils.rs	100%	100%	100%	100%
crates/configuration/src/shared
errors.rs	50%	100%	50%	50%
helpers.rs	100%	100%	100%	100%
node.rs	98.32%	100%	92.68%	98.93%	124, 130
resources.rs	96.09%	100%	94.52%	96.43%	102, 114, 126–128, 147–150
types.rs	91.20%	100%	86.14%	92.69%	279–281, 350–352, 375–376, 378–380, 60–62
crates/orchestrator/src
errors.rs	0%	100%	0%	0%	8
lib.rs	0%	100%	0%	0%	100–102, 106–109, 111–117, 119, 123, 125–140, 143–168, 172–181, 184–192, 194, 196–200, 203–220, 222–227, 229–238, 241–247, 250, 253–262, 265–269, 272, 274–287, 289–291, 294–298, 300, 302–315, 317–320, 322–324, 335–336, 343, 350–352, 354–363, 365–366, 368–376, 378–385, 387–394, 396–399, 40, 400–407, 41, 410, 42–45, 47–52, 54–60, 62–67, 70, 72–86, 92–97, 99
network.rs	0%	100%	0%	0%	100–108, 111–117, 119–127, 130–154, 156, 158–167, 170, 172–176, 185–189, 191, 193, 195–199, 201, 203, 205–213, 215, 217, 220–225, 227–229, 231–233, 235–237, 239–241, 243–245, 247–249, 251–253, 32–40, 43, 56–70, 77–82, 85–94, 96, 98–99
network_spec.rs	96.25%	100%	91.67%	97.06%	37
spawner.rs	0%	100%	0%	0%	100–106, 108, 112–114, 117–119, 121, 127–147, 149, 151–167, 17, 37–50, 52–53, 55, 57–63, 67–68, 70, 73–93, 95–99
crates/orchestrator/src/generators
bootnode_addr.rs	0%	100%	0%	0%	10–28, 30, 33–34, 5–9
chain_spec.rs	3.90%	100%	8.33%	3.48%	100–111, 113–128, 130, 135–146, 149–155, 157–159, 162–190, 192–195, 197–199, 201–214, 216–218, 221–223, 225, 227–239, 241, 244–249, 251–252, 254–267, 269, 272–277, 279–280, 283–296, 298–303, 305–307, 309–311, 315–344, 347, 350–353, 356, 358–359, 361–376, 378, 38, 380–381, 386–427, 438–446, 456–464, 466–478, 480–488, 491–493, 497–500, 502–503, 508–519, 523–539, 541, 544–545, 548, 550–556, 558–563, 569–574, 576, 578, 580–587, 589–592, 595–596, 599–601, 603–604, 606–608, 610–612, 614–616, 619–621, 624–628, 630, 632–640, 642–643, 648–655, 657–667, 671–674, 676, 678, 680–685, 687–697, 699–715, 717–724, 726, 730–749, 75, 750–751, 753, 76, 762–769, 77, 770–772, 774, 776–794, 796–800, 802, 804, 84–87, 95–99
command.rs	0%	100%	0%	0%	100–111, 113–119, 121, 124–129, 131–132, 135–147, 149, 15, 152–154, 156–157, 16, 160–169, 17, 170–179, 18, 180–189, 19, 190–196, 198, 20, 200, 202–209, 21, 210–219, 22, 220–223, 225–228, 23, 230–232, 234–239, 24, 240–242, 245–249, 251–255, 257, 260–268, 270, 273–289, 292–297, 299, 302–304, 306–307, 310–332, 334, 336, 43–61, 63–65, 67–75, 78–96, 99
errors.rs	0%	100%	0%	0%	4
identity.rs	82.35%	100%	60%	86.21%	11
key.rs	84.48%	100%	50%	90%	31
keystore.rs	0%	100%	0%	0%	19–26, 28–31, 34–46, 48–55, 57–64, 66, 68–86
para_artifact.rs	24.64%	100%	40%	22.03%	44–46, 48–64, 66–69, 72–81, 83–88, 92–93
port.rs	84.62%	100%	50%	90.91%
crates/orchestrator/src/network
node.rs	0%	100%	0%	0%	17–29, 33–41, 5
parachain.rs	0%	100%	0%	0%	15–23, 25–37, 5
relaychain.rs	0%	100%	0%	0%	14–21, 5
crates/orchestrator/src/network_spec
node.rs	40.83%	100%	33.33%	41.23%	108, 146–150, 152–153, 155, 158, 160–163, 165–168, 173–178, 180, 183, 185–215, 217, 90–91, 93–96
parachain.rs	73.91%	100%	75%	73.86%	104, 122, 127–130, 133, 144–147, 150, 68–70, 73, 75–77, 83, 92, 97
relaychain.rs	91.80%	100%	83.33%	92.73%	67, 90, 97
crates/orchestrator/src/shared
types.rs	47.37%	100%	38.46%	52%	43–47, 50
crates/provider/src
lib.rs	25%	100%	0%	50%
native.rs	90.42%	100%	81.88%	91.54%	1020, 1024, 1057, 1066, 1087, 1094, 1125, 1152, 1176, 1181, 1209, 1235, 1245, 1252, 1269, 1319, 132, 1376–1378, 1423, 1499–1500, 1524, 155, 1565, 190–194, 262, 281, 294, 296, 303, 305–306, 319, 341, 355–357, 359–361, 363–365, 387–389, 391–393, 41, 434, 454, 49, 503, 552, 581–582, 587–589, 707, 799, 806, 811, 816, 821, 828, 841, 90, 912, 936, 941, 945, 974, 990
crates/provider/src/shared
types.rs	84.47%	100%	74.29%	86.41%	123, 140–146, 15–18, 230, 249–256, 79
crates/support/src
fs.rs	75%	100%	66.67%	80%
net.rs	0%	100%	0%	0%	10, 4–9
crates/support/src/fs
in_memory.rs	95.55%	100%	96.48%	95.37%	149, 271, 321, 372, 424, 432, 440, 448, 470, 478, 593, 615, 643, 660, 678, 685, 704, 731, 748, 79–82, 84, 86–89, 91, 96
local.rs	98.25%	100%	96.30%	98.75%	51, 8
crates/test-runner/src
lib.rs	100%	100%	100%	100%

github-actions · 2023-09-28T00:21:19Z

Coverage after merging feat/glue/network-spec_provider into main

78.42%

Coverage Report

File	Stmts	Branches	Funcs	Lines	Uncovered Lines
crates/configuration/src
global_settings.rs	99.12%	100%	96.43%	99.50%
hrmp_channel.rs	95.88%	100%	89.47%	97.44%
network.rs	99.34%	100%	97.78%	99.49%	111, 116, 124, 80
parachain.rs	98.82%	100%	97.06%	99.02%	51–53, 67–70
relaychain.rs	98.99%	100%	95.38%	99.44%
utils.rs	100%	100%	100%	100%
crates/configuration/src/shared
errors.rs	50%	100%	50%	50%
helpers.rs	100%	100%	100%	100%
node.rs	98.32%	100%	92.68%	98.93%	124, 130
resources.rs	96.09%	100%	94.52%	96.43%	102, 114, 126–128, 147–150
types.rs	91.20%	100%	86.14%	92.69%	279–281, 350–352, 375–376, 378–380, 60–62
crates/orchestrator/src
errors.rs	0%	100%	0%	0%	8
lib.rs	0%	100%	0%	0%	100–102, 106–109, 111–117, 119, 123, 125–140, 143–168, 172–181, 184–192, 194, 196–200, 203–220, 222–227, 229–238, 241–247, 250, 253–262, 265–269, 272, 274–287, 289–291, 294–298, 300, 302–315, 317–320, 322–324, 335–336, 343, 350–352, 354–363, 365–366, 368–376, 378–385, 387–394, 396–399, 40, 400–407, 41, 410, 42–45, 47–52, 54–60, 62–67, 70, 72–86, 92–97, 99
network.rs	0%	100%	0%	0%	100–108, 111–117, 119–127, 130–154, 156, 158–167, 170, 172–176, 185–189, 191, 193, 195–199, 201, 203, 205–213, 215, 217, 220–225, 227–229, 231–233, 235–237, 239–241, 243–245, 247–249, 251–253, 32–40, 43, 56–70, 77–82, 85–94, 96, 98–99
network_spec.rs	96.25%	100%	91.67%	97.06%	37
spawner.rs	0%	100%	0%	0%	100–103, 105, 109–111, 114–116, 118, 124–144, 146, 148–164, 17, 37–50, 52–53, 55, 57–60, 64–65, 67, 70–90, 92–99
crates/orchestrator/src/generators
bootnode_addr.rs	96.63%	100%	100%	96.25%
chain_spec.rs	3.90%	100%	8.33%	3.48%	100–111, 113–128, 130, 135–146, 149–155, 157–159, 162–190, 192–195, 197–199, 201–214, 216–218, 221–223, 225, 227–239, 241, 244–249, 251–252, 254–267, 269, 272–277, 279–280, 283–296, 298–303, 305–307, 309–311, 315–344, 347, 350–353, 356, 358–359, 361–376, 378, 38, 380–381, 386–427, 438–446, 456–464, 466–478, 480–488, 491–493, 497–500, 502–503, 508–519, 523–539, 541, 544–545, 548, 550–556, 558–563, 569–574, 576, 578, 580–587, 589–592, 595–596, 599–601, 603–604, 606–608, 610–612, 614–616, 619–621, 624–628, 630, 632–640, 642–643, 648–655, 657–667, 671–674, 676, 678, 680–685, 687–697, 699–715, 717–724, 726, 730–749, 75, 750–751, 753, 76, 762–769, 77, 770–772, 774, 776–794, 796–800, 802, 804, 84–87, 95–99
command.rs	0%	100%	0%	0%	100–112, 114–120, 122, 125–130, 132–133, 136–148, 15, 150, 153–155, 157–158, 16, 161–169, 17, 170–179, 18, 180–189, 19, 190–197, 199, 20, 201, 203–209, 21, 210–219, 22, 220–224, 226–229, 23, 231–233, 235–239, 24, 240–243, 246–250, 252–256, 258, 261–269, 271, 274–290, 293–298, 300, 303–305, 307–308, 311–333, 335, 337, 44–62, 64–66, 68–76, 79–97
errors.rs	0%	100%	0%	0%	4
identity.rs	82.35%	100%	60%	86.21%	13
key.rs	84.48%	100%	50%	90%	31
keystore.rs	0%	100%	0%	0%	19–26, 28–31, 34–46, 48–55, 57–64, 66, 68–86
para_artifact.rs	24.64%	100%	40%	22.03%	44–46, 48–64, 66–69, 72–81, 83–88, 92–93
port.rs	93.75%	100%	83.33%	96.15%
crates/orchestrator/src/network
node.rs	0%	100%	0%	0%	17–29, 33–41, 5
parachain.rs	0%	100%	0%	0%	15–23, 25–37, 5
relaychain.rs	0%	100%	0%	0%	14–21, 5
crates/orchestrator/src/network_spec
node.rs	40.83%	100%	33.33%	41.23%	108, 146–150, 152–153, 155, 158, 160–163, 165–168, 173–178, 180, 183, 185–215, 217, 90–91, 93–96
parachain.rs	73.91%	100%	75%	73.86%	104, 122, 127–130, 133, 144–147, 150, 68–70, 73, 75–77, 83, 92, 97
relaychain.rs	91.80%	100%	83.33%	92.73%	67, 90, 97
crates/orchestrator/src/shared
types.rs	47.37%	100%	38.46%	52%	43–47, 50
crates/provider/src
lib.rs	25%	100%	0%	50%
native.rs	90.42%	100%	81.88%	91.54%	1020, 1024, 1057, 1066, 1087, 1094, 1125, 1152, 1176, 1181, 1209, 1235, 1245, 1252, 1269, 1319, 132, 1376–1378, 1423, 1499–1500, 1524, 155, 1565, 190–194, 262, 281, 294, 296, 303, 305–306, 319, 341, 355–357, 359–361, 363–365, 387–389, 391–393, 41, 434, 454, 49, 503, 552, 581–582, 587–589, 707, 799, 806, 811, 816, 821, 828, 841, 90, 912, 936, 941, 945, 974, 990
crates/provider/src/shared
types.rs	84.47%	100%	74.29%	86.41%	123, 140–146, 15–18, 230, 249–256, 79
crates/support/src
fs.rs	75%	100%	66.67%	80%
net.rs	0%	100%	0%	0%	10, 4–9
crates/support/src/fs
in_memory.rs	95.55%	100%	96.48%	95.37%	149, 271, 321, 372, 424, 432, 440, 448, 470, 478, 593, 615, 643, 660, 678, 685, 704, 731, 748, 79–82, 84, 86–89, 91, 96
local.rs	98.25%	100%	96.30%	98.75%	51, 8
crates/test-runner/src
lib.rs	100%	100%	100%	100%

github-actions · 2023-09-28T11:59:03Z

Coverage after merging feat/glue/network-spec_provider into main

78.56%

Coverage Report

File	Stmts	Branches	Funcs	Lines	Uncovered Lines
crates/configuration/src
global_settings.rs	99.12%	100%	96.43%	99.50%
hrmp_channel.rs	95.88%	100%	89.47%	97.44%
network.rs	99.34%	100%	97.78%	99.49%	111, 116, 124, 80
parachain.rs	98.82%	100%	97.06%	99.02%	51–53, 67–70
relaychain.rs	98.99%	100%	95.38%	99.44%
utils.rs	100%	100%	100%	100%
crates/configuration/src/shared
errors.rs	50%	100%	50%	50%
helpers.rs	100%	100%	100%	100%
node.rs	98.32%	100%	92.68%	98.93%	124, 130
resources.rs	96.09%	100%	94.52%	96.43%	102, 114, 126–128, 147–150
types.rs	91.20%	100%	86.14%	92.69%	279–281, 350–352, 375–376, 378–380, 60–62
crates/orchestrator/src
errors.rs	0%	100%	0%	0%	8
lib.rs	0%	100%	0%	0%	100–102, 106–109, 111–117, 119, 123, 125–140, 143–168, 172–181, 184–192, 194, 196–200, 203–220, 222–227, 229–238, 241–247, 250, 253–262, 265–269, 272, 274–287, 289–291, 294–298, 300, 302–315, 317–320, 322–324, 335–336, 343, 350–352, 354–363, 365–366, 368–376, 378–385, 387–394, 396–399, 40, 400–407, 41, 410, 42–45, 47–52, 54–60, 62–67, 70, 72–86, 92–97, 99
network.rs	0%	100%	0%	0%	100–108, 111–117, 119–127, 130–154, 156, 158–167, 170, 172–176, 185–189, 191, 193, 195–199, 201, 203, 205–213, 215, 217, 220–225, 227–229, 231–233, 235–237, 239–241, 243–245, 247–249, 251–253, 32–40, 43, 56–70, 77–82, 85–94, 96, 98–99
network_spec.rs	96.25%	100%	91.67%	97.06%	37
spawner.rs	0%	100%	0%	0%	100–103, 105, 109–111, 114–116, 118, 124–144, 146, 148–164, 17, 37–50, 52–53, 55, 57–60, 64–65, 67, 70–90, 92–99
crates/orchestrator/src/generators
bootnode_addr.rs	97.20%	100%	100%	96.84%
chain_spec.rs	3.90%	100%	8.33%	3.48%	100–111, 113–128, 130, 135–146, 149–155, 157–159, 162–190, 192–195, 197–199, 201–214, 216–218, 221–223, 225, 227–239, 241, 244–249, 251–252, 254–267, 269, 272–277, 279–280, 283–296, 298–303, 305–307, 309–311, 315–344, 347, 350–353, 356, 358–359, 361–376, 378, 38, 380–381, 386–427, 438–446, 456–464, 466–478, 480–488, 491–493, 497–500, 502–503, 508–519, 523–539, 541, 544–545, 548, 550–556, 558–563, 569–574, 576, 578, 580–587, 589–592, 595–596, 599–601, 603–604, 606–608, 610–612, 614–616, 619–621, 624–628, 630, 632–640, 642–643, 648–655, 657–667, 671–674, 676, 678, 680–685, 687–697, 699–715, 717–724, 726, 730–749, 75, 750–751, 753, 76, 762–769, 77, 770–772, 774, 776–794, 796–800, 802, 804, 84–87, 95–99
command.rs	0%	100%	0%	0%	100–112, 114–120, 122, 125–130, 132–133, 136–148, 15, 150, 153–155, 157–158, 16, 161–169, 17, 170–179, 18, 180–189, 19, 190–197, 199, 20, 201, 203–209, 21, 210–219, 22, 220–224, 226–229, 23, 231–233, 235–239, 24, 240–243, 246–250, 252–256, 258, 261–269, 271, 274–290, 293–298, 300, 303–305, 307–308, 311–333, 335, 337, 44–62, 64–66, 68–76, 79–97
errors.rs	0%	100%	0%	0%	4
identity.rs	82.35%	100%	60%	86.21%	13
key.rs	95.05%	100%	84.62%	96.59%	31
keystore.rs	0%	100%	0%	0%	19–26, 28–31, 34–46, 48–55, 57–64, 66, 68–86
para_artifact.rs	24.64%	100%	40%	22.03%	44–46, 48–64, 66–69, 72–81, 83–88, 92–93
port.rs	93.75%	100%	83.33%	96.15%
crates/orchestrator/src/network
node.rs	0%	100%	0%	0%	17–29, 33–41, 5
parachain.rs	0%	100%	0%	0%	15–23, 25–37, 5
relaychain.rs	0%	100%	0%	0%	14–21, 5
crates/orchestrator/src/network_spec
node.rs	40.83%	100%	33.33%	41.23%	108, 146–150, 152–153, 155, 158, 160–163, 165–168, 173–178, 180, 183, 185–215, 217, 90–91, 93–96
parachain.rs	73.91%	100%	75%	73.86%	104, 122, 127–130, 133, 144–147, 150, 68–70, 73, 75–77, 83, 92, 97
relaychain.rs	91.80%	100%	83.33%	92.73%	67, 90, 97
crates/orchestrator/src/shared
types.rs	47.37%	100%	38.46%	52%	43–47, 50
crates/provider/src
lib.rs	25%	100%	0%	50%
native.rs	90.42%	100%	81.88%	91.54%	1020, 1024, 1057, 1066, 1087, 1094, 1125, 1152, 1176, 1181, 1209, 1235, 1245, 1252, 1269, 1319, 132, 1376–1378, 1423, 1499–1500, 1524, 155, 1565, 190–194, 262, 281, 294, 296, 303, 305–306, 319, 341, 355–357, 359–361, 363–365, 387–389, 391–393, 41, 434, 454, 49, 503, 552, 581–582, 587–589, 707, 799, 806, 811, 816, 821, 828, 841, 90, 912, 936, 941, 945, 974, 990
crates/provider/src/shared
types.rs	84.47%	100%	74.29%	86.41%	123, 140–146, 15–18, 230, 249–256, 79
crates/support/src
fs.rs	75%	100%	66.67%	80%
net.rs	0%	100%	0%	0%	10, 4–9
crates/support/src/fs
in_memory.rs	95.55%	100%	96.48%	95.37%	149, 271, 321, 372, 424, 432, 440, 448, 470, 478, 593, 615, 643, 660, 678, 685, 704, 731, 748, 79–82, 84, 86–89, 91, 96
local.rs	98.25%	100%	96.30%	98.75%	51, 8
crates/test-runner/src
lib.rs	100%	100%	100%	100%

l0r1s

Awesome work ! Asked many questions to understand more about how everything works and some todos we will tackle later 👏

l0r1s · 2023-09-27T21:03:11Z

crates/orchestrator/src/network_spec.rs

+        let mut errs = vec![];
+        let relaychain = RelaychainSpec::from_config(network_config.relaychain())?;
+        let mut parachains = vec![];
+
+        for para_config in network_config.parachains() {
+            match ParachainSpec::from_config(para_config) {
+                Ok(para) => parachains.push(para),
+                Err(err) => errs.push(err),
+            }
+        }


nit: map + fold

Suggested change

let mut errs = vec![];

let relaychain = RelaychainSpec::from_config(network_config.relaychain())?;

let mut parachains = vec![];

for para_config in network_config.parachains() {

match ParachainSpec::from_config(para_config) {

Ok(para) => parachains.push(para),

Err(err) => errs.push(err),

}

}

let (parachains, errs) = network_config

.parachains()

.iter()

.map(|para_config| ParachainSpec::from_config(para_config))

.fold((vec![], vec![]), |(mut parachains, mut errs), result| {

match result {

Ok(para) => parachains.push(para),

Err(err) => errs.push(err),

}

(parachains, errs)

});

Yes, I think we can use fold directly too :)

l0r1s · 2023-09-27T21:05:04Z

crates/orchestrator/src/network_spec.rs

+                Err(err) => errs.push(err),
+            }
+        }
+


We don't use errors here, maybe do something like in network_spec::node ? A bit hacky but we will revisit later:

Suggested change

if !errs.is_empty() {

// TODO: merge errs

return Err(errs.swap_remove(0));

}

crates/orchestrator/src/lib.rs

l0r1s · 2023-09-27T22:02:39Z

crates/orchestrator/src/network_spec/parachain.rs

+        let main_cmd = if let Some(cmd) = config.default_command() {
+            cmd
+        } else if let Some(first_node) = config.collators().first() {
+            let Some(cmd) = first_node.command() else {
+                return Err(OrchestratorError::InvalidConfig("Parachain, either default_command or command in the first node needs to be set.".to_string()));
+            };
+
+            cmd
+        } else {
+            return Err(OrchestratorError::InvalidConfig(
+                "Parachain without nodes and default_command isn't set.".to_string(),
+            ));
+        };


nit: same refacto as relaychain

Suggested change

let main_cmd = if let Some(cmd) = config.default_command() {

cmd

} else if let Some(first_node) = config.collators().first() {

let Some(cmd) = first_node.command() else {

return Err(OrchestratorError::InvalidConfig("Parachain, either default_command or command in the first node needs to be set.".to_string()));

};

cmd

} else {

return Err(OrchestratorError::InvalidConfig(

"Parachain without nodes and default_command isn't set.".to_string(),

));

};

let main_cmd = config

.default_command()

.or(config.collators().first().and_then(|node| node.command()))

.ok_or(OrchestratorError::InvalidConfig(

"Parachain, either default_command or first node with command needs to be set."

.to_string(),

));

TODO: extract errors to constants or variant with more details

Yes, in general Errors needs to be revamped as followup pr.

crates/orchestrator/src/generators/chain_spec.rs

l0r1s · 2023-09-28T18:57:57Z

crates/orchestrator/src/lib.rs

+            .filter(|para| match &para.registration_strategy {
+                RegistrationStrategy::InGenesis => true,
+                RegistrationStrategy::UsingExtrinsic => false,
+            })


TODO: add helper on RegistrationStrategy to do something like this:

registration_strategy.is_in_genesis(); registration_strategy.is_using_extrinsic();

crates/orchestrator/src/generators/chain_spec.rs

l0r1s · 2023-09-28T19:10:04Z

crates/orchestrator/src/lib.rs

+        // TODO: we want to still supporting spawn a dedicated bootnode??
+        let mut ctx = SpawnNodeCtx {
+            chain_id: &relay_chain_id,
+            parachain_id: None,
+            chain: relay_chain_name,
+            role: ZombieRole::Node,
+            ns: &ns,
+            scoped_fs: &scoped_fs,
+            parachain: None,
+            bootnodes_addr: &vec![],
+        };


What do you mean ? If there is no bootnodes, do we want to bootstrap a node ourselves ?

crates/orchestrator/src/lib.rs

github-actions · 2023-09-29T12:09:07Z

Coverage after merging feat/glue/network-spec_provider into main

78.56%

Coverage Report

File	Stmts	Branches	Funcs	Lines	Uncovered Lines
crates/configuration/src
global_settings.rs	99.12%	100%	96.43%	99.50%
hrmp_channel.rs	95.88%	100%	89.47%	97.44%
network.rs	99.34%	100%	97.78%	99.49%	111, 116, 124, 80
parachain.rs	98.82%	100%	97.06%	99.02%	51–53, 67–70
relaychain.rs	98.99%	100%	95.38%	99.44%
utils.rs	100%	100%	100%	100%
crates/configuration/src/shared
errors.rs	50%	100%	50%	50%
helpers.rs	100%	100%	100%	100%
node.rs	98.32%	100%	92.68%	98.93%	124, 130
resources.rs	96.09%	100%	94.52%	96.43%	102, 114, 126–128, 147–150
types.rs	91.20%	100%	86.14%	92.69%	279–281, 350–352, 375–376, 378–380, 60–62
crates/orchestrator/src
errors.rs	0%	100%	0%	0%	8
lib.rs	0%	100%	0%	0%	100–102, 106–109, 111–117, 119, 123, 125–140, 143–168, 172–181, 184–192, 194, 196–200, 203–220, 222–227, 229–238, 241–247, 250, 253–262, 265–269, 272, 274–287, 289–291, 294–298, 300, 302–315, 317–320, 322–324, 335–336, 346, 353–355, 357–366, 368–369, 371–379, 381–388, 390–397, 399, 40, 400–409, 41, 410, 413, 42–45, 47–52, 54–60, 62–67, 70, 72–86, 92–97, 99
network.rs	0%	100%	0%	0%	100–108, 111–117, 119–127, 130–154, 156, 158–167, 170, 172–176, 185–189, 191, 193, 195–199, 201, 203, 205–213, 215, 217, 220–225, 227–229, 231–233, 235–237, 239–241, 243–245, 247–249, 251–253, 32–40, 43, 56–70, 77–82, 85–94, 96, 98–99
network_spec.rs	96.25%	100%	91.67%	97.06%	38
spawner.rs	0%	100%	0%	0%	100–103, 105, 109–111, 114–116, 118, 124–144, 146, 148–164, 17, 37–50, 52–53, 55, 57–60, 64–65, 67, 70–90, 92–99
crates/orchestrator/src/generators
bootnode_addr.rs	97.20%	100%	100%	96.84%
chain_spec.rs	3.90%	100%	8.33%	3.48%	100–111, 113–128, 130, 135–146, 149–155, 157–159, 162–190, 192–195, 197–199, 201–214, 216–218, 221–223, 225, 227–239, 241, 244–249, 251–252, 254–267, 269, 272–277, 279–280, 283–296, 298–303, 305–307, 309–311, 315–344, 347, 350–353, 356, 358–359, 361–376, 378, 38, 380–381, 386–427, 438–446, 456–464, 466–478, 480–488, 491–493, 497–500, 502–503, 508–519, 523–539, 541, 544–545, 548, 550–556, 558–563, 569–574, 576, 578, 580–587, 589–592, 595–596, 599–601, 603–604, 606–608, 610–612, 614–616, 619–621, 624–628, 630, 632–640, 642–643, 648–655, 657–667, 671–674, 676, 678, 680–685, 687–697, 699–715, 717–724, 726, 730–749, 75, 750–751, 753, 76, 762–769, 77, 770–772, 774, 776–794, 796–800, 802, 804, 84–87, 95–99
command.rs	0%	100%	0%	0%	100–112, 114–120, 122, 125–130, 132–133, 136–148, 15, 150, 153–155, 157–158, 16, 161–169, 17, 170–179, 18, 180–189, 19, 190–197, 199, 20, 201, 203–209, 21, 210–219, 22, 220–224, 226–229, 23, 231–233, 235–239, 24, 240–243, 246–250, 252–256, 258, 261–269, 271, 274–290, 293–298, 300, 303–305, 307–308, 311–333, 335, 337, 44–62, 64–66, 68–76, 79–97
errors.rs	0%	100%	0%	0%	4
identity.rs	82.35%	100%	60%	86.21%	13
key.rs	95.05%	100%	84.62%	96.59%	31
keystore.rs	0%	100%	0%	0%	19–26, 28–31, 34–46, 48–55, 57–64, 66, 68–86
para_artifact.rs	24.64%	100%	40%	22.03%	44–46, 48–64, 66–69, 72–81, 83–88, 92–93
port.rs	93.75%	100%	83.33%	96.15%
crates/orchestrator/src/network
node.rs	0%	100%	0%	0%	17–29, 33–41, 5
parachain.rs	0%	100%	0%	0%	15–23, 25–37, 5
relaychain.rs	0%	100%	0%	0%	14–21, 5
crates/orchestrator/src/network_spec
node.rs	40.83%	100%	33.33%	41.23%	108, 146–150, 152–153, 155, 158, 160–163, 165–168, 173–178, 180, 183, 185–215, 217, 90–91, 93–96
parachain.rs	73.91%	100%	75%	73.86%	104, 122, 127–130, 133, 144–147, 150, 68–70, 73, 75–77, 83, 92, 97
relaychain.rs	91.80%	100%	83.33%	92.73%	67, 90, 97
crates/orchestrator/src/shared
types.rs	47.37%	100%	38.46%	52%	43–47, 50
crates/provider/src
lib.rs	25%	100%	0%	50%
native.rs	90.42%	100%	81.88%	91.54%	1020, 1024, 1057, 1066, 1087, 1094, 1125, 1152, 1176, 1181, 1209, 1235, 1245, 1252, 1269, 1319, 132, 1376–1378, 1423, 1499–1500, 1524, 155, 1565, 190–194, 262, 281, 294, 296, 303, 305–306, 319, 341, 355–357, 359–361, 363–365, 387–389, 391–393, 41, 434, 454, 49, 503, 552, 581–582, 587–589, 707, 799, 806, 811, 816, 821, 828, 841, 90, 912, 936, 941, 945, 974, 990
crates/provider/src/shared
types.rs	84.47%	100%	74.29%	86.41%	123, 140–146, 15–18, 230, 249–256, 79
crates/support/src
fs.rs	75%	100%	66.67%	80%
net.rs	0%	100%	0%	0%	10, 4–9
crates/support/src/fs
in_memory.rs	95.55%	100%	96.48%	95.37%	149, 271, 321, 372, 424, 432, 440, 448, 470, 478, 593, 615, 643, 660, 678, 685, 704, 731, 748, 79–82, 84, 86–89, 91, 96
local.rs	98.25%	100%	96.30%	98.75%	51, 8
crates/test-runner/src
lib.rs	100%	100%	100%	100%

pepoviola · 2023-09-29T12:12:01Z

Merging now @wirednkod / @l0r1s, thanks!

pepoviola and others added 30 commits August 8, 2023 12:13

wip, refactor provider trait

776226e

fmt

5ff367e

wip

f99f009

wip, add from_config to NetworkSpec

95816cb

config reorgs

92f6ccc

add logic for create NetworkSpec

a8bce1d

feat: refactored FileSystem trait and errors

b20050e

chore: removed unused local_file, stdout will be hardcoded in provide…

6a7c8de

…r at the moment

feat: moved MockFileSystem to InMemoryFileSystem, files too, and adde…

9b010bb

…d mirror implementation of tokio::fs with tests

feat: added new method append to FileSystem trait

7549c71

feat: implemented new append method on InMemoryFileSystem

12371a2

feat: added anyhow dependencies to support crate

4a02844

feat: refactored FileSystem trait to expose only a single wrapped error

7aa5d07

feat: refacto InMemoryFileSystem following FileSystem trait refacto

b9004ff

feat: added conversion between io::Error and FileSystemError

ca9932a

feat: added implementation of LocalFileSystem using tokio::fs

08e2571

feat: added nix crate dependency to workspace and provider crate

cf5e9ba

feat: added uuid with v4 as dev-dependencies for testing in support c…

061abe2

…rate

feat: added unit tests for LocalFileSystem

c9ee23c

feat: updated Provider, ProviderNamespace and ProviderNode traits and…

e993090

… related DTO

feat: added anyhow as dependency to provider crate

d0657c0

feat: updated ProviderError

b773d9d

feat: work on NativeProvider with individual Node, Namespace and Prov…

9ff812a

…ider struct threadsafe using RwLock and Arc, added implementation logic of resume/restart/pause, destroy and helpers for logs

feat: rename some provider constants

0910156

Merge pull request #99 from paritytech/refactor/provider-trait-l0r1s

19846cb

Refactor/provider trait l0r1s

wip

665a447

feat: added uuid with v4 features as normal dependency

0f28ee7

feat: added new set_mode method on FileSystem trait to modify permiss…

8f69cee

…ions bits, added implementations and tests

feat: added builder for options types used in provider traits methods…

81a3046

…, moved error next to provider traits

feat: added modified implementation of run_script/run_command/copy_fr…

c2173da

…om_node in NativeProvider, removed unused comments