Introduce --mastermix mode: Common mix of mix master with control over monitoring

[cwerling]

See related issue: #599

Server only:
      --mastermix       Master mix server mode where all clients hear the
                        mix of the mix master with two exceptions:
                        - They still control their own channel in their mix
                        - They still control the mix master in their mix
                        The mix master is the first client to connect
                        and can solo others 'secretly' (i.e. without
                        affecting the shared mix)

You can find a beta release of this feature here.

This PR introduces the new server mode --mastermix.

• Mix master will be the first to join after the server started/idled (i.e. client ID 0) [0]
• The mix master can 'secretly' SOLO people without affecting the common mix
• The non-mix masters will see all faders disabled with two exceptions:
• Exception 1: The non-mix masters control the mix master's channel in their own mix [1]
• Exception 2: Everybody control their own channel in their own mix (i.e. monitoring) [2]

This is a fully working implementation and has proven very valuable to my own choir. It was originally based on this discussion but has a different focus: Delegating mixing for ensembles (vs. server performance for large sessions in the OP).

[0] A new mix master can therefore be assigned like this: Everybody disconnects and the designated mix master re-joins first. Although this is a rather primitive way of handling this authorization problem, it might resemble the KISS principle best (open for discussion)
[1] Think of a scenario where somebody wants to hear the conductor (=mix master) louder than others or listeners might even want to mute him (say he has a click running).
[2] With a low-latency setup, my choir has a good experience with disabling Jamulus-routed monitoring.

You can download the latest binaries from the autobuild pipeline (In GitHub Checks > Artifacts).

Open tasks:

• Test a server with this feature enabled with clients who don't know this feature (and the new SingleMixStateMes)
  Expectation: Audio behaviour as described above, but no indication that the client's mixer controls are useless (could be warned of in the server's welcome message)
• Test a server without this feature enabled with clients with this feature
• Check out if heterogeneous network settings of the clients interfere with the expected behaviour
• Enable multi-threading
• Enable masters to solo people just for themselves
• Enable everyone (including the mix master) to control their own channel in their own mix (i.e. monitoring)
• Test, test, test
• Enable non-masters to set their own fader
• Bonus: Display gain and pan fader positions in clients' (disabled) mixer boards

[gene96817]

As we work on this feature, I hope there will be a clear explanation to a new ensemble as to how to understand each option and which configuration is best for them. Trial-and-error is not a good way to understand these features.

[dcorson-ticino-com]

Do I understand correctly that with no-monitor set each clients own channel is muted?
How is that client then supposed to sing along with the others if they can't hear themselves?
This is opposed to every counsel I have ever heard about using Jamulus. (and also the first rule of Jamulus)

[cwerling]

Do I understand correctly that with no-monitor set each clients own channel is muted?
How is that client then supposed to sing along with the others if they can't hear themselves?
This is opposed to every counsel I have ever heard about using Jamulus. (and also the first rule of Jamulus)

In our mixed choir‘s rehearsals we hear ourselves either acoustically only or through interface routing. Our conductor either has a click running to synchronize or conducts us over a low-latency Jitsi call.

I have the same experience playing trombone on Jamulus in a small Jazz combo. With acoustical instruments, since you cannot turn off physical resonance (through your body and the general room audio), I never saw any value in hearing myself with Jamulus-routed monitoring.

But again; these are just the experiences of myself with our 25 people choir and the jazz band. And so everybody is free to use either exact single mix (‘—singlemix monitor’) or the version without monitoring (‘—singlemix no-monitor’).

[dcorson-ticino-com]

I never saw any value in hearing myself with Jamulus-routed monitoring.

I play viola, an instrument jammed between chin and chest, and note that there is a big difference in the results playing using the signal from Jamulus. Just as one can overhear someone else speaking you can overhear your own playing. It is all in your head. Once the group gets the hang of it, which doesn't take very long actually, the results are musically much better. I would never recommend to anyone to mute their own signal. That gives them the signal it is better without and not to try acclimatizing to Jamulus, which will in the end give better results.

[cwerling]

I never saw any value in hearing myself with Jamulus-routed monitoring.

I play viola, an instrument jammed between chin and chest, and note that there is a big difference in the results playing using the signal from Jamulus. Just as one can overhear someone else speaking you can overhear your own playing. It is all in your head. Once the group gets the hang of it, which doesn't take very long actually, the results are musically much better. I would never recommend to anyone to mute their own signal. That gives them the signal it is better without and not to try acclimatizing to Jamulus, which will in the end give better results.

I appreciate this insight and will check that again some time.

As we work on this feature, I hope there will be a clear explanation to a new ensemble as to how to understand each option and which configuration is best for them. Trial-and-error is not a good way to understand these features.

I understand the current naming of the CLI args (--singlemix monitor/no-monitor) is not optimal. Would it help to call them differently you think?

Regarding the general process on this PR, I suggest we try to find consensus in this discussion. It provides a WIP overview of the different solutions (one of which (Single mix) this PR implements).

[DominikSchaller]

I want to bring a point up that was made in the other thread, but I think it is really important. The person with the mixing board needs to have an option to Solo individuals to find culprits. But this Solo function should only be applied locally. So a kind of local override mode for the Solo function (maybe also for the Mute).

The scenario is that the sound technician is creating a mix for everybody. Then when everybody is singing and there is a problem in the sound, the technician can quickly go through the singers by Solo-ing everybody one by one until he finds the problem. But the mix for the chorus should be undisturbed by this.

[cwerling]

I want to bring a point up that was made in the other thread, but I think it is really important. The person with the mixing board needs to have an option to Solo individuals to find culprits. But this Solo function should only be applied locally. So a kind of local override mode for the Solo function (maybe also for the Mute).

The scenario is that the sound technician is creating a mix for everybody. Then when everybody is singing and there is a problem in the sound, the technician can quickly go through the singers by Solo-ing everybody one by one until he finds the problem. But the mix for the chorus should be undisturbed by this.

Thanks for bringing this up. This is definitely a problem in my current implementation and a tricky one to solve!

I just had a look at live Jamulus traffic through the handy Wireshark plugin. On the protocol side "solo"ing somebody seems to be equivalent to muting everybody else or even turning every body else's fader all the way down to 0.

This totally makes sense (and I imagined it to be implemented like this), but it doesn't leave us with an easy option to "filter out" "solo"ing of the mix master from the single mix. Or am I missing something here?

This might be the single biggest reason (for me personally) to go with a copy-once approach. Will take this to the overview of solutions.

[DominikSchaller]

I would suggest that if the master presses Solo on somebody, that this is only applied to the mix of the master client. So we would send the same mix as before to all clients EXCEPT to the master client.

[cwerling]

I would suggest that if the master presses Solo on somebody, that this is only applied to the mix of the master client. So we would send the same mix as before to all clients EXCEPT to the master client.

So this would require an additional message in the Jamulus protocol then (sth like ‘SOLO_ACTIVATED([i])’) so that the server can detect and exclude it from the mix. Might look into the required changes for this in the coming days!

[DominikSchaller]

Wait, now I'm confused. If Solo is only applied locally (meaning, the server doesn't know what client pressed which solo buttons), no protocol is needed. Just leave it as is for the master client and disable the buttons for everyone else. (Why disable for everyone else? Well, we had our fair share of misuse in big groups, with people soloing themselves and not being able to hear the conductor.) If Solo is applied via the server, the server already knows what client pressed what buttons, an additional protocol message is also not needed. Christian Werling ***@***.***> schrieb am Mi., 31. März 2021, 08:39:

…

I would suggest that if the master presses Solo on somebody, that this is only applied to the mix of the master client. So we would send the same mix as before to all clients EXCEPT to the master client. So this would require an additional message in the Jamulus protocol then (sth like ‘SOLO_ACTIVATED([i])’) so that the server can detect and exclude it from the mix. Might look into the required changes for this in the coming days! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1381 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOQQ4TYLVNTZWW364WO6ZZ3TGK7QPANCNFSM4Z6MJQQA> .

[cwerling]

Wait, now I'm confused. If Solo is only applied locally (meaning, the server doesn't know what client pressed which solo buttons), no protocol is needed. Just leave it as is for the master client and disable the buttons for everyone else. (Why disable for everyone else? Well, we had our fair share of misuse in big groups, with people soloing themselves and not being able to hear the conductor.) If Solo is applied via the server, the server already knows what client pressed what buttons, an additional protocol message is also not needed. Christian Werling @.***> schrieb am Mi., 31. März 2021, 08:39:
…
I would suggest that if the master presses Solo on somebody, that this is only applied to the mix of the master client. So we would send the same mix as before to all clients EXCEPT to the master client. So this would require an additional message in the Jamulus protocol then (sth like ‘SOLO_ACTIVATED([i])’) so that the server can detect and exclude it from the mix. Might look into the required changes for this in the coming days! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1381 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOQQ4TYLVNTZWW364WO6ZZ3TGK7QPANCNFSM4Z6MJQQA .

I think (correct me if I'm wrong) all mixing happens on the server side, so that only the final (individual) mix is sent to each client. When I mute somebody in the mix, I just send a message to the server telling it to put client's X gain in my mix to 0. When I solo somebody, I send multiple messages to the server telling it to put all the other clients' gain to 0.

Looking at the traffic in Wireshark using the aforementioned Jamulus protocol plugin turning on and off "Solo" on client [1]:

Now in this example therei only one other client ([0]), but if there were more, it would send multiple individual CHANNEL_GAIN messages to the server.

BTW: MUTE_STATE_CHANGED is a message that I think is used to display this little indicator at the respective client's GUI that you can't here him/here.

Can anybody confirm my understanding here? Started looking into Jamulus internals only very recently.

Cheers!

[DominikSchaller]

Ah ok! Yes, then an additional message is needed.

Another interesting edge case:

• Imagine somebody has a bad connection or weird sound, so the sound engineer mutes them in the mix. Should they still be able to hear themselves in THEIR mix?

[DominikSchaller]

And I have another idea: What if - instead of a monitor/no-monitor mode - it is possible for each client to always change their own channel. Meaning they can decide if they want to mute themselves, make themselves louder or quieter, pan themselves to one side etc. But - as it is at the moment - this would only be applied to their own mix. Everyone else would still hear the master mix. You might be able to get rid of the command line parameter, if you don't have a reason to forbid this level of control globally.

[ann0see]

I think (correct me if I'm wrong) all mixing happens on the server side, so that only the final (individual) mix is sent to each client

Yes. That's correct.

@DominikSchaller I think you'd be better with a "follow mix" feature where personal mixes still exist but a "master" sets the initial levels.

Personally I see a huge benefit in a real single mix from a performance standpoint.

[DominikSchaller]

I think (correct me if I'm wrong) all mixing happens on the server side, so that only the final (individual) mix is sent to each client

Yes. That's correct.

@DominikSchaller I think you'd be better with a "follow mix" feature where personal mixes still exist but a "master" sets the initial levels.

Personally I see a huge benefit in a real single mix from a performance standpoint.

This is my use case: https://www.facebook.com/Barbershop.in.Germany/posts/4085942734804200
Two days ago we sang with 100 singers in Jamulus. You can watch the video with the link (even if you're not on facebook).

It was a huge success and lots of fun, but we had a lot of problems with singers using the Solo button and then talking because they could not hear anybody else. They couldn't even find their own mix controls again because the screen was not big enough ... We had to phone them to help them, while 98 other singers were listening. At the end we asked everybody else to mute them. I also had to mute some people who just had a bad sound or a bad connection.

My vision is one of a real live performance where you have one person sitting at the mixing board. Every singer or band member can change their own monitor level, but nothing else.

I know that other choruses also use the single mixer mode. They give pre-configured Raspberry Pis to their singers and these singers have no control options at all. They don't even see a GUI. Everything is mixed by one person. This person would like to have the Solo functionality to go through the clients one by one and adjust their volume WITHOUT everybody else noticing. I believe this is also how a real-life mixing board behaves.

[DominikSchaller]

By the way, my video is using the new delay pan feature. :)

[DominikSchaller]

About the mix master:

• Mix master will be the first to join after the server started/idled (i.e. client ID 0)
• A new mix master can therefore be assigned like this: Everybody disconnects and the designated mix master re-joins first
• Although this is a rather primitive way of handling this authorization problem, it might resemble the KISS principle best (open for discussion)

How about this idea: We create a new instrument for this feature which is called Sound Mixer (icon could be a mixing board). The first client who selects this instrument becomes the mix master. This would make re-assigning a lot easier. Because with a big chorus everybody disconnecting is a hassle. And maybe the sound mixer is late to rehearsal, so this would be a lot easier. It could be combined with your implementation like this:

• Mix master will be the first to join after the server started/idled (i.e. client ID 0) => This ensures that there is always one in control.
• The first to select the instrument Sound Mixer becomes the new mix master
• Reassigment would work like this:
  1. The current sound mixer selects a different instrument (they would still be the master)
  2. Someone else selects the Sound Mixer instrument (they take over the master)

[cwerling]

Personally I see a huge benefit in a real single mix from a performance standpoint.

Out of personal interest as well as a lack of time I can contribute, I would like to leave the performance optimization to a subsequent PR. We have to, for example, consider that all clients need to have the same network settings for this to work (as @corrados pointed out in his original POC). I therefore updated this PR to not close the original issue.

Every singer or band member can change their own monitor level, but nothing else.

This sounds like a feasable and sensible addition to the current state of this PR. But this will only work for the non-masters, since the master uses her own channel's fader to control her volume for everybody. [0]

How about this idea: We create a new instrument for this feature which is called Sound Mixer (icon could be a mixing board).

Although I'm a big fan of using existing entities to implement new functionality, I don't like using such a meta-data feature to change the logic of Jamulus. I think it should be a more explicit choice to get such a role. Additonally, the logic if more than one person uses this instrument ist not straight-forward. Combined with the first-joiner-becomes-master (regardless of the instrument) spans up a wide range of scenarios which all have to be (logically) covered on the server side.

In terms of 'authorization' to be the mix master, I think this should be coupled to the person running the server. I would also be fine with an optional CLI argument --mix-master-ip or alike to set it. I'm open to more suggestions! But let's make sure we keep things as KISS as they are right now in Jamulus :)

[0] If the mix master does not want to hear herself, she has to start two Jamulus instances: One for mixing with a dead output channel and one for listening with their own channel muted.

[cwerling]

This is my use case: https://www.facebook.com/Barbershop.in.Germany/posts/4085942734804200
Two days ago we sang with 100 singers in Jamulus. You can watch the video with the link (even if you're not on facebook).

This sounds very interesting, but I'm not able to see any info behind that link without a Facebook account. Can you provide more info?

[cwerling]

Does anybody have technical info on this?

Optional: Display gain and pan fader positions in clients' (disabled) mixer boards
Does anybody know if the server is currently able to remotely change fader positions at all? If so, this should be done quite easily.

[hoffie]

Optional: Display gain and pan fader positions in clients' (disabled) mixer boards
Does anybody know if the server is currently able to remotely change fader positions at all? If so, this should be done quite easily.

I don't think so.

I would like to leave the performance optimization to a subsequent PR.

A quick thought about having a generic performance optimization here: Attempt to re-use mixes if all parameters (gain, pan, mute, audio quality, ...) are equal. This would help for the feature discussed here, but would also help if people use the same mix via some other measure (e.g. everyone-at-100% or a shared mixer settings file). The downside is that this would require some tracking, which may get expensive. :(

[cwerling]

Quick question (can't test this atm): What's the current behaviour of Jamulus when you solo more than one channel? Will I hear all solo'd channels or only the latest solo'd channel?

[DominikSchaller]

You'll hear all. Which is different than a real mixing board or a DAW where a Solo button is actually exclusive.

[cwerling]

Just checked in GarageBand and you can actually solo multiple channels. So you're saying Jamulus behaves the same way currently?

[DominikSchaller]

Just checked in GarageBand and you can actually solo multiple channels. So you're saying Jamulus behaves the same way currently?

1. Jamulus: You can Solo multiple people at the same time.
2. Audacity: You can Solo just one channel. Audacity mutes everybody else if you press Solo on one channel.
3. Real mixing board: If you press Solo, you can listen to a single channel while the actual mix is undisturbed.
4. I don't know about all DAWs so it might be different in other programs.

[hoffie]

Ardour: You can Solo multiple tracks at the same time.

[DominikSchaller]

Actually, it is even different on real mixing boards: https://hub.yamaha.com/using-solo/

What I'd like to see for the single server mix mode eventually is a PFL (pre-fader level) Solo function.

[ann0see]

Hi mhilpolt, sorry for being a bit off topic here: Since tagging a GitHub user with @ automatically subscribes him/her to this PR it would be appreciated if you didn't tag corrados (he'll not answer probably).

Now coming back to the topic: I hope to try this PR soon (and have already set up a test server). So far I've experienced lower latency with --mastermix enabled and no multithreading. Not sure if multithreading adds latency?

[DominikSchaller]

So you had a higher CPU load with my fork with our without multithreading enabled? I would expect my fork to have the same performance behaviour as the original Jamulus. You're not saying it's higher than that, but only higher than the original PoC by corrados right?

In the singlemixserver branch, multithreading and fastupdate options are not supported, according to the description of @corrados, so it is not switched on. (options: --server --discononquit --singlemix --nogui --numchannels 50 --norecord --port 22124)
For the mastermix fork test, I switched on both (Options: --server --discononquit --mastermix --nogui --numchannels 50 --norecord --port 22130 --fastupdate --multithreading)
So the comparison goal was to see, if the server will also work with mastermix in our chorus setup. But then I saw, that the load was even higher with without using buffersize of 64 at all clients. But I guess it it because the server has the option "fastupdate" enabled. I can try to evaluate this further.

The old singlemixserver branch only uses one mix for everybody, therefore it doesn't need multithreading or even a lot of CPU power.

The new mastermix branch calculates a different mix for everybody, therefore it behaves like the "regular" Jamulus and needs a lot more CPU power, but also profits from multithreading. The fastupdate option shouldn't have any effect is no client uses a buffersize of 64.

[mhilpolt]

I did some performance measurements on my server (dedicated, Intel Celeron J4105, 4x Core)
Options singlemix: --server --discononquit --singlemix --nogui --numchannels 50 --norecord
Options mastermix: --server --discononquit --mastermix --nogui --numchannels 50 --norecord --fastupdate --multithreading

no. connections	singlemix (128)	mastermix (128)	mastermix (64)
10	11 %	15 %	5 %
20	14 %	30 %	20 %
30	24 %	49 %	34 %
40	25 %	39 %	23 %
50	25 %	46 %	30 %

The client connections were generated on two AWS machines each client playing a piece of music to be able to control the quality of the connection. CPU-Load measurement by nmon.

My comments:

• load is not really reproducable, I got difference of more that 10% when I repeat the measurement under similar conditions
• with more than 30 connections, singlemix server has a growing benefit because only one mix is created on the server
• surprising to me: mastermix with buffersize 64 needs less power than 128
• in all variants, CPU-load of one core does not exceed 70%

Summary is very positive for me: I can use both variants on my server with enough headroom.

[cwerling]

This is a very valuable test, thanks @mhilpolt! Would you be able to share (parts of) your test setup with us? I'm especially interested in the "client playing a piece of music to be able to control the quality of the connection" part.

I would like to emphasize that --mastermix should not be considered to be any different from the original Jamulus performance-wise (and does not claim to be). So I rather read your performance comparison as a test between --singlemix and the original Jamulus. (Unless I'm overlooking an aspect of my PR's code additions?)

[mhilpolt]

I've created a perl script which starts jackd and then a configurable number of clients. Please find the code attached. On the server, I run several instances on different ports to be able to test different versions and to have a backup server, if the mix-client crashes. Then I can switch as the first one to that server to get the mix-capablilities and afterwards telling the singers to switch to that server instance.
If running several Jamulis-client instances from a single computer (in my case a AWS machníne) there seems to be a limitation of 32 connections from the same IP-address - this is what I observed, so I used a second machine to start the other clients to get more than 32 connections.

TestJamulus_public.pl.txt

[mhilpolt]

While I was preparing our next rehearsal planning to use the mastermix fork, I have recognized a major blocker for my setup. As all of our singers are using JackTrip images without a Jamulus UI, they don't have the possibility to change their own level within the mix. That won't work in my setup where the sound engineer has to have the control over ALL channels.
I know that this feature was one of the reasons for that PR, but I didn't realize the consequences for my setup.
Maybe my use case is not a general one and adding the next switch to disable that feature would not help Jamulus users, which uses Jamulus with UI.
But I still believe that my setup has many advantages for bigger chorus, where most/many of the singers probably don't have any interest on meters and levels, they only want to hear the chorus sound where they contribute to. Moreover from the administration point of view, the JackTrip way is super easy and is sufficient for chorus singing. OK, but that are my concepts.
So I will remain on the singlemix branch as long as I was able to change the mastermix code for my "special" use case ...

[cwerling]

Documenting one more to do for myself here:

• Make sure the mastermix isn't lost when the mix master leaves (e.g. due to a short connection drop) -- otherwise all faders will probably be on max and people will have too loud mixes

[cwerling]

I blame these lines of code for a bug discovered in yesterday's rehearsal:
Some people were not able to control their own fader (the only enabled fader was the mix master's fader).

I noticed that the affected people's IPs showed up as "connected" twice in the server log:

2021-05-05 19:11:57, 94.223.115.113, connected (3) <-- connected once
2021-05-05 19:12:39, 94.223.115.113, connected (4) <-- connected twice
[...]
2021-05-05 19:16:01, 85.212.238.106, connected (12) <-- connected once
[...]
2021-05-05 19:19:49, 85.212.238.106, connected (18) <-- connected twice

Choir colleague 94.223.115.113 reported that she connected via WiFi initally and then plugged in Ethernet and turned of WiFi afterwards.

I assume this leaves the the CAudioMixerBoard in a somehow inconsistent state with iMyChannelID.

[cwerling]

To do:

• Reproduce
• Fix

[ann0see]

Any updates here? Setting to draft.

[cwerling]

Any updates here? Setting to draft.

No updates here, as my choir is currently able to practice in presence. Do you feel like my approach is still something the community is interested in? If so, I could try to make some room to finish up this MR.

My feeling was that alternative approaches like @pieterbos's could be more interesting.

[ann0see]

While his approach is more featured, my choir is still using yours since it’s simple - and seems more mature.
definitly I see people who are interested in this…

[cwerling]

While his approach is more featured, my choir is still using yours since it’s simple - and seems more mature.
definitly I see people who are interested in this…

It's nice to hear my code still finds use. Since I don't have good ways of testing any changes to the code, would you be able to help me troubleshoot open issues with it? For example, did you and your choir ever encounter this problem mentioned above?

Also, regarding compatibility with newer versions both on client and server side: Have you merged my changes ontop of a recent Jamulus version or are you using precisely my latest commit?

[ann0see]

No. I am using your last commit, and I didn’t really test for the bug. I‘m unsure if I can help too much with bug-hunting, since my choir is hopping between mastermix and non mastermix. A bug I still remember is that delay panning doesn’t seem to be synchronised.

[ann0see]

Merged this to a feature branch now.

[pljones]

@ann0see

Just looking through PRs... What happened here? This seems to have a merge with no commits.

[ann0see]

I‘ve merged it to a feature branch to allow others to work on it more easily. https://github.com/jamulussoftware/jamulus/tree/feature/cwerling/masterMix

[pljones]

Ah! Thanks... I remember now...