Multiprocess parallelization

[Thuener]

I'm running a very computationally heavy solution and I need to parallelize each request individually. That means I will have many processes running the code for one API request.

I'm trying to create an API using Oxygen that would allow to have a multi-process parallelization (using multiple cores).
Running Oxygen with julia -p 4 or

using Distributed
Distributed.addprocs(4)

Doesn't lead to parallelisation. Is there any recommendation on how to use multiple processes with Oxygen.jl?

[ndortega]

Hi @Thuener,

At the moment there is nothing built in for handling requests across different processes. Distributing common functions across processes and calling them from a request handler would probably be the quickest way to solve this issue. Just be wary of any race conditions that could come from this.

I will let you know that I have a private project in the works around solving this kind of issue, but it's still too early to release publicly. The aim is to make calling distributed code as simple as possible and completely thread/process safe. In the meantime, you can always look into other distributed compute libraries like Dagger.jl or MPI.jl which can help with the scheduling/communication between processes.

[Thuener]

Thanks.

In my case, I'm using SDDP.jl, and it has the built-in parallelization; I just had to activate it. I was able to activate it using some tricks. So I can confirm that Oxygen.jl works for my case. I have tried before Genie.jl but it did not worked.

I will leave the solution here just in case someone needs it in the future. I encapsulated all the content into one module (for example, ModuleExample.jl). I had to start with just one thread and use serve() instead of serveparallel().

On the API side I imported on all workers that module (ModuleExample.jl):

using Distributed
Distributed.remotecall_eval(Main, procs(), :(include("./ModuleExample.jl")))

That did the job and I was able to run SDDP.jl with multiple CPU cores.

[mzy2240]

@Thuener thats really cool! so all your server setup (Oxygen stuff) and functional logics are excapsulated into ModuleExample.jl, but then how did you manage the port conflict and does it support load-balancing in a magical way?