Neptune with Distributed Optuna

I am using Optuna’s distributed capability on a HPC (see https://optuna.readthedocs.io/en/stable/tutorial/003_rdb.html), but I can see that each worker creates their own neptune experiment in the neptune ui in the project instead of 1 canonical experiment (that has multiple nodes contributing). Is there a recommended way to create 1 experiment and allow multiple clients to select the “active” experiment? For this case, each worker runs with n_jobs =1 for optuna, but I could have N workers.

some of the code examples in Neptune and hyperparameter search with Tune look promising, but I have not looked into it deeply yet and I was wondering if there was an accepted good way of doing this, as I can guess there would be a race condition.

Hi @aannex,

Quick question:
Do you use our integration with optuna: https://docs.neptune.ai/integrations/optuna.html


One simple idea…
…is to create neptune experiment object, like this:

exp = neptune.create_experiment(name='canonical-exp-1')

Then pass exp object to method/class that will run computations for the experiment in question. Inside, you log to this exp, like this:

exp.log_metric()
exp.log_text()

Hope this helps :slight_smile: