Neptune + Optuna race condition

Conversation

User
https://docs.neptune.ai/integrations/optuna.html
study.optimize(objective, n_trials=100, callbacks=[monitor], n_jobs=2)
Using n_jobs > 1 causes a bug, for example the both of first two trials have trial.number == 0.

Kamil
thanks for reaching out
We will take a closer look at this example
In the meantime - did you manage to run optuna with neptune anyways?

User
yes, it worked great with n_jobs=1. One missing thing is I manually generate the optuna visualizations (slice, paralle-coordinates, contour plot, etc) at the end of experiment
It would be nice to generate them in realtime on each point update
instead of generating png and upload

Kuba
I created some updates to the optuna integration that actually upload those charts after the training is done as interactive plotly charts:

https://ui.neptune.ai/o/shared/org/showroom/e/SHOW-1018/artifacts?path=charts%2F&file=parallel_coordinate.html

You can also specify NeptuneCallback(log_charts=True) to log it (update) after every iteration.

Was that something you were thinking about?

The PR is under review right now.

Bad news is I couldn’t reproduce your n_jobs=2 problem on my machine.
Could you share a minimal failing example?

note

Originally posted on spectrum on May 17, 2020, migrated here on Jun 5, 2020.