In the last experiments my pods have remained in the waiting state for too long. For example, https://app.neptune.ml/meddulla/santander-customer-prediction/e/SAN2-10/details waited for 8 hours until I aborted it… The last one is now waiting for 8 minutes so maybe the same is going to happen…
We recently performed some maintenance on our infrastructure and we are currently experiencing some issues that sometimes cause experiments to not start properly.
We are currently working on a solution.
We deployed a fix that should solve this issue.