API Uploading Limit seems to cap very soon

Hi Neptune Community,

I am using Neptune for a simple logging of metrics. The experiment runs through quite fast and uploading the rest of the logs takes a very long time in comparison. I have the feeling that my use case should not reach any limit yet.

Local run-time for experiment: ~2min
Number of metrics to log: 9
Number of time-steps: 18000
Experiment logging size: ~15MB
Total run-time: ~25min

It seems to me that I am hitting the limit of API Calls and have to wait for a minute until I can upload the next batch. However, the data size is still quite small.
What is there that I can change to avoid this behaviour? Can I increase the batch-size until the upload or save the experiments locally and upload all in one go afterwards?

Thank you for your help!

Hi @fabianp,

Thanks for reaching out!

As I understand, you are sending 9*18k = 162k points, so it may take a while to digest it on our side.
You may also want to check your Internet connection and make sure it’s fast :slight_smile:

General rule of thump to avoid mentioned delay it is to simply log less data to Neptune. Any technique/trick that will send data less frequently will work. Also, consider logging large files at the very end of the experiment.

@Wajcha do you have some more tips/explanations?

Regards,
Kamil

Hi @kamil.kaczmarek,
thank you for your reply! Reducing the amount of points is one thing that I will try to speed things up a little.

I checked my Internet Connection and there should be no bottleneck there.

Could you give some specific examples on how the send data less frequently? I am using ‘log_metric(name, value)’ to indicate which metric to store for every timestep. How can I queue it up before sending it?
Is there a way to store it in a list / dictionary and send the whole list / dictionary in one step (and still have the plots automatically at hand while using the UI)?

Thank you for your help!
Best,
Fabian

Hi @fabianp,

log_metric() is queued automatically and values are sent approximately every 5 seconds as a package. Here, you cannot do anything else.

Speaking about sending data less frequently. I was thinking about the idea that you already mentioned, that is reducing the amount of points, which after all I would not recommend as this will reduce amount of info that you have available for analysis and comparison.

Unfortunately, at this point you cannot store/accumulate data locally and send later. However, good news are that we work on it!

Regards,
Kamil

Hi @kamil.kaczmarek,

unfortunately, there is no way around using less points for logging. Even though, as you pointed out already, I lose information this way. Are there any other limits than the number of API Calls per minute? I am asking as the upload speed varies strongly for every experiment, even though the code runs at pretty much the same speed.

Thank you for your answer!
Best,
Fabian

Hey @fabianp,

I talked more to the engineering team - it turn out that we had some temporary performance issues. This was a root cause of the delay and upload speed variation that you observed.
Situation is back to normal, you may want to try your experiment again - it should be faster.

Note that there is also storage limit for all your data in Neptune.

Regards,
Kamil

Hi @kamil.kaczmarek,

thank your for having a closer look.
I tried to run it again and the performance issues that I reported are still the same. However, I checked again and you are right there were several experiments which even took 20x longer during my testing time. This would be the ones that reflect the server delay. Unfortunately, this does not solve my problem.

I am looking forward to the feature of manually uploading the experiments in a batch.
Thank you for your help,
Regards,
Fabian

Hi @fabianp,

I will make sure that your need regarding uploading the experiments in a batch will make it’s way to the product team :slight_smile:

Cheers,
Kamil