thread workers parameter in PySAM ResourceTools.FetchResourceFiles

  • Greg Miller
  • Topic Author
More
03 Dec 2021 22:24 #10337 by Greg Miller
The PySAM ResourceTools.FetchResourceFiles function has a parameter called "workers", which allows you to specify the number of threads to use when parellelizing downloads (nrel-pysam.readthedocs.io/en/master/Tools.html?highlight=workers#files.ResourceTools.FetchResourceFiles).

I'm looking to allow this parameter to be dynamically defined on the user's machine depending on the capability of the machine to run parallel downloads. My question is whether parallelized downloads depends on the number of CPU threads on the machine, or if it is related to some other parameter about the computer or network.

Say one is trying to download hundreds of resource files - how would you suggest setting the workers parameter to maximize parallelization? In other words, what is the upper limit on the number of workers that can run in parallel?

Thank you!

Please Log in or Create an account to join the conversation.

  • Paul Gilman
More
07 Dec 2021 16:54 #10350 by Paul Gilman
Hi Greg,

The NSRDB API rate limits are likely to be more of a constraint than your machine's speed. See this NSRDB API guide for information about those limits: developer.nrel.gov/docs/solar/nsrdb/guide/. You'll have to do a few test downloads to see what parallelization settings result in the speeds you want.

The functions in FetchResourceFiles are designed to download a single file at a time. The guide describes how to download sets of files. You might want to fork off of ResourceTools.py to make modifications for requests of sets of files rather than a single one.

Best regards,
Paul.

Please Log in or Create an account to join the conversation.

Moderators: Paul Gilman
Powered by Kunena Forum