Parallel processing of jobs with OpenStudio 2.1 Linux

openstudio
workflow
linux
retag add tags

To run multiple models, I modify the contents in compact.osw by using Python, and then change the models by using the compact.osw. After that, I can run the new models. (''' cd path/to/Examples/compact_osw ./path/to/openstudio.exe run -w compact.osw ''') Now, I want to make the works parallel. Because I use OpenStudio 2.1 in Ubuntu and decide to do sampling methods locally, I do not want to use PAT to do the work. Does anyone have some experience about such work. Are any packages in Python, such as joblib.Parallel, pp, recommended to do the work?

816

Yunyang Ye

asked 2017-06-02 09:27:10 -0500

4.4k

__AmirRoth__

updated 2017-08-06 07:30:17 -0500

edit flag offensive 0 remove flag close merge delete

Comments

add a comment see more comments

2 Answers

Sort by

oldest

newest

most voted

You need to provision your machine with OpenStudio Server and invoke the OpenStudio Meta CLI directly for your application. OpenStudio Server has an embedded instance of R to manage parallelization of algorithms. Learn more here.

3.8k

ljbrackney

answered 2017-06-02 09:32:30 -0500

edit flag offensive 0 remove flag delete link

Comments

Does it use the snow (Simple Network of Workstations) package in R to manage parallelization of algorithms?

Yunyang Ye (2017-06-02 09:53:51 -0500) edit

Snowfall, which I believe is derivative of SNOW.

ljbrackney (2017-06-02 10:49:20 -0500) edit

add a comment see more comments

You could use python's threading module to make multiple calls to the CLI in parallel. The subprocess module can be used for the actual call. (For ruby, the parallel gem coupled with a system call would work nicely.)

You'll also need to specify a different run_directory (and maybe others?) in the OSW so that the runs don't try to use the same directory -- or simply put the OSWs in different directories.

11.8k

shorowit

answered 2017-06-02 10:50:05 -0500, updated 2017-06-02 10:52:20 -0500

edit flag offensive 0 remove flag delete link

Comments

Thanks. I will try to do the work by using the method that you mention.

Yunyang Ye (2017-06-02 11:47:36 -0500) edit

@Yunyang Ye Did this work?

Determinant (2017-09-13 18:51:31 -0500) edit

@Determinant I am working on that in a new supercomputer now. It seems work, and if I have a new progress, I will tell you.

Yunyang Ye (2017-09-28 15:37:06 -0500) edit

@Yunyang Ye that's really great. Thanks for doing this. If they stop supporting the PAT on Linux this will be our only way to go. Please do keep me informed.

Determinant (2017-10-04 17:36:54 -0500) edit

I have running code using Python's subprocess.call. It's great but the problem is that the directories in the OSW have to be absolute. The default directory used by the openstudio command doesn't work for some reason. Absolute paths make the results subdirectories messy. Any ideas?

Determinant (2018-08-27 15:12:38 -0500) edit