Ergo does not currently provide direct support for remote execution from the analysis framework; however, remote execution is still possible with the current version. This page will lay out some thoughts we've had for handling remote execution in the current version. There are plans for the next version of Ergo to refactor the analysis framework to bring in a scientific workflow system that can provide better support for accessing remote resources.
Requirements
The remote machine that will execute the arbitrary code must have the software and supporting libraries already installed on the machine
Custom Iterator
For the case of remote execution, the built in simple iterator will not be sufficient since all of the work will likely be done in a single job submission submitted to a remote queuing system. You can learn more about creating a custom iterator in the Ergo Developer Tutorial. The custom iterator will allow you to control how execution happens for an analysis task. The analysis task will be responsible for setting up the job, submitting the job, communicating progress with the queuing system and finally storing results once the analysis is complete.
Analysis Task
As mentioned previously, the analysis task you create will be responsible for:
- Staging data to the remote machine in the format the executable expects
- Setting up and Submitting the job to the remote machine
- Communicating with the remote queue to determine progress. This would also allow you to indicate progress in the Ergo workflow (not required).
- Staging data back to the local machine once the job is finished.
- Storing the data through Ergo's data mechanism for storing job results. This may require converting the results from the format the executable generates to what Ergo expects.
The developer tutorial provides a step by step example to illustrate creating a custom iterator and analysis task. This should be enough to provide the boilerplate of what would be required for getting started with remote execution. You will likely need to add new plug-ins/libraries to handle the communication and authentication/authorization with the remote server (e.g. REST, SSH, File Transfer, etc). For an example analysis that uses a custom iterator/task, see the edu.illinois.ncsa.ergo.eq.nbsr plug-in.