It has been a long hiatus since I last posted anything here so it is time to get back.
Today I will describe the following scenario: you have two or more machines (linux boxes or OS X) available in your LAN, and you would like to harness the power of their CPU to perform parallel computing with python. This is possible with IPython parallel and there are several ways to get it accomplished.
I will describe the steps required to configure a private IPython parallel cluster in your LAN using SSH. If everything works well, this should require about 30 min to 1 hour to be completed depending on your level of experience.
ipython profile create --parallel --profile=ssh
Specify the number of hosts and cores to be used:
c.SSHEngineSetLauncher.engines = {
'macnemmen' : 4,
'pcnemmen' : 8,
'pcraniere' : 4,
}
where you specify the appropriate names of the machines in your LAN.
Specify the IP of controller (the main machine you use to launch ipcluster):
c.LocalControllerLauncher.controller_args = ["--ip=xx.xxx.x.xxx"]
where you make sure the IP is correct.
In my case, I use the Anaconda distribution in all machines.
Create a .ssh/config file such that all hosts and corresponding usernames that will run the servers are conveniently referenced.
Setup passwordless SSH login in each client machine from the controller machine.
This is in order to avoid the clients not finding the binary if it is in a nonstandard location.
c.SSHEngineSetLauncher.engine_cmd = ['/opt/ipengine']
ipcluster start --profile='ssh' --debug
Now it is time to test if the engines were launched successfully.
import ipyparallel
c=ipyparallel.Client(profile='ssh')
c.ids
The output of the last command should be a list with the number of elements matching the number of engines you launched. Otherwise, something went wrong.
Don't forget that the configuration files are located in .ipython/profile_ssh.
To learn how to use in practice such cluster to do CPU intensive tasks, you can read this tutorial.
Today I will describe the following scenario: you have two or more machines (linux boxes or OS X) available in your LAN, and you would like to harness the power of their CPU to perform parallel computing with python. This is possible with IPython parallel and there are several ways to get it accomplished.
I will describe the steps required to configure a private IPython parallel cluster in your LAN using SSH. If everything works well, this should require about 30 min to 1 hour to be completed depending on your level of experience.
1. Command to create an IPython parallel profile:
ipython profile create --parallel --profile=ssh
2. Edit config file .ipython/profile_ssh/ipcluster_config.py in your home.
Specify the number of hosts and cores to be used:
c.SSHEngineSetLauncher.engines = {
'macnemmen' : 4,
'pcnemmen' : 8,
'pcraniere' : 4,
}
where you specify the appropriate names of the machines in your LAN.
Specify the IP of controller (the main machine you use to launch ipcluster):
c.LocalControllerLauncher.controller_args = ["--ip=xx.xxx.x.xxx"]
where you make sure the IP is correct.
3. Make sure python, jupyter, ipython and ipyparallel are installed in each computer.
In my case, I use the Anaconda distribution in all machines.
4. Setup SSH
Create a .ssh/config file such that all hosts and corresponding usernames that will run the servers are conveniently referenced.
Setup passwordless SSH login in each client machine from the controller machine.
5. Create a common alias in each client host pointing to the engine launcher binary:
This is in order to avoid the clients not finding the binary if it is in a nonstandard location.
e.g. create aliases in /opt/ipengine
6. Edit config file .ipython/ipcluster_config.py pointing to the launcher alias
c.SSHEngineSetLauncher.engine_cmd = ['/opt/ipengine']
7. Launch the engines in all machines in your "cluster"
ipcluster start --profile='ssh' --debug
Testing the engines
Now it is time to test if the engines were launched successfully.
Test that they are active in IPython:
import ipyparallel
c=ipyparallel.Client(profile='ssh')
c.ids
The output of the last command should be a list with the number of elements matching the number of engines you launched. Otherwise, something went wrong.
Don't forget that the configuration files are located in .ipython/profile_ssh.
To learn how to use in practice such cluster to do CPU intensive tasks, you can read this tutorial.
