Friday, February 19, 2016

How to setup an IPython parallel cluster in your LAN via SSH

It has been a long hiatus since I last posted anything here so it is time to get back.

Today I will describe the following scenario: you have two or more machines (linux boxes or OS X) available in your LAN, and you would like to harness the power of their CPU to perform parallel computing with python. This is possible with IPython parallel and there are several ways to get it accomplished.

I will describe the steps required to configure a private IPython parallel cluster in your LAN using SSH. If everything works well, this should require about 30 min to 1 hour to be completed depending on your level of experience.

1. Command to create an IPython parallel profile:


ipython profile create --parallel --profile=ssh


2. Edit config file .ipython/profile_ssh/ipcluster_config.py in your home.


Specify the number of hosts and cores to be used:

c.SSHEngineSetLauncher.engines = {
 'macnemmen' : 4,
 'pcnemmen' : 8,
 'pcraniere' : 4,
}
where you specify the appropriate names of the machines in your LAN.

Specify the IP of controller (the main machine you use to launch ipcluster):

c.LocalControllerLauncher.controller_args = ["--ip=xx.xxx.x.xxx"]
where you make sure the IP is correct.

3. Make sure python, jupyter, ipython and ipyparallel are installed in each computer.


In my case, I use the Anaconda distribution in all machines.

4. Setup SSH


Create a .ssh/config file such that all hosts and corresponding usernames that will run the servers are conveniently referenced.

Setup passwordless SSH login in each client machine from the controller machine.

5. Create a common alias in each client host pointing to the engine launcher binary: 


This is in order to avoid the clients not finding the binary if it is in a nonstandard location.
e.g. create aliases in /opt/ipengine

6. Edit config file .ipython/ipcluster_config.py pointing to the launcher alias


c.SSHEngineSetLauncher.engine_cmd = ['/opt/ipengine']


7. Launch the engines in all machines in your "cluster" 


ipcluster start --profile='ssh' --debug


Testing the engines


Now it is time to test if the engines were launched successfully.

Test that they are active in IPython:

import ipyparallel
c=ipyparallel.Client(profile='ssh')
c.ids

The output of the last command should be a list with the number of elements matching the number of engines you launched. Otherwise, something went wrong.

Don't forget that the configuration files are located in .ipython/profile_ssh.

To learn how to use in practice such cluster to do CPU intensive tasks, you can read this tutorial.


References

7 comments:

  1. The actual replica watches for sale comes after the actual traditional style along with slimmer lugs as well as hollowed out precious metal middle hyperlinks. Even though cartier replica sale adheres towards the traditional Datejust style when it comes to appearance, it will function a few crucial technical breakthroughs which let it perform upon par along with cheap replica watches choices. Whilst classic Rolex referrals existing a good unparalleled degree of cartier replica sale as well as collectability, they frequently absence the current functions which eventually improve functionality as well as easy rolex replica watches like a quick-set day perform. The significant function about the Rolex Replica Sale is actually it's utilization of Rolex's quality 3135 motion. Very first launched within 1988, the actual quality 3135 continues to be within manufacturing these days, as well as Rolex utilizes this in order to energy nearly all their own three-hand.

    ReplyDelete

  2. Well Done ! the blog is great and Interactive it is about AstroPython Blog useful for students and Python Developers for more updates on python

    python online course

    ReplyDelete
  3. Thanks for sharing about astropython. we are clearly understand this concept. This blog useful for students and python developers for more updates on python Hire Python Developer

    ReplyDelete
  4. This comment has been removed by the author.

    ReplyDelete
  5. We're here to explain how it all works with a world777 cricket betting tutorial in the following article. We'll go over the many formats utilized in the sport and how different bets might result in different odds of winning. We'll also go over how to bet on cricket, as well as the many ways you may bet on cricket and where you can put your bets, as well as answer some key questions concerning cricket betting.

    ReplyDelete
  6. So how do you know how much your Java developer salary will be? The answer depends on your experience, skillset and education level. Many variables will determine what your salary is as a Java developer. Salaries for Java Developers vary depending on location, experience, education, certifications and type of employment.

    ReplyDelete