This blog will not be updated anymore

From now I will be posting about data science, python, R, statistics etc on a new blog: Astronomical Data Science. This blog will not be updated anymore. 

Calling R from inside a Jupyter notebook

I use mostly python + jupyter in my research--in this case astronomical data science--workflow. One challenge I have been facing recently is dealing with time series. Some of the routines I need to use were written in R. There are versions of them for python but they seem to be inferior. The question then is: is it possible to easily interface with R functions from inside a jupyter python notebook? The answer is--amazingly--yes!

To illustrate how easy this is, I create a jupyter notebook available on Gist. This notebook demonstrates how to:

1. generate some simple mock data with python/numpy
2. import that data on R
3. perform a linear fit using R's methods and load the results back to python
4. plot the R fit with python

This may sound complicated but it really isn't.

How to setup an IPython parallel cluster in your LAN via SSH

It has been a long hiatus since I last posted anything here so it is time to get back.

Today I will describe the following scenario: you have two or more machines (linux boxes or OS X) available in your LAN, and you would like to harness the power of their CPU to perform parallel computing with python. This is possible with IPython parallel and there are several ways to get it accomplished.

I will describe the steps required to configure a private IPython parallel cluster in your LAN using SSH. If everything works well, this should require about 30 min to 1 hour to be completed depending on your level of experience.

1. Command to create an IPython parallel profile:

ipython profile create --parallel --profile=ssh

2. Edit config file .ipython/profile_ssh/ipcluster_config.py in your home.

Specify the number of hosts and cores to be used:

c.SSHEngineSetLauncher.engines = {
 'macnemmen' : 4,
 'pcnemmen' : 8,
 'pcraniere' : 4,
}
where you specify the appropriate names of the machines in your LAN.

Specify the IP of controller (the main machine you use to launch ipcluster):

c.LocalControllerLauncher.controller_args = ["--ip=xx.xxx.x.xxx"]
where you make sure the IP is correct.

3. Make sure python, jupyter, ipython and ipyparallel are installed in each computer.

In my case, I use the Anaconda distribution in all machines.

4. Setup SSH

Create a .ssh/config file such that all hosts and corresponding usernames that will run the servers are conveniently referenced.

Setup passwordless SSH login in each client machine from the controller machine.

5. Create a common alias in each client host pointing to the engine launcher binary: 

This is in order to avoid the clients not finding the binary if it is in a nonstandard location.

e.g. create aliases in /opt/ipengine

6. Edit config file .ipython/ipcluster_config.py pointing to the launcher alias

c.SSHEngineSetLauncher.engine_cmd = ['/opt/ipengine']

7. Launch the engines in all machines in your "cluster" 

ipcluster start --profile='ssh' --debug

Testing the engines

Now it is time to test if the engines were launched successfully.

Test that they are active in IPython:

import ipyparallel
c=ipyparallel.Client(profile='ssh')
c.ids

The output of the last command should be a list with the number of elements matching the number of engines you launched. Otherwise, something went wrong.

Don't forget that the configuration files are located in .ipython/profile_ssh.

To learn how to use in practice such cluster to do CPU intensive tasks, you can read this tutorial.

References

• Using IPython for parallel computing 
• Setting up an IPython Cluster via SSH
• How to Set Up a Private IPython Parallel Cluster
• SSH ipcluster with local and remote engines

Parallel computing with IPython and Python

I uploaded to github a quick tutorial on how to parallelize easy computing tasks. I have chosen embarrassingly parallel examples which illustrate some of the powerful features of IPython.parallel and the multiprocessing module.

Examples included:

1. Parallel function mapping to a list of arguments (multiprocessing module)
2. Parallel execution of array function (scatter/gather) + parallel execution of scripts
3. Easy parallel Monte Carlo (parallel magics)

Parallel computing with Python. 

Please stop using colormaps which don't translate well to grayscale

I recently printed a paper with very nice results in B&W but the color images simply did not make sense when printed in grayscale (here is the paper if you are curious). Why? Not the best choice of colormap. Jake Vanderplas reminded me of this issue with his very nice blog post.

Please don't choose colormaps for the images in your paper which do not translate well when printed in grayscale.

Check out Nature's advice on color coding. Also check out the advice here.

Feel free to suggest other useful references about this issue in the comments.

Python Installation instructions (including IPython / IPython Notebook)

This page describes how to install Python and the other packages (Numpy, Scipy, IPython, IPython Notebook, Matplotlib) required for the course for Mac OS X, Linux and Windows.

Linux

In Linux, the installation instructions are pretty straightforward. Assuming that you are running Debian or Ubuntu, you just need to execute the following command in the terminal:

sudo apt-get install python-numpy python-scipy python-matplotlib ipython-notebook

For Fedora users, you can use the yum tool.

Mac OS X, Linux, Windows

We recommend downloading and installing the Anaconda Python distribution. The installations instructions are available here. 

Just download the installer and execute it with bash.

Anaconda includes most of the packages we will use and it is pretty easy to install additional packages if required, using the conda or pip command-line tools.

If the above two methods do not work for OS X

The MacPorts way

You can try installing everything using MacPorts. First download and install macports and then issue the following command in a terminal:

sudo port install py27-zmq py27-tornado py27-nose

The avove dependencies are required in order to run IPython notebook. Then run:

sudo port install py27-numpy py27-matplotlib py27-scipy py27-ipython

The advantage of this method is that it easy to do. The downsides:

• It can take a couple of hours to finish the installation depending on your machine and internet connection, since macports will download and compile everything as it goes. 
• If you like having the bleeding edge versions, note that it can take a while for them to be released on macports 
• Finally, macports can create conflicts between different python interpreters installed in your system

Using Apple’s Python interpreted and pip

If you feel adventurous, you can use Apple’s builtin python interpreter and install everything using pip. Please follow the instructions described in this blog.

If you run into trouble

Leave a comment here with the issue you found.

Distributed arrays for parallel applications

I came across recently a very promising module: DistArray. The idea behind DistArray is to

provide general multidimensional NumPy-like distributed arrays to Python. It intends to bring the strengths of NumPy to data-parallel high-performance computing. 

Neat!

Some examples for easily creating distributed arrays are given in this IPython notebook.

Unfortunately I could not test DistArray so far because I am getting weird errors in my system, probably related to installation issues with my MPI installation.

Frequentism and Bayesianism

Jake VanderPlas has been writing a series of posts discussing frequentism and bayesianism. They are well-written, clear and insightful and use IPython for the statistical analysis. Here, I compiled his posts on the topic for convenience.

Frequentism and Bayesianism: A Practical Introduction
where he synthesizes the philosophical and pragmatic aspects of the frequentist and Bayesian approaches as they relate to the analysis of scientific data.

Frequentism and Bayesianism II: When Results Differ
where he discusses the difference between frequentist and Bayesian in the treatment of nuisance parameters.

Frequentism and Bayesianism III: Confidence, Credibility, and why Frequentism and Science do not Mix
where he discusses the subtle difference between frequentist confidence intervals and Bayesian credible intervals.

Frequentism and Bayesianism IV: How to be a Bayesian in Python
where he describes how to do Bayesian statistics in python with emcee, PyMC and PyStan.

Linear regression with errors in X and Y with Python: BCES

I finally had the chance to upload my BCES linear regression python code to Github!

If you need to do linear regression with measurement errors in X and Y, including intrinsic scatter, please check it out. Even better, if you have suggestions to improve or speed up the code, please contribute by all means!

LaTeX tricks

Here is a collection of useful latex tricks I have been using in my papers.

How to add "sticky note" annotations to your manuscript

In the preamble of the document, add
\usepackage{todonotes}


For sticky notes:
\todo{your sticky note comment}

For notes placed in the text:
\todo[inline]{your sticky note comment}

For a missing figure box, such that you want to remind yourself to add a plot there later:

\missingfigure{figure description}

Read this for more options and the documentation of the todonotes package.

How to highlight or cross-out text

In the preamble of the document, add
\usepackage{color, soul}
\setstcolor{red}


For highlighted text:
\hl{your highlighted text}

For crossing-out text:

Check this out for more options.

For commenting out larger chunks of text

In the preamble of the document, add
\usepackage{verbatim}


Then put the text you want to comment out inside the environment:
\begin{comment}
everything your want to not show up in the document
\end{comment}

Can be way more powerful and faster than appending the % character to the beginning of each line.

Video tutorial: Bayesian data analysis with PyMC 3

I highly recommend the tutorial by Thomas Wiecki on using PyMC 3 to perform Bayesian data analysis. Some of the cool things he demonstrates in this ~50 min video:

• summary of Bayesian analysis and Bayesian theorem
• application for parameter estimation for a few introductory examples: coin-flipping experiment, simple linear regression
• new features of PyMC 3 with respect to v2
• how to construct a model in PyMC 3 and a few notes on samplers
• more advanced example: application to time series of correlated stocks, then linear regression of correlated stocks with a time-dependent slope (!!)

Interactive plots in ipython notebooks

A couple of ways of doing interactive plots and animations in an ipython notebook:

• Using plotly
• Using a little hack called ipywidgets

I am sure there are many other ways, and if you know of nice ways please post in the comments. 

Links via reddit.

Citing IPython in papers

If you feel like citing IPython in your papers, you can actually cite the following paper by Pérez & Granger (2007).

You can find the appropriate bibtex citation here.

Easily run fortran code from an IPython notebook cell

I just learned a potentially very useful trick for people running fortran code. By following the instructions on this link, you can copy and paste fortran code to an IPython notebook cell and then use a ipython magic command to run it on-the-fly. I haven't had time to test it yet, so if you have success with it, please let me know. Kudos to @fperez_org.

On a separate note, IPython 1.1 is out!

Update (Sep. 23rd 2013): Here is a tutorial on how to use the fortran IPython magic.

IPython 1.0 is out!

Be sure to update your IPython installation (I did it using pip).

And also, check out the IPython subreddit.

Parallel Bayes with IPython

Excellent tutorial from the folks at Continuum Analytics/Wakari on how to parallelize bayesian parameter estimation using IPython (IPcluster) and pyMC.

Following this tutorial, I was able to make my bayesian spectral fitting code (for astronomical purposes) parallel -- and much faster -- very quickly.

Bayesian Estimation with PyMC and IPCluster

Installation instructions (from a Python Boot Camp course)

I wrote some instructions on how to install python and relevant dependencies (numpy, matplotlib, ipython, ipython notebook etc) for OS X, linux and windows. These instructions are for an upcoming Python Boot Camp where I work aimed at students, researchers and engineers (mostly Earth and physical sciences)

I wanted to share these instructions since they may be useful to more people. If you have been keeping up with this blog, note that there is some repetition here.

Python Installation Instructions

This page describes how to install Python and the other packages (Numpy, Scipy, IPython, Matplotlib) required for the course for Mac OS X, Linux and Windows.

Linux

In Linux, the installation instructions are pretty straightforward. Assuming that you are running Debian or Ubuntu, you just need to execute the following command in the terminal:

sudo apt-get install python-numpy python-scipy python-matplotlib ipython-notebook

For Fedora users, you can use the yum tool.

Mac OS X, Windows

If you are affiliated with an academic institution

Then the easiest way to install Python and the other packages is to request an academic license and download the Enthought Canopy Python distribution. Enthought includes all the packages we will use during the course.

The installation instructions are available here, which has installers for Mac OS X and Windows.

Note that you need the academic license in order to install the 64-bit (recommended) version. The 32-bit version is free for all.

If you are not affiliated with an university

This is the case for example for GSFC employees and many GSFC postdocs. In this case, we recommend downloading and installing the Anaconda Python distribution. The installations instructions are available here.

Just download the installer and execute it.

Anaconda includes most of the packages we will use and it is pretty easy to install additional packages if required.

If the above two methods do not work for OS X

The MacPorts way

You can try installing everything using MacPorts. First download and install macports and then issue the following command in a terminal:

sudo port install py27-zmq py27-tornado py27-nose

The avove dependencies are required in order to run IPython notebook. Then run:

sudo port install py27-numpy py27-matplotlib py27-scipy py27-ipython

The advantage of this method is that it easy to do. The downsides:

• It can take a couple of hours to finish the installation depending on your machine and internet connection, since macports will download and compile everything as it goes. 
• If you like having the bleeding edge versions, note that it can take a while for them to be released on macports 
• Finally, macports can create conflicts between different python interpreters installed in your system

Using Apple’s Python interpreted and pip

If you feel adventurous, you can use Apple’s builtin python interpreter and install everything using pip. Please follow the instructions described in this blog.

If you run into trouble

Feel free to contact us. Leave a comment here with the issue you found.

Easy way of installing Python and scientific packages for OS X (non-academic users): Anaconda

Introduction

Mac OS X users know that getting python and its scientific packages (numpy, scipy, ipython etc) installed properly can be a tricky job.

If you are affiliated with a university, you can try the freely available and easy to install Enthought distribution (if you are not affiliated with a university, you can still get the Enthought dsitribution for free but you are stuck with 32 bits binaries).

One way of getting things installed properly (even if you are not affiliated with a university) is by using the native OS X python interpreter and installing everything via pip as I describe in this post. This method is a little tricky but works well for me. One advantage is that it is easy to install new packages in this way: just use pip.

Another way is using macports. This is relatively straightforward, but can take a long time to compile all the dependencies (hours) and, worse of all, can potentially create conflicts between different libraries and different python interpreters.

Anaconda

I recently came across an easy way of getting python and the scientific packages installed. It also provides a convenient framework for installing extra packages. This is the Anaconda python distribution provided by Continuum Analytics. Anaconda provides the most popular packages: numpy, scipy, ipython (+notebook) and even more - astropy, spyder, pandas etc. And 64 bits binaries! For everybody. For free.

How to install it? 

Pretty easy. Go to the downloads page. Download the installer for your operating system. After downloading the file, for OS X you just need to issue the command

 sh <downloaded file.sh>  

Piece of cake. You can choose to let the installer change your PATH variable. After that, when you invoke python or ipython, it will automatically call the appropriate binaries and you will have available all the important packages.

To install additional packages, you can use their conda package manager or the usual pip.

Comparing execution speed of Python, NumPy, Matlab, Fortran, etc

This is an interesting comparison of the numerical performance of python, numpy, fortran, java and matlab in performing some array operations.

The post is outdated (it is from 2009). Still, it demonstrates the importance of vectorizing array operations. It also shows that lower level languages like fortran will almost always have the upper hand in execution speed (but development time can easily balance that in python).

It would be worth repeating this comparison with the newer versions of numpy, python, matlab and fortran compilers to see what has changed in terms of relative speedups among the different languages. It would also be nice to include a "parallel" case of these examples using some multiprocessing module in python in a multicore machine (e.g. multiprocessing, iPython parallel).

Video tutorial on iPython Parallel

Here is a 1 hour long video tutorial on iPython parallel by Min Ragan-Kelley. Pretty cool. The material he went over is available here.