Calling R from inside a Jupyter notebook

I use mostly python + jupyter in my research--in this case astronomical data science--workflow. One challenge I have been facing recently is dealing with time series. Some of the routines I need to use were written in R. There are versions of them for python but they seem to be inferior. The question then is: is it possible to easily interface with R functions from inside a jupyter python notebook? The answer is--amazingly--yes!

To illustrate how easy this is, I create a jupyter notebook available on Gist. This notebook demonstrates how to:

1. generate some simple mock data with python/numpy
2. import that data on R
3. perform a linear fit using R's methods and load the results back to python
4. plot the R fit with python

This may sound complicated but it really isn't.

Parallel computing with IPython and Python

I uploaded to github a quick tutorial on how to parallelize easy computing tasks. I have chosen embarrassingly parallel examples which illustrate some of the powerful features of IPython.parallel and the multiprocessing module.

Examples included:

1. Parallel function mapping to a list of arguments (multiprocessing module)
2. Parallel execution of array function (scatter/gather) + parallel execution of scripts
3. Easy parallel Monte Carlo (parallel magics)

Parallel computing with Python. 

LaTeX tricks

Here is a collection of useful latex tricks I have been using in my papers.

How to add "sticky note" annotations to your manuscript

In the preamble of the document, add
\usepackage{todonotes}


For sticky notes:
\todo{your sticky note comment}

For notes placed in the text:
\todo[inline]{your sticky note comment}

For a missing figure box, such that you want to remind yourself to add a plot there later:

\missingfigure{figure description}

Read this for more options and the documentation of the todonotes package.

How to highlight or cross-out text

In the preamble of the document, add
\usepackage{color, soul}
\setstcolor{red}


For highlighted text:
\hl{your highlighted text}

For crossing-out text:

Check this out for more options.

For commenting out larger chunks of text

In the preamble of the document, add
\usepackage{verbatim}


Then put the text you want to comment out inside the environment:
\begin{comment}
everything your want to not show up in the document
\end{comment}

Can be way more powerful and faster than appending the % character to the beginning of each line.

Easily run fortran code from an IPython notebook cell

I just learned a potentially very useful trick for people running fortran code. By following the instructions on this link, you can copy and paste fortran code to an IPython notebook cell and then use a ipython magic command to run it on-the-fly. I haven't had time to test it yet, so if you have success with it, please let me know. Kudos to @fperez_org.

On a separate note, IPython 1.1 is out!

Update (Sep. 23rd 2013): Here is a tutorial on how to use the fortran IPython magic.

Parallel Bayes with IPython

Excellent tutorial from the folks at Continuum Analytics/Wakari on how to parallelize bayesian parameter estimation using IPython (IPcluster) and pyMC.

Following this tutorial, I was able to make my bayesian spectral fitting code (for astronomical purposes) parallel -- and much faster -- very quickly.

Bayesian Estimation with PyMC and IPCluster

Shortcut to open an iPython notebook and web browser from the terminal

In order to start an iPython notebook server and open a notebook in your browser you can issue the command

>>> ipython notebook --pylab inline

Sometimes, mysteriously, the web browser will not automatically open after issue the above command on Mac OS X. In that case, you can use the command

>>> ipython notebook --pylab inline && open http://127.0.0.1:8888/'

Example of how to do a great animation in Python

If you are interested in creating animation of simulation data in Python, perhaps this should give you some inspiration: Animating the Lorenz System in 3D.

The author solved the Lorenz system of equations and plotted the time evolution of the system in 3D. It illustrates how to make a great animation in a simple way.

You should also check out the Matplotlib Animation Tutorial written by the author, which provides the background on the Lorenz animation.

MacPorts update broke your ipython/matplotlib installation?

I recently updated my MacPorts installation just to find out that it broke my ipython installation, even though my python environment is configured independently from macports.

The error message was more or less like this: I run the command

ipython --pylab

and matplotlib does not work, complaining about the missing the library file libpng14.14.dylib with a message "image not found".

If you installed python, ipython and matplotlib following my tutorial (i.e. independently from the macports python installation), you just need to reinstall matplotlib with pip:

pip uninstall matplotlib
pip install matplotlib

This will reconfigure matplotlib with the updated dependencies.

Parallel computing in Python for the masses

Case scenario: you wrote a python routine that does some kind of time-consuming computation. Then you think, wow, my computer has N cores but my program is using only one of them at a time. What a waste of computing resources. Is there a reasonably easy way of modifying my code to make it exploit all the cores of my multicore machine?

The answer is yes and there are different ways of doing it. It depends on how complex your code is and which method you choose to parallelize your computation.

I will talk here about one relatively easy way of speeding up your code using the multiprocessing python package. I should mention that there are many other options out there but the multiprocessing package comes pre-installed with any python distribution by default.

I am assuming that you really need to make your code parallel. You will have to stop and spend time thinking about how to break your computation in smaller parts that will be sent to the different cores. And I should mention that debugging is harder for parallel code compared to serial code, obviously.

Parallelization is one way of optimizing your code. Other ideas for optimizing your code is using Cython or f2py. Both these approaches may imply >10x speedup and are worth exploring depending on your situation. But both will involve using the C or Fortran languages along with your python code.

The ideal case is when your problem "embarassingly parallel". What I mean by this is: your problem  can be made parallel in a reasonably easy way since the computations which correspond to the bottleneck of the code can be carried out independently and do not need to communicate between each other. Examples:

• You have a "grid" of parameters that you need to pass to a time-consuming model (e.g., a 1000x1000 matrix with the values of two parameters). Your model needs to evaluate those parameters and provide some output.
• Your code performs a Monte Carlo simulation with 100000 trials which are carried out in a loop. You can then easily "dismember" this loop and send it to be computed independently by the cores in your machine.

Instead of giving code examples myself, I will point out the material I used to learn parallelization. I learned the basics of parallel programming by reading the excellent tutorial "introduction to parallel programming" written by Blaise Barney.

The next step was learning how to use the multiprocessing package. I learned this with the examples posted in the AstroBetter blog. I began by reading the example implemented with the pprocess package. The caveat here is that 'pprocess' is a non-standard package. The multiprocessing package which comes with python should be used instead. Somebody posted the original example discussed in the blog ported to the multiprocessing package.

As the posts above explain, the basic idea behind using 'multiprocessing' is to use the parallel map method to evaluate your time-consuming function using the many cores in your machine. Once you figure out a way of expressing your calculation in terms of the 'map' method, the rest is easy.

In my experience doing parallel programming in python using 'multiprocessing' I learned a few things which I want to share:

1. Do not forget to close the parallel engine with the close() method after your computation is done! If you do not do this, you will end up leaving a lot of orphan processes which can quickly consume the available memory in your machine.
2. Avoid using lambda functions when passing arguments to the parallel 'map' at all costs! Trust me,  multiprocessing does not play well with lambda constructs.
3. Finally, as I mentioned before, parallelizing a code increases development time and the complexity of debugging your code. Only resort to parallelization if you really need it, i.e. if you think you will get a big speedup in your code execution. For example, if you code takes 24 hours to execute and you think you can get a 6x speedup by resorting to 'multiprocessing', then the execution time can be reduced to 4 hours which is not bad.

Hidden features of Python

I learned about this link with many useful hidden features of Python via Eduardo.

I particularly like:

• the use of enumerator in loops: for i,x in enumerate(array)
• decorators as a simple way of enhancing methods: @method
• one-line swapping of variables: a,b=b,a

Simple progress bar in terminal

If you need to incorporate a simple progress bar in your code, there is a module that does that and is very easy to use: fish.

The following script illustrates how to implement a simple progress bar which advances each time a loop counter increases.

 import fish  
 import time  
   
 steps=input('How many steps? ')  
   
 # Progress bar initialization  
 peixe = fish.ProgressFish(total=steps)       
   
 for i in range(steps):  
      # Progress bar  
      peixe.animate(amount=i)  
        
      time.sleep(0.1)  

Here is a screenshot of what the progress bar looks like in action:

Additional color names for matplotlib plots

When creating plots with matplotlib, we usually use the "default" color names:

• b : blue
• g : green
• r : red
• c : cyan
• m : magenta
• y : yellow
• k : black
• w : white

It turns out that matplotlib accepts not only these default color names, but the full range of html color names! So for example, you can plot some data like this:

 pylab.plot(x, y, 'DarkOrange')  

and get a "dark orange" color.

This is an easy way of specifying colors in addition to the standard ones.