Parallel computing with package ‘snowfall’

Lately I have been looking for ways to decrease the amount of time it takes me to run multiple regressions over a very large data set. There are several options that I am investigating to do this, and certainly more that I don’t know of yet.

  • Code more efficiently.
  • Compute several operations in parallel over a two or more CPU cores.
  • Tap into a network of computers, and further expand the number of CPU cores to parallelize calculations.

Because many of my computer jobs are “embarassingly parallel”, the options mentioned above would immediately improve the speed I can compute (and re-compute) jobs. This post will go through an example using the CRAN package snowfall to parallelize a computation over several CPU cores on the same computer (bullet #2 above).

The CRAN package snowfall is built to make it easy to create parallel processes. I recommend taking a look at the associated vignette and tutorial.

Before beginning to use snowfall, do the following:

  1. Upgrade to the latest version of R – as of this post version 2.14.1 (or the patched version of R-2.13.0 – available here). FYI – There is a bug in version 2.13.0 (for MS Windows 7) that prevents snowfall from operating smoothly.
  2. Install the latest version of the package snowfall ( install.packages('snowfall', dependencies = TRUE) )
  3. Find out how many cores you have on the CPU of the machine you will be using.  In my example below, I am using a machine with 8 CPU cores and running Windows 7.
  4. Convert any ‘for’ loops into a function that you can call using apply(). See my previous post that outlines this process.

Using snowfall: A simple example

The reason I put together this post is because I couldn’t easily find a ‘plug’n play’ code example in the existing online literature to execute the type of parallelization I wanted. Out of necessity I worked through the wrinkles and am now successfully utilizing multiple CPU cores in R.  –  Note: By default, R uses only one CPU core unless you explicitly code it to use multiple cores (as in this example). Continue reading “Parallel computing with package ‘snowfall’”


Why use R? A grad student’s 2 cents

One of the problems I faced this past year was deciding which software package to use — for statistical analyses, homework problems, and my thesis research. A handful of professors here use SAS, many use Stata, a few use Matlab, and one uses R (that I know of). After a semester using SAS, and despite having only one professor on the R “team” — I decided to learn R.

Here’s why:

  • R is free.  While I could get student discounts on SAS or Stata, or use the school computer lab, I like my software to be there for me always.  If I want to run a regression at 2:00 am using the wi-fi of a Holiday Inn Express, I should be able to run that awesome regression.  I can install R on every computer that I need (home, office, laptop, friends, enemies, etc.).  This is helpful because the I like to work in a variety of places, and having all my tools on my person is required.  *If I had to boil this list down to the one reason I’m using R right now, it’s because of price.  You can’t. beat. free.
  • R has really good online documentation; and the community is unparalleled.  One of the primary motivations for this blog is to give back to the R community that has helped me learn and appreciate the software.  I want to mitigate the fixed-costs of learning R, help others in their quest to tackle data-driven analyses, and spread the good word.  The more people who use R, the more people with which I can potentially collaborate.
  • I like the command-line interface.  You can use the command-line interface in other programs like SAS and Stata.  But, when you are starting out — is that really what you use?  It wasn’t for me.  Why?  Because I didn’t know any better — I was just starting out!  The command-line interface is perfect for learning by doing.  You can immediately see the results from inputting a single line of code.  If there are errors, you can fiddle with your code and re-hit [enter].  This is the way I learn things, and surely I’m not alone.
  • R is on the cutting edge, and expanding rapidly.  If you follow any of the online communities that work with R, you will notice all the new packages being rolled out — almost daily!  R is on the forefront of statistical methods, and can be integrated from any number of other languages – be it Python, Java, Fortran, etc.
  • The R programming language is intuitive.  One of the aspects I liked about R when I first started out is that it just worked.  I wrote a function that followed my thought process, and bam! – it worked.  Immediately it was improving my productivity, without having to know too much about coding or dig through a manual.
  • R creates stunning visuals.  See below; some of my favorites.  And I’m still a beginner.  Using Hadley Wickham’s ggplot2 and the stock imaging platform, it is straightforward to generate sharp diagrams.
  • R and LaTeX work together — seamlessly.  If you use LaTeX, you are in luck.  I am writing my thesis in LaTeX, and just recently stumbled upon R’s tikzDevice package.  This package outputs images as TikZ code for direct compilation in .tex.  For outputting multiple images, using loops, and reducing the file size of my thesis, this has been a huge plus.
  • R is used by practitioners in a plethora of academic disciplines.  R users come from myriad industries and academic departments, be it sociology, immunology, economics, statistics, paleontology, anthropology, finance, marketing analytics, etc.  This cross pollination is healthy for the enterprising student.  By seeing familiar concepts used in other disciplines, and through a different lens, it helps solidify your own understanding.  Furthermore, this expanded user base increases the likelihood that something useful to you will be added to the next CRAN package or version of R.
  • R makes you think.  Some statistical packages make it easy to perform many useful tasks via canned functions. For economists, Stata is one of those such programs.  However, being forced to code a procedure by hand, though more time consuming, helps make it “stick”.  And the more you get acquainted with R’s many packages, the more you will stumble upon a canned function that will do exactly what you want.  But even if that availability exists, R makes is relatively straightforward to code your own procedure, and then check to make sure the two routes return the same results.
  • There’s always more than one way to accomplish something.  Similar to the preceding point, I find it extremely helpful to tackle a problem two ways (or more), and make sure my results match.  When I find that they don’t, I am forced to really learn what’s going on “under the hood” — and in consequence, expand my knowledge of R and econometrics.
So, do a bit of research and make an informed decision about what software you invest the time and energy to learn.  If you do, I’m confident you’ll see the potential in R and give it a shot.

Did I forget anything?  — Why do you use R to dominate your data analysis?

R: apply() + function = no need for loops

In my research, I am constantly running the same computation over every combination of month-day-year-hour in a given sample’s time period. Traditionally, this can be done using loops, like so:


[sourcecode language=”r”]
k = 2008 # year start
j = 1 # month start
i = 1 # day start
h = 1 # hour start

# start nested loops:
for (k in 2008:2010) {
for (j in 1:12) {
for (i in 1:31) {
for (h in 1:24) {

print(paste(‘The date is ‘,paste(j,i,k,sep=’/’),’ hour ‘,h,sep=”))


However, there is a cleaner, more efficient way to go. That is, to write a function that takes the day, month, year, etc. as input parameters, and call it using apply(). For a great explanation and introduction to using apply(), sapply(), lapply(), and other derivatives of apply(), see this excellent poston Neil Saunders blog: “What You’re Doing is Rather Desperate”.

To follow our silly example from above, we could create a function that prints the date and hour:

[sourcecode language=”r”]
dateprint = function(MM,DD,YR,HR) {
print(paste(‘The date is ‘,paste(MM,DD,YR,sep=’/’),’ hour ‘,HR,sep=”))

Then we could call the function as follows:

[sourcecode language=”r”]
k = c(2008:2010) # year range
j = c(1:12) # month range
i = c(1:31) # day range
h = c(1:24) # hour range

# Call function using apply() and defined parameters
output = apply(expand.grid(j,i,k,h), 1,
function(x,y,z,a) dateprint(x[1],x[2],x[3],x[4]))

# Apply stores the output as a list
# I like to convert it to a dataframe for easier viewing and manipulation.

Notice that you are essentially giving apply() an “input matrix” created by expand.grid(); apply() takes parameters from each row of that “input matrix” and feeds them to our dateprint() function. You can tell apply() to take parameters from each column by changing the “1” to a “2” within your call of apply().
I am not too close with the back end of R, so I am not certain that using apply() will increase the computational efficiency of your code. That said, it is another approach to solving a common problem, and one I use often. Furthermore, it cleans up your code a scintilla.
Clean code = happy code.