View source for Tutorial: Computing Pi on an AWS MPI-Cluster

--[[User:Thiebaut|D. Thiebaut]] ([[User talk:Thiebaut|talk]]) 22:55, 28 October 2013 (EDT)<br />
Updated --[[User:Thiebaut|D. Thiebaut]] ([[User talk:Thiebaut|talk]]) 12:13, 16 March 2017 (EDT)
----
<bluebox>
This tutorial is the continuation of the tutorial [http://www.science.smith.edu/dftwiki/index.php/Tutorial:_Create_an_MPI_Cluster_on_the_Amazon_Elastic_Cloud_(EC2) Creating and MPI-Cluster on AWS].  Make sure you go through it first.  The tutorial also assumes that you have available credentials to access the Amazon AWS system.
</bluebox>


=Reference Material=
<br />
I have used 
* the [http://mpitutorial.com/launching-an-amazon-ec2-mpi-cluster/ the MPI Tutorial from mpitutorial.com], and
* the [http://star.mit.edu/cluster/docs/latest/quickstart.html StarCluster quick-start tutorial] from the StarCluster project<br />
for inspiration and adapted them in the present page to setup a cluster for the CSC352 class.  The setup uses  MIT's [http://star.mit.edu/cluster/ Star Cluster] package.

<br />

=An MPI C-Program for Approximating Pi=
<br />
The program below works for any number of processes on an MPI cluster or on a single computer.  
<br />
<br />
<source lang="C">
// piN.c
// D. Thiebaut
// Computes Pi using N processes under MPI
// 
// To compile and run:
// mpicc -o piN piN.cpp
// time mpirun -np 2 ./piN 100000000
//
// Output
// Process 1 of 2 started on beowulf2.  N= 50000000
// Process 0 of 2 started on beowulf2.  N= 50000000
//  50000000 iterations: Pi = 3.14159
//
//  real0m1.251s
//  user0m1.240s
//  sys0m0.000s
//

#include "mpi.h"
#include <stdio.h>
#include <stdlib.h>

#define MANAGER 0


//--------------------------------------------------------------------
//                         P R O T O T Y P E S
//--------------------------------------------------------------------
void doManager( int, int );
void doWorker( );

//--------------------------------------------------------------------
//                           M  A  I  N
//--------------------------------------------------------------------
int main(int argc, char *argv[]) {
  int myId, noProcs, nameLen;
  char procName[MPI_MAX_PROCESSOR_NAME];
  int n;

  if ( argc<2 ) {
    printf( "Syntax: mpirun -np noProcs piN n\n" );
    return 1;
  }

  // get the number of samples to generate
  n = atoi( argv[1] );
  
  //--- start MPI ---
  MPI_Init( &argc, &argv);
  MPI_Comm_rank( MPI_COMM_WORLD, &myId );
  MPI_Comm_size( MPI_COMM_WORLD, &noProcs );
  MPI_Get_processor_name( procName, &nameLen );
  
  //--- display which process we are, and how many there are ---
  printf( "Process %d of %d started on %s. n = %d\n", 
          myId,  noProcs, procName, n );

  //--- farm out the work: 1 manager, several workers ---
  if ( myId == MANAGER ) 
    doManager( n, noProcs );
  else
    doWorker( );
  
  //--- close up MPI ---
  MPI_Finalize();
  
  return 0;
}

//--------------------------------------------------------------------
// The function to be evaluated
//--------------------------------------------------------------------
double f( double x ) {
  return 4.0 / ( 1.0 + x*x );
}

//--------------------------------------------------------------------
// The manager's main work function.  Note that the function
// can and should be made more efficient (and faster) by sending
// an array of 3 ints rather than 3 separate ints to each worker.  
// However the current method is explicit and better highlights the
// communication pattern between Manager and Workers.
//--------------------------------------------------------------------
void doManager( int n, int noProcs ) {
  double sum0 = 0, sum1;
  double deltaX = 1.0/n;
  int i, begin, end;

  MPI_Status status;

  //--- first send n and bounds of series to all workers ---
  end = n/noProcs;
  for ( i=1; i<noProcs; i++ ) {
    begin = end;
    end   = (i+1) * n / noProcs;
    
    MPI_Send( &begin, 1, MPI_INT, i /*node i*/, 0, MPI_COMM_WORLD );
    MPI_Send( &end, 1, MPI_INT, i /*node i*/, 0, MPI_COMM_WORLD );
    MPI_Send( &n, 1, MPI_INT, i /*node i*/, 0, MPI_COMM_WORLD );
  }

  //--- perform summation over 1st interval of the series ---
  begin = 0;
  end   = n/noProcs;

  for ( i = begin; i < end; i++ )
    sum0 += f( i * deltaX );

  //--- wait for other half from worker ---
  for ( i=1; i<noProcs; i++ ) {
    MPI_Recv( &sum1, 1, MPI_DOUBLE, MPI_ANY_SOURCE, 0, MPI_COMM_WORLD, &status );
    sum0 += sum1;
  }

  //--- output result ---
  printf( "%d iterations: Pi = %f\n", n, sum0 *deltaX );
}

//--------------------------------------------------------------------
// The worker's main work function.  Same comment as for the
// Manager.  The 3 ints would benefit from being sent in an array.
//--------------------------------------------------------------------
void doWorker( ) {
  int begin, end, n, i;

  //--- get n and bounds for summation from manager ---
  MPI_Status status;
  MPI_Recv( &begin, 1, MPI_INT, MPI_ANY_SOURCE, 0, MPI_COMM_WORLD, &status );
  MPI_Recv( &end, 1, MPI_INT, MPI_ANY_SOURCE, 0, MPI_COMM_WORLD, &status );
  MPI_Recv( &n, 1, MPI_INT, MPI_ANY_SOURCE, 0, MPI_COMM_WORLD, &status );

  //--- sum over boundaries received ---
  double sum = 0;
  double deltaX = 1.0/n;

  for ( i=begin; i< end; i++ )
    sum += f( i * deltaX );

  //-- send result to manager ---
  MPI_Send( &sum, 1, MPI_DOUBLE, MANAGER, 0, MPI_COMM_WORLD );
}

</source>
<br />

=Creating a 10-node cluster=
<br />
<tanbox>
If you are using the AWS account that was given to you in class, it is ''imperative'' that you identify your clusters in a unique way, as Amazon somehow stores the clusters in the same "space" reserved for your instructor.  Therefore, make sure you suffix your cluster names with your initials!
</tanbox>
<br />
* Connect to the machine you have access to, and on which '''starcluster''' is installed (your laptop, or aurora).
* Make sure you do not have a cluster running:
 
  '''starcluster listclusters'''
 
:make sure a cluster with your initials does not appear in the output.  If you see one, then terminate it (replace ABC by your initials):

  '''starcluster terminate myclusterABC'''

:or, if the first command is not successful:

  '''starcluster terminate -f myclusterABC'''


* Edit the starcluster '''configuration file''' on your machine (laptop or aurora):

  '''emacs -nw ~/.starcluster/config'''
 
* Locate the following line, and set the number of nodes to 10:

  # number of ec2 instances to launch                                                                                
  CLUSTER_SIZE = '''10'''

* Save the config file.

* Start up a new cluster of 10 instances.  It's fine to use the same name:

 ''' starcluster start myclusterABC'''
<source lang="text">

StarCluster - (http://star.mit.edu/cluster) (v. 0.95.6)
Software Tools for Academics and Researchers (STAR)
Please submit bug reports to starcluster@mit.edu

*** WARNING - Setting 'EC2_PRIVATE_KEY' from environment...
*** WARNING - Setting 'EC2_CERT' from environment...
>>> Using default cluster template: smallcluster
  .
  .
  .
You can activate a 'stopped' cluster by passing the -x
option to the 'start' command:

    $ starcluster start -x myclusterABC

This will start all 'stopped' nodes and reconfigure the
cluster.
</source>
<br />

=SSH to the Master node=
<br />

SSH to the master node, and become User '''sgeadmin''':

  '''starcluster sshmaster myclusterABC'''
  ...
 root@master:~# '''su - sgeadmin'''
 sgeadmin@master:~$  
 
<br />
=Upload the '''PiN.c''' program to the Cluster=
<br />
This step will upload a program from the machine you use to type your '''starcluster''' command, to your AWS cluster.  If you are using starcluster directly on your laptop, then you will need to have the '''piN.c''' program on your laptop.  If you are using aurora to enter '''starcluster''' commands, then you need the '''piN.c''' program to be on aurora.

* Go to the directory where your version of '''piN.c''' is located.  
 
  '''cd'''
  '''cd   ''pathToWhereYourFileIsLocated'' '''
  
* Upload '''piN.c''' to your AWS cluster:

  '''starcluster  put  myclusterABC  --user sgeadmin piN.c  .''' &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;''(don't forget the dot at the end!)''


<br />

=Create a '''host''' file=
<br />
Mpirun.mpich2 will by default launch all the processes you start on the node on which you run the command.  By default mpirun.mpich2 assumes only one node in the cluster.  To force it to launch the processes on other nodes in the cluster you need to put the names of all nodes in a file called the '''host file'''.

Starcluster uses a simple system to name the nodes.  The master node is called '''master''' and the workers are called '''node001''', '''node002''', etc...

* You can use emacs to create a file called '''hosts''' in the /home/mpi directory with the name of your nodes, or you can use bash and do it with a one-line command:

 sgeadmin@master:~$ '''cat /etc/hosts | tail -n 10 | cut -d' ' -f 2 > hosts'''

* Verify that the hosts file contains the required information:
  
  sgeadmin@master:~$ '''cat hosts'''
  master
  node001
  node002
  node003
  node004
  node005
  node006
  node007
  node008
  node009

This list will be used by the '''mpirun.mpich2''' program to figure out which node to distribute the processes to.
<br />
=Compile and Run the PiN.c program on the 10-node cluster=
<br />
  
  sgeadmin@master:~$ '''mpicc -o piN piN.c'''
 
 
  sgeadmin@master:~$ '''mpirun.mpich2 -n 10 -f hosts ./piN 10000000'''
  Process 0 of 10 started on master. n = 10000000
 Process 7 of 10 started on node007. n = 10000000
 Process 3 of 10 started on node003. n = 10000000
 Process 6 of 10 started on node006. n = 10000000
 Process 2 of 10 started on node002. n = 10000000
 Process 8 of 10 started on node008. n = 10000000
 Process 9 of 10 started on node009. n = 10000000
 Process 5 of 10 started on node005. n = 10000000
 Process 4 of 10 started on node004. n = 10000000
 Process 1 of 10 started on node001. n = 10000000
 10000000 iterations: Pi = 3.141593
 

<br />
If you play around you will find that for large values of ''n'' the computation of pi will get noticeably off the mark.  The formula we use works well only for a limited number of terms.
<br />

=Timing the Execution Time of your MPI Program=
<br />
 sgeadmin@master:~$  '''time mpirun.mpich2 -n 10 -f hosts ./piN 1000000000'''
 Process 0 of 10 started on master. n = 1000000000
 Process 3 of 10 started on node003. n = 1000000000
 Process 6 of 10 started on node006. n = 1000000000
 Process 2 of 10 started on node002. n = 1000000000
 Process 4 of 10 started on node004. n = 1000000000
 Process 7 of 10 started on node007. n = 1000000000
 Process 9 of 10 started on node009. n = 1000000000
 Process 1 of 10 started on node001. n = 1000000000
 Process 5 of 10 started on node005. n = 1000000000
 Process 8 of 10 started on node008. n = 1000000000
 1000000000 iterations: Pi = 3.171246
 
 ''real	0m4.911s''
 ''user	0m4.072s''
 ''sys	0m0.156s''
 
<br />
<br />
<br />

{| style="width:100%; background:orange"
|-
|

===Challenge 1===
|}
[[Image:QuestionMark1.jpg|right|120px]]
<br />
* Running ''P'' MPI processes on ''P'' nodes seems logical, but it certainly is not the limitation.  The figure below shows a situation where ''2P'' MPI processes are running on ''P'' nodes.
<br />
[[Image:2N_MPIProcessesOnNNodes.png|600px|center]]
<br />
* While keeping number of terms in the summation ''N'' constant, play with the number of MPI processes, equal to the number of nodes in your cluster, twice the number of nodes, three times, etc, and time the execution of the parallel program.   See if there is a configuration that runs faster than the others.
<br />

<br />
{| style="width:100%; background:orange"
|-
|
===Challenge 2===
|}
[[Image:QuestionMark2.jpg|right|120px]]
<br />
Terminate your cluster, and edit your starcluster config file (emacs ~/.starcluster/config).  Change the '''m3.medium''' instance type to '''m3.large'''.   '''M3.large''' provides 2 cores (that Amazon calls ''virtual CPUs''), rather than 1 for '''m3.medium'''.    Restart your cluster, upload your '''piN.c''' file  again, compile it, create a host file, and perform the same measurements you did in Challenge 1.   Do you observe any improvement in execution speed?
<br />
When you are done, return your instance type to '''m3.medium'''.
<br />
<br />
<br />
<br />
<br />
<br />
{| style="width:100%; background:orange"
|-
|

===Challenge 3===
|}
[[Image:QuestionMark3.jpg|right|120px]]
<br />Looking at the charts below, figure out how much it has cost to run your MPI application on AWS on the two different clusters (assuming that you were using spot instances, and not free tier instances).  Note that Amazon charges by the hour.  So if you use 5 minutes of time on your cluster (between start and stop operations), Amazon charges for an hour of computer time.   Also, if you use 10 machines in your cluster, you need to multiply the price for the instance by 10!
<br />
<br />
<br /><br />
<br /><br />
<br /><br />
<br />

===Types of AMIs===
<br />
For information, below is the comparison chart of the different ''spot instances'' or ''AMIs'' (Amazon Machine Instances) available.    Spot prices (see [https://aws.amazon.com/ec2/spot/pricing/ aws.amazon.com/ec2/spot/pricing]) are set by Amazon fEC2 and fluctuate periodically depending on the supply of and demand for Spot instance capacity.  The table below is taken from Amazon's spot pricing page, and displays the Spot price for each region and instance type.  The page is updated every 5 minutes.  

The default we set in the '''Starcluster''' config file is the '''m3.medium''', but notice that there are many others available.  Of course the more powerful machines come at a higher price.  But given that some have many cores, the elevated cost for them is worth the potential boost in performance to be gained.
<br />
<center>[[Image:awsInstancesMType2017.png|600px]]
</center>
<br />

<br />
<br /><br /><br /><br /><br /><br /><br /><br /><br /><br />
<br /><br /><br /><br /><br /><br /><br /><br /><br /><br />
[[Category:Tutorials]][[Category:C]][[Category:AWS]][[Category:MPI]]