Tutorial: Running MPI Programs on Hadoop Cluster

--D. Thiebaut (talk) 13:57, 15 October 2013 (EDT)
Revised: --D. Thiebaut (talk) 12:02, 15 March 2017 (EDT)

---

Setup to run MPI Hadoop servers

• Follow general directions from this page, or follow the steps summarized below.

1. ssh to hadoop01

This section is only visible to computers located at Smith College

1. enter the following commands

ssh-keygen -t rsa  (and press ENTER 3 times)
ls .ssh
cd .ssh
mv id_rsa id_rsa.mpi
mv id_rsa.pub id_rsa.mpi.pub
ssh yourusername@hadoop02.dyndns.org mkdir -p .ssh
cat id_rsa.mpi.pub | ssh dominique@hadoop02.dyndns.org 'cat >> .ssh/authorized_keys'

</onlysmith>

• Now ssh to hadoop02 and verify that you can ssh without password, as the authentication is now done through rsa keys.
• exit from hadoop02 and find yourself again on hadoop01
• repeat the last 2 commands above (ssh and cat) for hadoop03, and hadoop04.
• verify that you can ssh to hadoop03 and hadoop04 without password.

Configuration

• create a file called hosts in the directory where the mpi programs are located.
• Store the following IP addresses in it:

This section is only visible to computers located at Smith College