Difference between revisions of "Tutorial: Running MPI Programs on Hadoop Cluster"
(→Setup password-less ssh) |
|||
| Line 1: | Line 1: | ||
| − | --[[User:Thiebaut|D. Thiebaut]] ([[User talk:Thiebaut|talk]]) 13:57, 15 October 2013 (EDT) | + | --[[User:Thiebaut|D. Thiebaut]] ([[User talk:Thiebaut|talk]]) 13:57, 15 October 2013 (EDT)<br /> |
| + | Revised: --[[User:Thiebaut|D. Thiebaut]] ([[User talk:Thiebaut|talk]]) 12:02, 15 March 2017 (EDT) | ||
---- | ---- | ||
| − | =Setup to run MPI | + | =Setup to run MPI Hadoop servers= |
| − | * Follow general directions from [ | + | * Follow general directions from [http://www.linuxproblem.org/art_9.html this page], or follow the steps summarized below. |
| − | + | :# ssh to hadoop01 | |
| − | + | <onlysmith> | |
| − | + | ||
| − | + | ssh -Y yourusername@hadoop01.dyndns.org | |
| − | <onlysmith> | + | |
| − | |||
| − | |||
| − | |||
</onlysmith> | </onlysmith> | ||
| − | + | :# enter the following commands | |
| − | |||
ssh-keygen -t rsa ''(and press ENTER 3 times)'' | ssh-keygen -t rsa ''(and press ENTER 3 times)'' | ||
ls .ssh | ls .ssh | ||
| Line 21: | Line 18: | ||
mv id_rsa id_rsa.mpi | mv id_rsa id_rsa.mpi | ||
mv id_rsa.pub id_rsa.mpi.pub | mv id_rsa.pub id_rsa.mpi.pub | ||
| − | + | ssh yourusername@hadoop02.dyndns.org mkdir -p .ssh | |
| − | + | cat id_rsa.mpi.pub | ssh dominique@hadoop02.dyndns.org 'cat >> .ssh/authorized_keys' | |
| − | + | ||
| − | + | </onlysmith> | |
| − | + | * Now ssh to hadoop02 and verify that you can ssh without password, as the authentication is now done through rsa keys. | |
| − | + | * exit from hadoop02 and find yourself again on hadoop01 | |
| − | + | * repeat the last 2 commands above (ssh and cat) for hadoop03, and hadoop04. | |
| − | + | * verify that you can ssh to hadoop03 and hadoop04 without password. | |
| − | |||
| − | * | ||
=Configuration= | =Configuration= | ||
Revision as of 11:02, 15 March 2017
--D. Thiebaut (talk) 13:57, 15 October 2013 (EDT)
Revised: --D. Thiebaut (talk) 12:02, 15 March 2017 (EDT)
Setup to run MPI Hadoop servers
- Follow general directions from this page, or follow the steps summarized below.
- ssh to hadoop01
- enter the following commands
ssh-keygen -t rsa (and press ENTER 3 times) ls .ssh cd .ssh mv id_rsa id_rsa.mpi mv id_rsa.pub id_rsa.mpi.pub ssh yourusername@hadoop02.dyndns.org mkdir -p .ssh cat id_rsa.mpi.pub | ssh dominique@hadoop02.dyndns.org 'cat >> .ssh/authorized_keys'
</onlysmith>
- Now ssh to hadoop02 and verify that you can ssh without password, as the authentication is now done through rsa keys.
- exit from hadoop02 and find yourself again on hadoop01
- repeat the last 2 commands above (ssh and cat) for hadoop03, and hadoop04.
- verify that you can ssh to hadoop03 and hadoop04 without password.
Configuration
- create a file called hosts in the directory where the mpi programs are located.
- Store the following IP addresses in it:
