Difference between revisions of "Tutorial: Running MPI Programs on Hadoop Cluster"
| Line 3: | Line 3: | ||
---- | ---- | ||
| + | <br /> | ||
=Setup to run MPI Hadoop servers= | =Setup to run MPI Hadoop servers= | ||
| + | <br /> | ||
| + | <onlysmith> | ||
* Follow general directions from [http://www.linuxproblem.org/art_9.html this page], or follow the steps summarized below. | * Follow general directions from [http://www.linuxproblem.org/art_9.html this page], or follow the steps summarized below. | ||
| + | :* ssh to hadoop01 | ||
| + | <br /> | ||
| + | <source lang="text"> | ||
| + | ssh -Y yourusername@hadoop01.dyndns.org | ||
| + | </source> | ||
| + | <br /> | ||
| + | :* enter the following commands | ||
| − | + | <br /> | |
| − | < | + | <source lang="text"> |
| − | |||
| − | |||
| − | |||
| − | < | ||
| − | |||
ssh-keygen -t rsa ''(and press ENTER 3 times)'' | ssh-keygen -t rsa ''(and press ENTER 3 times)'' | ||
ls .ssh | ls .ssh | ||
| Line 20: | Line 25: | ||
ssh yourusername@hadoop02.dyndns.org mkdir -p .ssh | ssh yourusername@hadoop02.dyndns.org mkdir -p .ssh | ||
cat id_rsa.mpi.pub | ssh dominique@hadoop02.dyndns.org 'cat >> .ssh/authorized_keys' | cat id_rsa.mpi.pub | ssh dominique@hadoop02.dyndns.org 'cat >> .ssh/authorized_keys' | ||
| + | </source> | ||
| + | <br /> | ||
| + | |||
| + | :* Now ssh to hadoop02 and verify that you can ssh without password, as the authentication is now done through rsa keys. | ||
| + | :* exit from hadoop02 and find yourself again on hadoop01 | ||
| + | :* repeat the last 2 commands above (ssh and cat) for hadoop03, and hadoop04. | ||
| + | :* verify that you can ssh to hadoop03 and hadoop04 without password. | ||
| + | * You should now be all set with passwordless ssh to the hadoop01, 02, 03, and 04 cluster. | ||
</onlysmith> | </onlysmith> | ||
| − | |||
| − | |||
| − | |||
| − | |||
=Configuration= | =Configuration= | ||
Revision as of 11:04, 15 March 2017
--D. Thiebaut (talk) 13:57, 15 October 2013 (EDT)
Revised: --D. Thiebaut (talk) 12:02, 15 March 2017 (EDT)
Setup to run MPI Hadoop servers
Configuration
- create a file called hosts in the directory where the mpi programs are located.
- Store the following IP addresses in it:
