Counting the Number of Hosts in an XGrid

This short tutorial shows how to count the number of Hosts and Cores in an XGrid

Setup

The idea is to run a short Python program that finds the host name, IP address, and number of cores of the Mac it runs on, and to make this program run on as many hosts in the XGrid as possible.

Getting the System Info

The program source, mostly due to Margaret Zaccardi, is shown below. It gets the information (host name, Ip, and number of cores) from the host it runs on.

.

#! /usr/bin/env python
# xgridCount.py
# Margaret Zaccardi
# Counts the number of Macs in the XGrid
#
# Typical usage:
#
#   ./xgridCount.py
#   xgridmac.local (131.229.101.251) #8
#


def getSystemInfo():
    import socket
    import subprocess
    import re
    import time

    name   = socket.gethostname()
    IP     = socket.gethostbyname( name )

    systemInfo = subprocess.Popen(
                ['/usr/sbin/system_profiler SPHardwareDataType'],
                shell=True, 
                stdout=subprocess.PIPE).communicate()
    regex = re.compile(r'Total Number Of Cores:(.*?)\n')
    cores = regex.findall(systemInfo[0])[0].strip()
    print "%s (%s) #%s" % ( name, IP, str( cores ) )

    # sleep 3 seconds to make sure another version of this program will be
    # forked on another host, and not the current host.
    time.sleep( 3 )

getSystemInfo()

.

Running 100 copies of this program

We can use a batch file to create 100 instances of the program, or use a bash-for loop.

Batch File

{
    jobSpecification =     {
        applicationIdentifier = "com.apple.xgrid.cli";
        inputFiles =         {
              "xgridCount.py" = { fileData = <2321202f 7573722f 62696e2f 656e7620 70797468 6f6e0a23 20786772 6964436f 756e742e 70790a23 204d6172 67617265 74205a61 63636172 64690a23 20436f75 6e747320 74686520 6e756d62 6572206f 66204d61 63732069 6e207468 65205847 7269640a 230a0a0a 64656620 67657453 79737465 6d496e66 6f28293a 0a202020 20696d70 6f727420 736f636b 65740a20 20202069 6d706f72 74207375 6270726f 63657373 0a202020 20696d70 6f727420 72650a20 20202069 6d706f72 74207469 6d650a0a 20202020 6e616d65 2020203d 20736f63 6b65742e 67657468 6f73746e 616d6528 290a2020 20204950 20202020 203d2073 6f636b65 742e6765 74686f73 7462796e 616d6528 206e616d 6520290a 0a202020 20737973 74656d49 6e666f20 3d207375 6270726f 63657373 2e506f70 656e280a 20202020 20202020 20202020 20202020 5b272f75 73722f73 62696e2f 73797374 656d5f70 726f6669 6c657220 53504861 72647761 72654461 74615479 7065275d 2c0a2020 20202020 20202020 20202020 20207368 656c6c3d 54727565 2c200a20 20202020 20202020 20202020 20202073 74646f75 743d7375 6270726f 63657373 2e504950 45292e63 6f6d6d75 6e696361 74652829 0a202020 20726567 6578203d 2072652e 636f6d70 696c6528 7227546f 74616c20 4e756d62 6572204f 6620436f 7265733a 282e2a3f 295c6e27 290a2020 2020636f 72657320 3d207265 6765782e 66696e64 616c6c28 73797374 656d496e 666f5b30 5d295b30 5d2e7374 72697028 290a2020 20202370 72696e74 2022686f 7374206e 616d6520 20202020 20203d20 222c206e 616d650a 20202020 23707269 6e742022 49502061 64647265 73732020 20202020 3d20222c 2049500a 20202020 23707269 6e742022 4e756d62 6572206f 6620636f 72657320 3d20222c 20636f72 65730a20 20202070 72696e74 20222573 20282573 29202325 73222025 2028206e 616d652c 2049502c 20737472 2820636f 72657320 2920290a 20202020 74696d65 2e736c65 65702820 3320290a 0a676574 53797374 656d496e 666f2829 0a0a0a>; isExecutable = YES; }; 
        };
        name = dft;
        schedulerHints =  { 0 = mathgrid5; };
        submissionIdentifier = "dft batch job";
        taskSpecifications = {
        0 = { arguments = (  ); command = "./xgridCount.py" ;  };
        1 = { arguments = (  ); command = "./xgridCount.py" ;  };
        2 = { arguments = (  ); command = "./xgridCount.py" ;  };
        3 = { arguments = (  ); command = "./xgridCount.py" ;  };
        4 = { arguments = (  ); command = "./xgridCount.py" ;  };
        .
        .
        .
        97 = { arguments = (  ); command = "./xgridCount.py" ;  };
        98 = { arguments = (  ); command = "./xgridCount.py" ;  };
        99 = { arguments = (  ); command = "./xgridCount.py" ;  };
        100 = { arguments = (  ); command = "./xgridCount.py" ;  };
        };
    };
}

For-Loop

We can use a for-loop to run the 100 different instances:

for i in {1..100}; do xgrid -job submit xgridCount.py ; done | ./getXGridOutput.py 

Job 2745 stopped: Execution time: 4.000000 seconds
ford-243-16.local (131.229.99.229) #2
Job 2746 stopped: Execution time: 3.000000 seconds
xgridmac.local (131.229.101.251) #8
Job 2747 stopped: Execution time: 3.000000 seconds
ford-243-12.local (131.229.102.224) #2
Job 2748 stopped: Execution time: 3.000000 seconds
ford-241-13.local (131.229.100.229) #2
Job 2749 stopped: Execution time: 3.000000 seconds
xgridmac.local (131.229.101.251) #8
Job 2750 stopped: Execution time: 3.000000 seconds
xgridmac.local (131.229.101.251) #8

Total execution time: 14.000000 seconds

Filtering the Output

We scan the output for unique hostname/Ip configurations, extract the number of hosts and print the total for both:

.
#! /usr/bin/env python
import sys

dico = {}
for line in sys.stdin.readlines():
    if line.find( "131.229" )==-1: continue
    noCores = int( line.split( '#' )[1].strip() )
    dico[ line ] = noCores

noCores = 0
noHosts = 0
for key in dico.keys():
    noHosts += 1
    noCores += dico[key]

print "Number of servers in XGrid: ", noHosts
print "Number of cores in XGrid:   ", noCores

.

Wrap Up

Finding the number of hosts and core is now simply a matter of piping all the various commands together:

for i in {1..100}; do xgrid -job submit xgridCount.py ; done | ./getXGridOutput.py | ./countHosts.py 

Number of servers in XGrid:  26
Number of cores in XGrid:    58

or

xgrid -job batch run100.batch | getXGridOutput.py | countHosts.py 

Number of servers in XGrid:  26
Number of cores in XGrid:    58