Running Qt C++ Programs on the XGrid

From dftwiki3
Revision as of 11:17, 3 June 2010 by Thiebaut (talk | contribs) (→‎Running the Qt3 Application on the XGrid)
Jump to: navigation, search

--D. Thiebaut 16:12, 3 June 2010 (UTC)

XgridLogo.png

This short How-To page shows how to deploy a Qt3 C++ application on the XGrid. The problem addressed is that the Qt libraries might not be installed on the XGrid client, which will make the application fail when it starts running on a client.

The Problem

  • We have an executable generated by compiling a C++ program using the Qt 3.6 libraries.
  • Want to run the Qt executable on the XGrid.
  • Qt 3.6 is not installed on the XGrid clients
  • Qt was installed assuming its libraries would be loaded dynamically.
  • We don't want to reinstall or reconfigure Qt.

The solution

  • Use otool to find all the libraries that are non system libraries (we assume they are all in /opt/local/lib) and assume that they will not be on the XGrid clients.
  • Copy all these libraries next to the executable, and modify the executable with install_name_tool such that the path of the libraries as it wants to refer to them is not absolute, but becomes whatever the executable path is.
For example, if the executable contains a reference to /opt/local/lib/libqt-mt.3.dylib, it should be changed to @executable_path/libqt-mt.3.dylib.
  • This process is repeated for all the libraries brought in, so that if they too rely on non-system libraries, they will be modified to have a new path to them, and these libraries will be brought in in turn.
  • makeStaticPackage.py must be started in an empty directory and copies into it the original executable, as well as all the non-system libraries, one after the other. It then generates a text file that can be used by another Python script, makeBatchMulti.py, for creating an XGrid batch file.

Example: creating a Qt3 application and deploying it on the XGrid

Compiling

  • Simply go through the process of compiling Qt3 C++ applications:
cd filterwiki9
ls
main.cpp  engine.h  engine.cpp  Makefile filterwiki.pro
  
export QTDIR=/opt/local
rm Makefile
qmake -makefile
gmake

g++-4.0 -c -pipe -Wall -W -Os  -D__DARWIN_X11__ -DQT_NO_DEBUG -I/opt/local/lib/qt3/mkspecs/default -I. \
      -I/opt/local/include/qt3 -I/opt/local/lib/qt3/include -I/opt/local/include -o main.o main.cpp
g++-4.0 -c -pipe -Wall -W -Os  -D__DARWIN_X11__ -DQT_NO_DEBUG -I/opt/local/lib/qt3/mkspecs/default -I. \
      -I/opt/local/include/qt3 -I/opt/local/lib/qt3/include -I/opt/local/include -o engine.o engine.cpp
g++-4.0 -c -pipe -Wall -W -Os  -D__DARWIN_X11__ -DQT_NO_DEBUG -I/opt/local/lib/qt3/mkspecs/default -I. \
      -I/opt/local/include/qt3 -I/opt/local/lib/qt3/include -I/opt/local/include -o moc_engine.o moc_engine.cpp
test -d ../../../bin/ || mkdir -p ../../../bin/
g++-4.0 -Wl,-dylib_file,/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGL.dylib: \
      /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGL.dylib -prebind \
      -o ../../../bin/filterwiki9 main.o engine.o moc_engine.o   -L/opt/local/lib -L/opt/local/lib -L/opt/local/lib \ 
      -L/opt/local/include/qt3/ -lqt-mt -lXext -lX11 -lm

Creating the "static package"

cp ~/bin/filterwiki9 .
mkdir XGridVersion
cd XGridVersion
makeStaticPackage.py ../filterwiki9
----------------------- SUMMARY ----------------------  
Copied and processed /opt/local/lib/libfreetype.6.dylib
Copied and processed /opt/local/lib/libqt-mt.3.dylib
Copied and processed /opt/local/lib/libICE.6.dylib
Copied and processed /opt/local/lib/libcrypto.0.9.8.dylib
Copied and processed /opt/local/lib/libXmu.6.dylib
Copied and processed /opt/local/lib/libXfixes.3.dylib
Copied and processed /opt/local/lib/libXcursor.1.dylib
Copied and processed /opt/local/lib/libSM.6.dylib
Copied and processed /opt/local/lib/libGL.1.dylib
Copied and processed /opt/local/lib/libXext.6.dylib
Copied and processed /opt/local/lib/libXt.6.dylib
Copied and processed /opt/local/lib/libXdmcp.6.dylib
Copied and processed /opt/local/lib/libfontconfig.1.dylib
Copied and processed /opt/local/lib/libz.1.dylib
Copied and processed /opt/local/lib/libX11.6.dylib
Copied and processed /opt/local/lib/libXinerama.1.dylib
Copied and processed /opt/local/lib/libXau.6.dylib
Copied and processed /opt/local/lib/libexpat.1.dylib
Copied and processed /opt/local/lib/libssl.0.9.8.dylib
Copied and processed /opt/local/lib/libXrender.1.dylib
Copied and processed ../filterwiki9
Copied and processed /opt/local/lib/libXrandr.2.dylib
Copied and processed /opt/local/lib/mysql5/mysql/libmysqlclient.16.dylib
Copied and processed /opt/local/lib/libXft.2.dylib
Copied and processed /opt/local/lib/libiconv.2.dylib
------------------------------------------------------


To create batch file, type
makeBatchMulti.py filterwiki9.batch < filterwiki9.in

Creating the XGrid Batch file

  • Simply execute the command listed by the previous execution of makeStaticPackage.py
makeBatchMulti.py filterwiki9.batch < filterwiki9.in


XGrid batch file maker 
Creates a batch file for the XGrid system from a collection of 
data files and programs


Please enter your name to identify the batch job: 
Please enter the names of the different programs needed by the batch job.
Enter them one per line.  Press Enter twice when done.
Program #1 > Program #2 > Program #3 > Program #4 > Program #5 > Program #6 > Program #7 > 
Program #8 >  Program #9 > Program #10 > Program #11 > Program #12 > Program #13 > Program #14 > 
Program #15 > Program #16  > Program #17 > Program #18 > Program #19 > Program #20 > Program #21 > 
Program #22 > Program #23 > Program #24 > Program #25 > Program #26 > 
Please enter the names of the different data files needed. 
No need to list the name of temporary files created by the programs.
Enter them one per line.  Press Enter twice when done.
Data file #1 > 
Enter the commands that the XGrid should run.  Enter each command on one line.
For each line, enter the name of the program followed by all the arguments. 
Do not use redirection or pipes in the command lines.
Enter an empty line to stop.
Command #1 > Command #2 > [xgridmac]


ls -ltr
total 89144
-rwxr-xr-x  1 thiebaut  staff   6334116 Jun  3 11:55 libqt-mt.3.dylib*
-rwxr-xr-x  1 thiebaut  staff     65372 Jun  3 11:55 libXext.6.dylib*
-rwxr-xr-x  1 thiebaut  staff     23012 Jun  3 11:55 libXdmcp.6.dylib*
-rwxr-xr-x  1 thiebaut  staff     13784 Jun  3 11:55 libXau.6.dylib*
-rwxr-xr-x  1 thiebaut  staff   1204428 Jun  3 11:55 libX11.6.dylib*
-rwxr-xr-x  1 thiebaut  staff     91468 Jun  3 11:55 filterwiki9*
-rwxr-xr-x  1 thiebaut  staff     83040 Jun  3 11:55 libz.1.dylib*
-rwxr-xr-x  1 thiebaut  staff    320932 Jun  3 11:55 libssl.0.9.8.dylib*
-rwxr-xr-x  1 thiebaut  staff   1404052 Jun  3 11:55 libmysqlclient.16.dylib*
-rw-r--r--  1 thiebaut  staff   1065668 Jun  3 11:55 libiconv.2.dylib
-rwxr-xr-x  1 thiebaut  staff    548736 Jun  3 11:55 libfreetype.6.dylib*
-rwxr-xr-x  1 thiebaut  staff    201836 Jun  3 11:55 libfontconfig.1.dylib*
-rwxr-xr-x  1 thiebaut  staff    146384 Jun  3 11:55 libexpat.1.dylib*
-rwxr-xr-x  1 thiebaut  staff   1435388 Jun  3 11:55 libcrypto.0.9.8.dylib*
-rwxr-xr-x  1 thiebaut  staff    343540 Jun  3 11:55 libXt.6.dylib*
-rwxr-xr-x  1 thiebaut  staff     40988 Jun  3 11:55 libXrender.1.dylib*
-rwxr-xr-x  1 thiebaut  staff     36720 Jun  3 11:55 libXrandr.2.dylib*
-rwxr-xr-x  1 thiebaut  staff     90916 Jun  3 11:55 libXmu.6.dylib*
-rwxr-xr-x  1 thiebaut  staff     13588 Jun  3 11:55 libXinerama.1.dylib*
-rwxr-xr-x  1 thiebaut  staff     76988 Jun  3 11:55 libXft.2.dylib*
-rwxr-xr-x  1 thiebaut  staff     23276 Jun  3 11:55 libXfixes.3.dylib*
-rwxr-xr-x  1 thiebaut  staff     39184 Jun  3 11:55 libXcursor.1.dylib*
-rwxr-xr-x  1 thiebaut  staff     37056 Jun  3 11:55 libSM.6.dylib*
-rwxr-xr-x  1 thiebaut  staff     91216 Jun  3 11:55 libICE.6.dylib*
-rwxr-xr-x  1 thiebaut  staff    292212 Jun  3 11:55 libGL.1.dylib*
-rw-r--r--  1 thiebaut  staff       449 Jun  3 11:55 filterwiki9.in
-rw-r--r--  1 thiebaut  staff  31555722 Jun  3 11:58 filterwiki9.batch
  • This create quite a large file to distribute: 31 MB. But a static version of the Qt executable would have been quite large as well...

Running the Qt3 Application on the XGrid

  • Simply submit the batch file created to the XGrid
    • Here we simply verify that the program gets loaded and checks the command line arguments, which are missing. It prints the syntax information back--good enough a test for us!
    • We rely on another Python script, getXGridOutput.py, to interact with the XGrid, get the Pids and collect the outputs.


time xgrid -job batch filterwiki9.batch | getXGridOutput.py 

Syntax: /var/xgrid/agent/tasks/jhv4aScs/working/filterwiki9 rawXmlWikiFileName -d dir [-1] [--onefile]

will take a raw wikipedia xml file and filter out the titles, ids,
categories, and concepts, and store them in a file with the same
name, but with a .filtered.xml extension
-d dir: the absolute path of the directory where to store the xml files
-D: put in debug mode
[-N nnnn]: max number of pages to process
[--onefile]: store the xml into one large file instead of several small ones
[-1] indicates that 1-word concepts should be kept


Total execution time: 0.000000 seconds




real	0m5.275s
user	0m1.261s
sys	0m0.715s











Retrieved from "http://dominiquethiebaut.com/dftwiki3/index.php?title=Running_Qt_C%2B%2B_Programs_on_the_XGrid&oldid=7390"