CoG Shell
From Java CoG Kit
Contents |
CoG Shell
The purpose of the CoG Shell is to provide a very simple interface to the Grid. We will develop the code in SVN. Although it principly does not matter which language we chose, we have initially started to implement it in Python as it provides a simple command line interface.
Availability
The code is much in flux and changes are expected. You are welcome to participate in the development.
We will post outstanding tasks in this web page. We hope that some of you help making this a success.
svn co https://svn.sourceforge.net/svnroot/cogkit/trunk/cog-shell
Task A: Implementation of the CoG-Shell Selector
Given a hash table with keys and values, develop a program that loads the table from a YAML file, displays the contents in GUI table. Row in the table is selectable with a checkbox. For a look and feel of the application see the following sketch:
The YAML file will look similar to this:
- label: sunny type: computer value: 2 junk: 9 iwantyou: true - label:hot type: data value: 3.2 junk: 9 iwantyou: false - label: cold type: data value: 1.0 junk: 9 iwantyou: true
The program should be called via a class such as
table = TableSelector(filename, “iwantyou”, [“label”, “value”])
from Python. Please note that in case the attribute “iwantyou” is not defined as part of elements in the YAML file it will be added when you press the save button.
When choosing a GUI framework, make sure it is easily installable on other machines and works in Windows, MAC, and Linux. Do not use Tix as it seems not to install in MAC.
Suitable window frameworks are wxPython, Tkinter, …, SWT.
The program could also be written in a language other than python. In case you use Java, you should use SWING. In this case make sure you provide a suitable os.system call that invokes the program properly with the desired parameters. At this thime I do not think it is useful to write this application in Jython. Please use either plain Java or plain python.
Extensions
- Make the fields in the table editable.
Conditions
- In case you use code from others make sure you document where you got it from and which license the code is under. It may very well be that the component exists already and has been developed by others. For example a similar component exists in wxPython.
Task B: Implementing a generic form
Assume we have a generic hash table with key value pairs. We need to develop a generic attribute editor that allows us to change the values. The implementation is prefered in wxPython or Tkinter. Do not use Tix as we have not found out how to easily install it on MAC OSX. Assume the hash containes
label: sunny type: computer value: 2 junk: 9 iwantyou: true
Than a sketch for the GUI may look like
--------------------------------------
title
--------------------------------------
label: sunny________________
type: computer_____________
value: 2____________________
junk: 9____________________
iwantyou: true_________________
--------------------------------------
[Cancel] [Save]
--------------------------------------
An API may look like
editor = GUIAttributeEditor('title', ['label','type','value','junk','iwantyou'])
print editor.get() # returns the hash table after the save button was pressed.
Task C: Python Data Structures
We are in the need for the support of two data structures. The one is a Queue and the other one is a Graph. As we may be able to use operator overloading we want to design some convenient specifications to add and remove nodes
Queue
The current example demonstrates what we could do. Here the proposal:
q = cogkit.Queue() q += task # adds the task object to the queue q -= task # removes the task from the queue task = q.pop() # removes the last task from the queue task = q.top() # returns the topmost task without removing it q[i] # returns the ith task len(q) # returns the number of elements in the queue q.empty() # returns true if the queue is empty
It will be necessary to make this class thread save. You should start first with a non threadsave class.
Graph
I HAVE NOT SPEND ANY TIME ON THIS AS WE CAN USE karajan
The current example demonstrates what we could do. Here the proposal:
g = cogkit.Graph()
g += task # adds the task object to the graph
g -= task # removes the task from the graph and the apropiate edges
g[i] # returns the ith task of the graph
g.numberOfedges() # returns the number of nodes in the graph
g.numberOfNodes() # returns the number of nodes in the graph
g.empty() # returns true if the queue is empty
g[i].nodes() returns the nodes neighbored to i
g[i].edges() returns the edges form i
g[i].edges("in") returns the ingoing edges form i
g[i].edges("out") returns the outgoing edges form i
... see networkx on how to get dot exports. It will be necessary to make this class thread save. You should start first with a non threadsave class. One thing which needs to be explored is if networx is all we need?
Task C.1: Persistant Tasks
While using a simple queue we like to first be able to add the following commands to the shell (see the code realted to jobs.txt in the current shell and the ps command).
- add -task "task-a"
- del -task "task-a"
- load -task -- default filename is ~/.cog/jobs.txt
- load -task filename.txt
- save -task
- save -task filename.txt -- default filename is ~/.cog/jobs.txt
- submit "task-a" submists the job with the label job
- list "task-a" lists the details for that job
- list -task lists alltasks
- ps - lists the status of all tasks similar to ps, should also be available as ts for task status
- update - updates the status of all tasks
Note: we should probably use the word task instead of job
- will like to do the same with a graph.
add -dependency "from" "to" -- adds a dependency between the jobs
Please note that each job has a default label, this is created by a number and the prefix "job"
We need the simple queue within the next two weeks (before Oct 20).
Task D: Google Maps
We are in the need of displaying a number of IP adresses in a Googlemap.
Given is a file with the following format:
ipaddress | coment ...
Task is to create a googlemap form it. Please note that the map is not that frequently cahnging so it would be ok to place the contents in a file on a public html server.
Code Availability
- Implemented by George in the directrory toolsofcomputing_org. The ones mostly relevant to the above are DEQueue and DIGraph.
- toolsofcomputing_org/adt/DEQueue.py <-- Queue (doubly-ended)
- toolsofcomputing_org/adt/Digraph.py <-- Graph (Directed)
- toolsofcomputing_org/adt/Graph.py <-- Graph
![[ ]](/images/task.png "Regular task"){kind=small}
links to the svn should be added.
Outstanding Tasks
links to the svn should be added.
Identify if we should not use the same package prefix to make things easiser. We could also find another prefix than "cogkit"
Documentaion. As the book is out of print and the Tools of Computing web page is not maintained, we need another way of documenting waht we have.
We need some simple examples.
Evaluate how networX can help and if we should use that.
Task E : Distributed Ping
a start for this isin gui/graph.py
Goal is to develop a distributed ping that displayes the ping information in a variety of formats.
The first is a format that shows from a given host a table of ping results. (sorting should be possible)
The second is a format that shows pinds in a matrix with values (sorting should be possible.
The third is a dynamic graph that attempts to place the nodes in a particular distance from each other based on the falues returned.
The fourth is a contour or mountain plot of the ping values.
Please note that the return values from pings may be different based on the OS.
Task F : Distributed Traceroute
This task is to develop a graphical representation of traceroute.
There is a start of this program in traceroute2dot, but it does not parse all the vales returned from traceroute. The output would be visualized
a) with a graph created by graphviz b) with a tool that places the nodes in a particular distance. (See ping project)
Clarification to the Distributed Traceroute program:
Here the first task is to develop parsers for the traceroute entries and to transform them to graphviz dot format. Once that is done the vizualization of the data is trivial.
So lets assume you get
###################################################################### #Traceroute from login-co.ncsa.teragrid.org to login-w.ncsa.teragrid.org traceroute to login-w.ncsa.teragrid.org (141.142.38.2), 30 hops max, 40 byte packets 1 core-hpc-vlan783.ncsa.uiuc.edu (141.142.30.193) 0.330 ms 0.226 ms 0.189 ms 2 w-core-2-core-10.gw.ncsa.edu (141.142.0.150) 1.623 ms 1.998 ms 2.169 ms 3 w-b-w-core.gw.ncsa.edu (141.142.11.170) 2.277 ms 1.497 ms 2.278 ms 4 tunb.ncsa.uiuc.edu (141.142.38.2) 1.254 ms 1.466 ms 1.251 ms
Than this should be translated to
login-co.ncsa.teragrid.org -> core-hpc-vlan783.ncsa.uiuc.edu [len = 0.330] core-hpc-vlan783.ncsa.uiuc.edu->w-core-2-core-10.gw.ncsa.edu [len=1.623 ms] w-core-2-core-10.gw.ncsa.edu->w-b-w-core.gw.ncsa.edu [len=2.278] w-b-w-core.gw.ncsa.edu->tunb.ncsa.uiuc.edu [len=1.254 ms]
I should have some ability to select one of the three values through an api
This is essentaily how the current ping traceroute works that is in our svn at
svn co https://svn.sourceforge.net/projects/cogkit/trunk/cog-shell
However here are different traces that can not yet be parsed and the program should be appropiately modified.
The first case is easy as it requires jyst to remove (ttl=.+!) form the line the second one is a bit more involved, as you need to read up on traceroute and actually understand what it means. ... In any case its probably a good idea to read up on traceroute.
###################################################################### #Traceroute from tg-login1.lemieux.psc.teragrid.org to tg-login1.iu.teragrid.org traceroute to th1.uits.iupui.edu (149.165.230.231): 1-30 hops, 38 byte packets 1 burden.psc.teragrid.org (128.182.112.1) 0.0 ms 0.0 ms 0.0 ms 2 behemoth-burden-te0-1-0-2.psc.teragrid.org (128.182.64.20) 1.95 ms 0.977 ms 0.977 ms 3 192.5.175.54 (192.5.175.54) 12.6 ms (ttl=252!) 12.6 ms (ttl=252!) 12.6 ms (ttl=252!) 4 ipgrid-ge-010.gw.teragrid.org (192.5.175.59) 17.5 ms (ttl=251!) ipgrid-ge-110.gw.teragrid.org (192.5.175.61) 17.5 ms (ttl=251!) ipgrid-ge-010.gw.teragrid.org (192.5.175.59) 17.5 ms (ttl=251!) 5 if10-sbr.iu.teragrid.org (149.165.227.5) 21.4 ms 30.2 ms 17.5 ms 6 th1.uits.iupui.edu (149.165.230.231) 18.5 ms 18.5 ms 18.5 ms
here is another
###################################################################### #Traceroute from login-co.ncsa.teragrid.org to tg-login.sdsc.teragrid.org traceroute to tg-login.sdsc.teragrid.org (198.202.112.33), 30 hops max, 40 byte packets 1 core-hpc-vlan783.ncsa.uiuc.edu (141.142.30.193) 0.330 ms 0.200 ms 0.189 ms 2 198.17.196.25 0.283 ms 0.234 ms 0.236 ms 3 chi-so-130.gw.teragrid.org (192.5.175.20) 2.820 ms 2.791 ms chi-so-220.gw.teragrid.org (192.5.175.16) 2.840 ms 4 dnv-so-000.gw.teargrid.org (192.5.175.241) 27.012 ms 26.945 ms 26.947 m s 5 la-so-030.gw.teragrid.org (192.5.175.242) 56.363 ms 56.334 ms 56.317 ms 6 sdsc-so-000.gw.teragrid.org (192.5.175.35) 59.913 ms sdsc-so-030.gw.teragrid.org (192.5.175.33) 60.022 ms 60.011 ms 7 tg-login1.sdsc.teragrid.org (198.202.112.33) 60.152 ms 60.132 ms 60.126 ms
and another
Traceroute references that may be of use or not
some others:
- Cisco: http://www.cisco.com/en/US/products/sw/iosswrel/ps1831/products_tech_note09186a00800a6057.shtml
- Windows: www.techsupportalert.com/pdf/p1815.pdf
- http://help.expedient.com/general/ping_traceroute.shtml
Task G : Distributed showbf
In the cog shell we have a program that displays all showbf results from a number of resources. Task is to develop an more suitable datastructure taht allows us to sort them and develop a class that allows us to identify suitable resources for submission.
Obviously column 0 needs to be the hostname. Within a host, we should remove all equla lines.
cog> showbf Partition Tasks Nodes StartOffset Duration StartDate --------- ----- ----- ------------ ------------ -------------- tg-login.uc.teragrid.org: ALL 242 129 00:00:00 1:20:57:56 11:02:04_10/08 ALL 241 128 00:00:00 INFINITY 11:02:04_10/08 ALL 241 128 00:00:00 INFINITY 11:02:04_10/08 ALL 241 128 00:00:00 INFINITY 11:02:04_10/08 ALL 241 128 00:00:00 INFINITY 11:02:04_10/08 ALL 241 128 00:00:00 INFINITY 11:02:04_10/08 ALL 241 128 00:00:00 INFINITY 11:02:04_10/08 ALL 241 128 00:00:00 INFINITY 11:02:04_10/08 ALL 241 128 00:00:00 INFINITY 11:02:04_10/08 ALL 241 128 00:00:00 INFINITY 11:02:04_10/08 ALL 241 128 00:00:00 INFINITY 11:02:04_10/08 ALL 241 128 00:00:00 INFINITY 11:02:04_10/08 ALL 241 128 00:00:00 INFINITY 11:02:04_10/08 login-co.ncsa.teragrid.org: ALL 78 6 00:00:00 1:19:29 11:02:09_10/08 ALL 22 5 00:00:00 84:13:57:51 11:02:09_10/08 login-hg.ncsa.teragrid.org: login-cu.ncsa.teragrid.org: showbf: Command not found. tg-login.purdue.teragrid.org: ALL 673 344 00:00:00 19:57:37 12:02:23_10/08
A table may be a good way to represent the data rows may look like
Example tg-login will become a matrix such as
tg-login.uc.teragrid.org ALL 242 129 00:00:00 1:20:57:56 11:02:04_10/08 tg-login.uc.teragrid.org ALL 241 128 00:00:00 INFINITY 11:02:04_10/08 login-co.ncsa.teragrid.org ALL 78 6 00:00:00 1:19:29 11:02:09_10/08 login-co.ncsa.teragrid.org ALL 22 5 00:00:00 84:13:57:51 11:02:09_10/08 tg-login.purdue.teragrid.org ALL 673 344 00:00:00 19:57:37 12:02:23_10/08
One should be dealing with issues when there is no information returned or the command is not implemented on the remote machine.
one should have a function that sorts the table by a column such as time, nodes, ...
After this simple effort has been done you could move on to providing a search function that can return me search based on a specification.
Appendix: API
Abstractions: Access to Batch Globus Jobs
The access to Globus is conducted with the help of a mediator service. This is a host that has Globus installed on it. To access a globus submit command from python you will need to install keychain on your machine and make sure that authentication through ssh keys is enabled to the mediator host.
Once this is done you can use the following program to submit jobs.
SERVICES_MEDIATOR = """-
label: default
provider: ssh-globus
username: gregor
host: tg-login.uc.teragrid.org
"""
m = mediator()
m.load(stream=SERVICES_MEDIATOR)
m.label = 'default'
g = globus(m)
g.authenticate()
service='tg-grid.uc.teragrid.org/jobmanager-fork'
id = g.submit (service, '/bin/date')
status = g.status(id,service)
output = g.getOutput(id,service)
Although the example above uses at this time pre-WS-GRAM it is straight forward to change this to use WS-GRAM. However at this time WS-GRAM is not yet well documented on the TeraGrid, so we use the GT2 style submission commands.
