Java CoG Kit Workflow Guide: Proposed Extensions

From Java CoG Kit

Jump to: navigation, search

Contents

[edit]

Appendix

The following text is not yet approved and under discussion

[edit]

Performance

  • Total execution time: 245.094 s
  • Total elements executed: 2360204
  • Average element execution rate: -7893 elements/second
  • Total events: 4720538
  • Average event rate: 19260 events/second

We need a performance study.

[edit]

Proposed outline of the Workflow Guide

Please complete sections so we can discuss them

  • Introduction
    • History
    • Java CoG Kit
    • Karajan
  • Install
  • Hello World
  • Basic Language Constructs
    • variables
    • control flow
      • if
      • while
      • for
    • modularization
  • Advanced Langauge Constructs
    • ErrorHandeling
    • Channels
  • Paralelism and Grids
    • sequential
    • parallel
    • unsynchronized
    • scheduler
  • Native Code Integration
    • Java
    • Python
  • Shell script integration
    • Perl
  • Interfaces to other tools
    • Matlab
    • Mathematica


... i put in more here ...

  • Tools 
Proposed Enhancements: Advanced Concepts =


[edit]

Argument and return Values using the CHANNEL concept

[edit]

Element caching

[edit]

Error handling

In this example we demonstrate the use of the onError feature by starting a remote program on a machine but we have not issuesd a grid proxy init call. Based on the error handeling function, we start up the visual gridproxy init and continue once the error has been removed by creating a valid proxy.

The complete code for this example can be found ...

[edit]

Channels

[edit]

Named arguments with Channels

[edit]

How to create and use channels

[edit]

Proposed enhancements: Timers

{X} This feature is not yet available.

It is easy to measure the time it takes to perform a particular block of the workflow, you can use timers. that measure the time between two instances of time. The values of the timers can be printed individually by specifying the name and the action tag. All timers can be printed while using the All command. 

<project>
  <include file="cogkit.xml"/>
  <timer name="execution time">
      <execute executable="/bin/date" 
                    stdout="thedate" 
                    host="hot.mcs.anl.gov" 
                    provider="GT2"/>
  </timer>
  <echo message="Job completed. Transferring the output"/>
  <timer name="transfer time">
      <transfer srchost="hot.mcs.anl.gov" 
                     srcfile="thedate" 
                     desthost="localhost" 
                     provider="gridftp"/>
  </timer>
  <echo message="Transfer complete"/>
  <timer name="all" action="print"> 
</project>
[edit]

Proposed enhancements: Nested Timers

To allow the use of nested timers the </timer> tag can be augmented with an action parameter with the name parameter. If this parameter is missing, the last name will be used. 


<project>
  <include file="cogkit.xml"/>
  <timer name="total time">
      <timer name="execution time">
          <execute executable="/bin/date" 
                        stdout="thedate" 
                        host="hot.mcs.anl.gov" 
                        provider="GT2"/>
      </timer>
      <echo message="Job completed. Transferring the output"/>
      <timer name="transfer time">
          <transfer srchost="hot.mcs.anl.gov" 
                         srcfile="thedate" 
                         desthost="localhost" 
                         provider="gridftp"/>
      </timer>
      <echo message="Transfer complete"/>
  </timer>
  <timer name="all" action="print"> 
</project>

In order to increase readability of the code the end timer tag can be augmented with the name of the timer. 


<project>
  <include file="cogkit.xml"/>
  <timer name="total time">
      ... large peace of code
  </timer name="total time">
  <timer name="all" action="print"> 
</project>
[edit]

Proposed enhancements: Resuming timers

In order to use the same timer and add on its value you can use the same name between consecutive calls of the timer. 


<project>
  <include file="cogkit.xml"/>
  <timer name="work time">
      <execute executable="/bin/date" 
                    stdout="thedate" 
                    host="hot.mcs.anl.gov" 
                    provider="GT2"/>
  </timer>
  <echo message="Job completed. Transfering the output"/>
  <timer name="work time">
      <transfer srchost="hot.mcs.anl.gov" 
                     srcfile="thedate" 
                     desthost="localhost" 
                     provider="gridftp"/>
  </timer>
  <echo message="Transfer complete"/>
  <timer name="all" action="print"> 
</project>
[edit]

Proposed enhancements: Repetitive Timer usage

Timers can be used to conduct simple performance measurements of repetitive tasks. In this example we try to measure the time of n experiments and print the minimum, maximum, mean, and standard deviation of the result indicated by the analyze parameter. To only analyze a range of the measured times one can specify the range with the range parameter. 

<project>
  <include file="cogkit.xml"/>
  <set name="n" value="10">
  <for name="iteration" range="1, n">  
     <timer name="execution time" index="itteration">
          <execute executable="/bin/date" 
                        stdout="thedate" 
                        host="hot.mcs.anl.gov" 
                        provider="GT2"/>
      </timer>
      <timer name="execution time" action="print">
  </for>
  <timer name="execution time" action="analyze">
  <timer name="execution time" range="3,n" action="analyze">
</project>
[edit]

Proposed enhancements: Timer Events

As things can go wrong and a program may not terminate it is possible to use timer events to help with this. 

... define some future time
<set name="futuretime1" value="12:04:12 01/31/2010"/>
<onTimer when="12:04:12 01/31/2010"/>
   ... do this actions once the future time has arrived ...
</onTimer>
<onTimer when="{futuretime1}">
   ... do this actions once the future time has arrived ...
</onTimer>

Mike's note: the above would be a bad idea. This can already be done using parallelChoice: 

<set name="result">
  <race>
    <sequential>
      <stuff/>
      <string>Succeeded</string>
    </sequential>
    
    <sequential>
      <wait delay="{timeout}"/>
      <string>Timed-out</string>
    </sequential>
  </race>
</set>
<print message="Result: {result}"/>

The advantage of parallel choice is that only one of the racing elements will return values, not all. ParallelChoice also aborts all sub-threads that have not terminated, something that I find hard to implement with something like <onTimer/>.

Note: This can already be done with 

<unsynchronized>
  <wait delay=.../>
  <do your stuff/>
</unsynchronized>

Therefore, we are not sure if we should introduce it. It is convenient though. Maybe the approach to take would be to define an element that can doe it.


[edit]

Timers as they are implemented now

set(times 

 list(
   parallel(
     time(task1)
     time(task2)
     time(task3)
   )
 )

)

[edit]

Code Integration

[edit]

Proposed enhancements: Calling Java Methods

In this section we show how esy it is to call Java methods from within Karajan. We declare an element that calls the System properties of the JVM and prints them. We are making use of the Java properties. 

<project
   ... TBD ...
</project>

We are explaining this program now in more detail.

  • The first part of the program retrieves the properties.
  • The second part of the program prints out the properties.
  • The last example shows hoe to retrieve a named property. Here we print the operating system.

An additional example is provided in the CVS. This example consists of two parts:

[edit]

Proposed enhancements: Java Code

The Java CoG Kit workflow engine allows you to integrate arbitrary java code as part of the definition of a workflow. This is done with the help of the <java> tag. 


<project>
  <include file="java.xml"/>
  <java> 
    String hello = ("hello world from java");
    System.out.println(hello);
 </java>
</project>
[edit]

Proposed Enhancements: Python Code

The Java CoG Kit workflow engine allows to integrate arbitrary python code as part of the definition of a workflow. This is done with the help of the <java> tag. 

<project>
  <include file="python.xml"/>
  <python> 
     import sys
     hello = "hello world from python"
     print hello
  </python>
</project>
[edit]

Alternate syntax

The Java CoG Kit contains a workflow engine that is based on interpreting our XML based language. However as it can be cumbersome to read and write programs in that language we are providing you with a convenient way to write workflows in a way that reduces the writing of too many < and >.

The advantage is that the language becomes more readable and is more suitable for prototyping. For this to work, we have developed a simple preprocessor that is invoked before the actual workflow is interpreted.

channels.k: 

map(
  map:entry("name", "channels3")
  map:entry("test",
    element([]
      element(chantest, [channel(one), channel(two), ...]
        for(i, ..., "0:{i}")
        for(i, one, "1:{i}")
        for(i, two, "2:{i}")
     )
     list(
       chantest(
           "a"
           "b"
           channel:to(one, "a", "b")
           channel:to(two, "a", "b")
       )
     )
   )
  )
  map:entry("expected-result", list("0:a", "0:b", "1:a", "1:b", "2:a", "2:b"))
  map:entry("comparison", default-comparison)
)


element-list.k: 


map(
 map:entry("name", "element list")
 map:entry("test",
   element([]
     set(list1,
       elementList(
         "1"
         "2"
       )
     )
     set(list2,
       list(
         element([], "1")
         element([], "2")
       )
     )
     list(
       for(item, list1, executeElement(item))
       for(item, list2, executeElement(item))
     )
   )
 )
 map:entry("expected-result", list("1", "2", "1", "2"))
 map:entry("comparison", math-comparison)
)


factorial.k 

map(
 map:entry("name", "factorial")
 map:entry("test",
   element([]
     element(factorial, [n],
       if(
         ?(<=(n, 1))
         then(
           1
         )
         else(
           *(n, factorial(-(n, 1)))
         )
       )
     )
     factorial(10)
   )
 )
 map:entry("expected-result", 3628800)
 map:entry("comparison", math-comparison)
)


list.k 

map(
 map:entry("name", "list")
 map:entry("test",
   element([]
     list(
       list:size(
         list(1, 2)
         list(3, 4)
       )
       
       list:size(
         list:concat(
           list(1, 2)
           list(3, 4)
         )
       )
       
       set(mylist, list(1, 2, 3, 4))
       
       list:butFirst(mylist)
       list:butLast(mylist)
       list:first(mylist)
       list:last(mylist)
     )
   )
 )
 map:entry("expected-result", [2, 4, [2, 3, 4], [1, 2, 3], 1, 4])
 map:entry("comparison", math-comparison)
)


sum.k: 

map(
 map:entry("name", "sum")
 map:entry("test",
   element([]
     +(1, 2, 3)
   )
 )
 map:entry("expected-result", 6)
)
[edit]

Proposed enhancements: Globus Toolkit 4

[edit]

Globus Toolkit 4 Provider

We have developed providers for the Globus Toolkit 4 that allow you to use the Workflow in the same way as with GT2 and GT3. All you need to do is specify in your tasks that you submit to "GT4"

The following example submits a Job to a GT4 execution service. As you can see it is essentially the same example as introduced earlier 


<project>
  <include file="cogkit.xml"/>
  <execute executable="/bin/sleep" arguments="10" 
                host="hot.mcs.anl.gov" provider="GT4"/>
  <echo message="Job completed"/>
</project>
[edit]

Proposed enhancements: Globus Toolkit 4 pass through

{X} As the Globus Toolkit has changed significantly its specification language for submitting Jobs we have developed a pass through mechanism that allows us to pass a specification unchanged to a GT4 submission service. To do so a special include statement has to be added. The RSL can either be specified in a file or in a string. 


<project>
  <include file="cogkit.xml"/>
  <include file="cogkit-gt4-plugin.xml"/>
  <task:gt4 filename="myRSL.xml"/>
</project>



<project>
  <include file="cogkit.xml"/>
  <include file="cogkit-gt4-plugin.xml"/>
  <task:gt4>
     <string>
        ... put your rsl here ...
     </string>
  </task:gt4/>
</project>

Note: an example with slightly different syntax is available from CVS.

[edit]

Proposed enhancements: Debugging

Sometimes it is important to describe to others on which system you are running the Java CoG Kit. To simplify the process of debugging we have provided a simlple environment task that dumps a number of information to the standrad output. To use it you need to include the debug.xml file. The environment is printed through the invocation of the element debug:environment 

<project>
  <include file="cogkit.xml"/>
  <include file="debug.xml"/>
  <debug:environment/>
  <echo message="Printing out the environment completed."/>
</project>

Internally this program executes the shell script cog-env on *NIX and cygwin, and cog-env.bat under Windows. These programs can be adapted and enhanced. please let us know about programs that we should include.

At this time they call for *NIX 

/usr/bin/env
/usr/bin/hostname
/user/bin/uname -a
/grid-proxy-info
/usr/bin/which java
java -version
/usr/bin/which ant
ant -version

The script also prints the environment variables for 

$ANT_HOME
$JAVA_HOME
$COG_INSTALL_PATH
[edit]

Submission of Bugs

In case you find a bug we like that you first inspect the output of the cog-env script. If you think your environment is corectly set up but you can not identify the bug, we do recommend in karajan to run the debug environment statemnet priorto the submiision of the bug.


[edit]

Debug Properties

Debug properties such as controling teh debug level can be achieved through setting apropiate debug parametsers 


<project>
  <include file="cogkit.xml"/>
  <include file="debug.xml"/>
  <debug:parameter level="5"/>
  ... include your program to be debuged here ...
</project>

We distinguish the following debug levels

  • 1: TBD
  • 2: TBD
  • 3: TBD
  • 4: TBD
  • 5: TBD
[edit]

Proposed enhancements: Tools

[edit]

Proposed enhancements: Debugger

[edit]

Proposed enhancements: Simple Workflow Visualizer

[edit]

Proposed enhancements: Workflow Monitor Table

[edit]

Proposed enhancements: Translators

[edit]

Proposed enhancements: xml2karajan

[edit]

Proposed enhancements: karajan2xml

{X} we are still debating if we should include the plugin directly with the cogkit.xml file.

Something to this extent is already included. If you start Karajan with the '-intermediate' command line flag, whenever a .k file is translated, a .kml file is created with the XML translation of the .k file.

[edit]

Proposed Enhancement: Graphs

We propose to include the event based scheduling engine from earlliier versions within karajan.

We will add two special cases

  • Queues
  • hierarchical graphs

Queues are just a special case of hierarchical graphs, but as some sers only need a queue we like to develop an easy interface to to to make explicitly clear what we can do. For discussion is stil if wues should allow also the execution of graphs in a single queue entry. this makes sense especially when a user needs to cancel such a queueentry.

[edit]

Set 

<task:set:add ... label="uniqueid">
<task:set:start/>
<task:set:stop/>
<task:set:resume/>
[edit]

Queue

Event based paralelism as introduced in GECCO. 

<task:queue:add ... label="uniqueid">
<task:queue:start/>
<task:queue:stop/>
<task:queue:resume/>

Same as graph, but with dedicated name.

[edit]

Hierarchical graph 

<task:graph:add ... name="uniquegraphname" label="uniqueid">
<task:graph:dependency action="add" from="label1" to="label2">
<task:graph:dependency action="remove" from="label1" to="label2">
<task:graph:start/>
<task:graph:stop/>
<task:graph:resume/>

Adding graphs within graphs could be done with 

<task:graph:addGraph name="nameofgraph" 
                     node"labelofthenode" 
                     insert="nameofgraphtobeinserted"

graph(
  node(id=0
    element([], task:execute(...))
  )
  node(id=1
    element([], task:transfer(...))
  )
  edge(from=0, to=1)
)
[edit]

Issues

  • hierachical graphs
  • ingesting tasks at runtime
  • return list of active,inactive, completed, failed tasks
  • poping up the queue manipulation gui
  • targeting more than 10000 jobs
[edit]

Testing Karajan

The Karajan workflow system contains a simple testing framework that executes a number of tests that are part to test general karajan tests.

This testing couldalso be used to develop customised tests to conduct your own production tests in a Grid environment.

[edit]

What is avialable?

  • Readme - contains information on how to write your own test. The existing tests in the tests subdirectory should provide examples on how different tests can be written.
  • runtest.xml or runtest.k - is the main test script. It reads individual tests from the tests subdirectory, executes them, and checks the results. Two versions are provided for, one for the native syntax, and one for XML. Two versions of each test is also provided, and each runtest version will run the respective version of theindividual tests.
  • settings - contains various settings like global/test number of passes to be run, and settings for Grid resources to be used with remote tasks.
[edit]

Included Tests (so far)

channels1.xml channels1.k Does a basic channel test by channeling a return value from the default channel to another channel and back.
channels2.xml channels2.k Another channel test that nests two channels.
channels3.xml channels3.k Tests the definition of an element with channel arguments.
element-list.xml element-list.k Tests the equivalence of elementList(e1, e2, ...) with list(element(e1), element(e2), ...)
execution-scheduled-alloc.xml execution-scheduled-alloc.k Tests scheduled execution using allocateHost.
execution-scheduled.xml execution-scheduled.k Tests scheduled execution.
execution-unscheduled.xml execution-unscheduled.k Tests non-scheduled execution.
factorial.xml factorial.k Tests recursion using a recursive definition of factorial.
for.xml for.k Tests for iterators in various ways.
futures.xml futures.k Tests futures.
html.xml html.k A quasi-test of the html library.
list.xml list.k Tests the list-related elements.
parallel-element.xml parallel-element.k Tests the parallel element using a consumer/provider pair.
parallelfor.xml parallelfor.k Tests the functioning of the parallel iterator.
stress.xml stress.k Does a stress test using nested iterators, some parallel, some sequential.
sum.xml sum.k Adds a few numbers.
while.xml while.k Tests the while loop.
Retrieved from "http://wiki.cogkit.org/index.php/Java_CoG_Kit_Workflow_Guide:_Proposed_Extensions"
Personal tools
Collaboration and Jobs