Java CoG Kit http://wiki.cogkit.org/wiki/Main_Page MediaWiki 1.15.5 first-letter Media Special Talk User User talk Java CoG Kit Java CoG Kit talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk V V TALK 1884 6597 2006-08-28T15:09:49Z Gregor 2 [[Special:Export/{{PAGENAME}}|XML]] Mike Hategan and Gregor von Laszewski (gregor@mcs.anl.gov) Administrative Notes: ;Editing: The maintenance of this document is rather simple. We do ''NOT'' recommend that you edit this document in a single edit. instead, we do require the edit as part of a subsection. This will prevent the chance that data will get lost. ; Versioning: It is extremely easy to adapt the Document to a new version. All that needs to be done is creating a new page with the appropriate version number and pasting and copying the contents of the current document into it. Than you just have to replace the view places where the actual version occurs. This is best done with an editor such as emacs that has easy query replace capabilities. = About this Document = This page contains the Reference manual. Although it contains all of the features, we do recommend that you start with the examples document at http://wiki.cogkit.org/index.php/Java_CoG_Kit_Workflow_Guide == Command Line == Usage: cog-workflow <options> file <arguments> == Options == ;(-execute | -e) <string>: Execute the script given as argument ;-showstats: Show various execution statistics at the end of the execution ;-debug: Enable debugging. This will enable a number of internal tests at the expense of speed. You should not use this since it is useful only for catching subtle consistency issues with the interpreter. ;-monitor: Shows a resource monitor ;-dumpstate: If specified, in case of a fatal error, the interpreter will dump the state in a file ;-intermediate: Saves intermediate XML code resulting from the translation of .k files ;(-help | -h): Display usage information == Experiemental Options == The following options have been added and are considered to be experiemental. They are still in the debugging stage. ;-debugger: Starts the internal graphical debugger ;-cache: Enables cache persistence == File == The file represents the name of the script to run, in either CoG Kit XML or CoG Kit K syntax. A CoG Kit karajan file structure is rather simple and similar to what you would are used to form other programming languages. A CoG Kit Karajan file starts with a series of imports, followed by a block of statements. It is important to note that we have two ways to express CoG Kit karajan syntax. Both are explained in more detail in the following sections. All Karajan files usually start with the import of the most generally used libraries in k syntax: {{newel|kernel|import}}("cogkit.xml") {{newel|sys|print}}("hello world") in xml syntax: <{{newel|kernel|import}} file="cogkit.xml"> <{{newel|sys|print}} message="Hello World"> == Arguments == Arguments to the script can be specified after the file name. They are distinct from the interpreter options, and are passed to the script as a constant, in the form of a list named "cmdline:arguments". More information about commandlines can be found [http://wiki.cogkit.org/index.php/Java_CoG_Kit_Karajan_Workflow_Reference_Manual_4.1.4#kernel:cmdline:arguments here] = The Karajan Language = Karajan supports two syntax modes: a syntax called k and an equivalent form called XML. There is no difference on the semantic level between the two forms. == The Karajan Syntax == The following conventions are used: (xy) is used to group x and y [x] indicates that x is optional x+ denotes at least one occurrence of x x* denotes zero or more occurrences of x x|y means either x or y 'x' is to be interpreted as the literal x &epsilon; represents the empty production === Elements === The Karajan semantics revolve around the notion of elements. An element is relatively similar to a function in that it has a name, can accept arguments, and may return values. The general syntax for an element is: element ::= identifier '(' [arguments] ')' Example: {{newel|sys|print}}(1) {{newel|sys|false}}() === Identifiers === An identifier can consist of alpha-numeric characters and certain symbols, but no whitespace. Symbols that cannot be used in an identifier are symbols that have other syntactic functions, such as brackets (all of them), commas, double quotes, and operators (’+’, ’-’, ’*’, ’/’, ’%’, ’^’, ’=’, ’<’, ’>’, ’&’, ’|’). Identifiers are case insensitive. identifier ::= (Letter | Digit | '!' | '@' | '#' | '$' | '%' | | '_' | ':' | ';' | <nowiki>'''</nowiki> | '.' | '?' | '\' | '`' | '~')+ Example: var, i, v123, @, a$, big_list, grid:task, file.list Identifiers cannot begin with a digit. === Arguments === The arguments can either be other elements or values, such as numeric values or strings. Elements can be separated by commas or the new line character (or both): arguments ::= argument [separator arguments]) | &epsilon; separator ::= ',' | Newline Example: {{newel|list|list}}({{newel|sys|true}}(), {{newel|sys|false}}()) {{newel|list|list}}( {{newel|sys|true}}() {{newel|sys|false}}() ) {{newel|list|list}}({{newel|sys|true}}(), {{newel|sys|false}}()) Arguments come in two flavors. Named arguments and unnamed arguments: argument ::= named_argument | unnamed_argument === Named Arguments === Named arguments provide a way of explicitly binding arguments to formal parameters: named_argument ::= identifier '=' unnamed_argument Example: {{newel|sys|print}}({{newel|sys|true}}(), nl = {{newel|sys|false}}()) === Unnamed Arguments === Unnamed arguments can be either immediate values, elements or expressions. Immediate values can be numeric literals, string literals, variables, or quoted lists: unnamed_argument ::= numeric_literal | string_literal | variable | quoted_list | | element | expression numeric_literal ::= ['+'|'-'] digit+ ['.' digit+] digit ::= '0'... '9' string_literal ::= '"' any_characters_but_double_quotes '"' variable ::= identifier Example: {{newel|list|list}}(1, 2.3, -4.56, +7.890, "A string", {{newel|list|list}}("Another string value in a nested list", "*2")) === Expressions === Expressions consist of unnamed arguments to which operators are applied. Parantheses can be used to override the default precedence of operators in expressions. expression ::= unnamed_argument operator unnamed_argument | | '(' expression ')' Examples of expressions: {{newel|sys|set}}(a, 1+2*3-4) {{newel|sys|set}}(b, {{newel|math|subtraction}}({{newel|math|sum}}(1, {{newel|math|product}}(2, 3)), 4)) {{newel|sys|print}}(a, " = ", b) <{{newel|sys|set}} name="a"> <{{newel|math|subtraction}}> <{{newel|math|sum}}> <{{newel|math|product}}> <{{newel|kernel|number}}>2</number> <{{newel|kernel|number}}>3</number> </product> </sum> <{{newel|kernel|number}}>4</number> </subtraction> </set> <{{newel|sys|print}} message="a = {a}"/> === Operators === The native Karajan syntax supports basic arithmetic and logic operators: operator ::= '*' | '/' | '%' | '+' | '-' | '<=' | '>=' | '<' | '>' | | '==' | '!=' | '&' | '|' The following lists enumerates operators in the order of precedence, starting with the highest precedence. While the XML syntax does not support the use of operators, each operator has an equivalent element which can be used in both syntaxes (shown in parentheses). * Multiplicative operators ** <code>*</code> ({{newel|math|product}}) Multiplication ** <code>/</code> ({{newel|math|quotient}}) Division ** <code>%</code> ({{newel|math|remainder}}) Remainder * Additive operators ** <code>+</code> ({{newel|math|sum}}) Addition ** <code>-</code> ({{newel|math|subtraction}}) Subtraction * High priority relational operators ** <code>&lt;=</code> ({{newel|math|lessOrEqual}}) Less or equal ** <code>&gt;=</code> ({{newel|math|greaterOrEqual}}) Greater or equal ** <code>&lt;</code> ({{newel|math|lessThan}}) Strictly less ** <code>&gt;</code> ({{newel|math|greaterThan}}) Strictly greater * Low priority relational operators ** <code>==</code> ({{newel|sys|equals}}) Equals ** <code>!=</code> (No equivalent, but {{newel|sys|not}}({{newel|sys|equals}}(...)) can be used) Does not equal * Multiplicative logic operators ** <code>&amp;</code> ({{newel|sys|and}}) Logical AND * Additive logic operators ** <code>|</code> ({{newel|sys|or}}) Logical OR === Quoted Lists === A quoted list is a special element that produces a list of identifiers. What is specific about a quoted list is that if its arguments are variables, the variables will not be evaluated. Instead their identifiers will be added to the list. Quoted lists are convenience syntax for expressing a list of formal arguments: quoted_list ::= '[' arguments ']' However, quoted lists are not limited to expressing list of arguments. They can also be used to express lists of values. The only thing to remember is that variable evaluation will not take place for immediate arguments of a quoted list. Example: {{newel|list|list}}("A quoted list follows", [a, b, c]) === Programs === A Karajan program is a list of arguments: program ::= arguments There exists an implicit root element that sits at the top of the element tree, and implements certain system functions. === Comments === And finally, Karajan uses C-style comments. Single-line comments begin with two forward slashes and end at the following new line character, while multi-line comments are delimited by ’/*’ and ’*/’: //This is a comment {{newel|sys|print}}("This is not a comment") /*This is also a comment */ == The XML Syntax == Karajan also supports XML as its syntax. In the XML syntax, each XML element corresponds to a Karajan element. Arguments can be expressed either through XML attributes or nested elements. === Particularities of Using XML === One of the particular aspects of using XML with Karajan is that when using XML attributes for arguments, it is impossible to make a syntactic distinction between a numeric value and its string representation. In general, Karajan will try to use the context to figure out which one is desired, but there are instances when it is impossible to do so. Therefore, when using the XML syntax, the following elements can be used for the purpose of differentiating between numeric and string values: number and string <{{newel|list|list}}> <{{newel|kernel|number}}>1</number> <{{newel|kernel|string}}>1</string> </list> The equivalent Karajan construct would be: {{newel|list|list}}(1, "1") Karajan can load and interpret arbitrary XML files, provided that definitions exist for the XML elements present in the file, but XML mixed content is not handled properly. The unfortunate aspect is that it is impossible to handle XML mixed content in a generic way. For example, it cannot be known whether whitespace between two XML elements is to be interpreted as content or not, without knowledge of the implementation of an element. Since Karajan is a dynamic language, the implementation of an element is not known statically, at the time the parsing takes place. Therefore, the following rule was adopted: An element will consider textual content content if and only if no nested elements exist. If nested elements exist, textual content will be ignored. If processed, textual content will be mapped as a string argument. A consequence of the above rule is that textual content and multiple arguments are mutually exclusive. Lastly, a well-formed XML document must always have a root element. While in the native Karajan syntax, the root element is implicit, in XML the project or karajan elements can be used as root elements. == Parameters and Return Values == An element can accept any number of arguments and can generate any number of return values, and that includes an infinite number of arguments and/or return values (at least in theory). === Parameters and Arguments === Arguments are divided into two major types: single value arguments and channels. As their name implies, single value arguments can have only one value. By contrast, channels can be used for any number of values. ==== Single Value Arguments ==== Single value arguments can be specified using the named argument form. For example, the print element has a message argument. Thus passing a string as the message argument to print element can be done in the following way: {{newel|sys|print}}(message = "Some string") Or in XML: <{{newel|sys|print}} message = "Some string"/> <!-- OR --> <{{newel|sys|print}}> <{{newel|kernel|argument}} name = "message" value = "Some string"/> </print> Single value arguments can be further divided into mandatory and optional arguments. ==== Channels ==== Channels can be used to pass multiple arguments to an element. Each channel has a name, except for the default channel. The default channel is similar to the notion of variable arguments in C. Passing arguments on the default channel is done implicitly when arguments are not passed as single value arguments: k: list(1, "value") xml: <list> <number>1</number> <string>value</string> </list> See Also: {{newel|list|list}}, {{newel|kernel|number}}, {{newel|kernel|string}} In the above case, 1 and ”value” are both passed to the list element on the default channel. Elements define whether they do receive arguments on a specific channel or not. One possibly interesting aspect is that an element that does not process arguments on a channel, will automatically return all values received on that channel. It is therefore possible to use named channels to return values to elements other than the immediate parent. Assuming that foo is an element that does not take any arguments on any channels, the following will produce the same result: {{newel|list|list}}(1, 2, 3) {{newel|list|list}}(foo(1, 2, 3)) <{{newel|list|list}}> <{{newel|kernel|number}}>1</number> <{{newel|kernel|number}}>2</number> <{{newel|kernel|number}}>3</number> </list> <{{newel|list|list}}> <foo> <{{newel|kernel|number}}>1</number> <{{newel|kernel|number}}>2</number> <{{newel|kernel|number}}>3</number> </foo> </list> Since foo does not process any arguments, all the arguments it receives on the default channel will be returned to the parent element. ==== Argument Mapping ==== It is not always convenient to use the named argument form to pass arguments to an element. Elements in Karajan will automatically map arguments received on the default channel to single value arguments. The mapping is done dynamically, in the order arguments are received. Suppose there is an element foo that takes three arguments, namely {{arg|one}}, {{arg|two}} and {{arg|three}}. The following would then be equivalent: foo(one = 1, two = 2, three = 3) foo(one = 1, two = 2, 3) foo(one = 1, 2, 3) foo(1, 2, 3) foo(1, 2, three = 3) ... <foo one = "1" two = "2" three = "3"/> <foo one = "1" two = "2"> <number>3</number> </foo> <foo one = "1"> <{{newel|kernel|number}}>2</number> <{{newel|kernel|number}}>3</number> </foo> <foo> <{{newel|kernel|number}}>1</number> <{{newel|kernel|number}}>2</number> <{{newel|kernel|number}}>3</number> </foo> <foo> <{{newel|kernel|number}}>1</number> <{{newel|kernel|number}}>2</number> <{{newel|kernel|argument}} name="three" value="3"/> </foo> ... ==== Optional Arguments ==== The unfortunate side-effect of using automatic mapping of default channel arguments to single value arguments is that elements that would accept both single value arguments and arguments on the default channel (variable arguments) cannot avoid mapping of variable arguments to certain single value arguments unless different semantics are introduced: optional arguments. Optional arguments do not need to be specified. However, if specified, the named form must always be used. An example is the print element, which has an optional argument named {{arg|nl}} . It can be set to <code>false</code> to indicate that no new-line character should be appended at the end of the message argument: {{newel|sys|print}}(message = "Message", nl = {{newel|sys|false}}()) or {{newel|sys|print}}("Message", nl = {{newel|sys|false}}()) The following however, is not valid: {{newel|sys|print}}("Message", {{newel|sys|false}}()) === Return values === Return values are a mirror image of the arguments concept. Whatever can be accepted as an argument by an element can also be returned by another. Thus, it is possible to define a single element that returns all arguments to any given element. The following example defines an element that returns both a message and the named form of the {{arg|nl}} argument, suitable for the {{newel|sys|print}} element: //The following defines an element foo() which takes no //arguments and returns "Message" and nl = false() {{newel|sys|element}}(foo, [] "Message", nl = {{newel|sys|false}}() ) {{newel|sys|print}}(foo()) <{{newel|sys|element}} name="foo" arguments=""> <{{newel|kernel|string}}>Message</string> <{{newel|kernel|argument}} name="nl"> <{{newel|sys|false}}/> </argument> </element> <{{newel|sys|print}}> <foo/> </print> === Argument Evaluation Order === There is no imposed order for evaluating arguments. The order is controlled by each element. Most elements, by default, evaluate their arguments in sequential order. However, it is very easy to override the default order by using elements that use a different execution order. For example, the {{newel|sys|parallel}} element evaluates all of its arguments in parallel, returning all the resulting values. Evaluating the arguments to an element in parallel then becomes as easy as surrounding them with a {{newel|sys|parallel}} element: {{newel|list|list}}( {{newel|sys|parallel}}( "Value1" "Value2" ) ) <{{newel|list|list}}> <{{newel|sys|parallel}}> <{{newel|kernel|string}}>Value1</string> <{{newel|kernel|string}}>Value2</string> </parallel> </list> Furthermore, the way in which an element processes the arguments is also left to each element. For example, an element can choose to start executing after the evaluation of all arguments has been completed, or process arguments as they arrive. In other words, and in the most general case, arguments are both generated and processed asynchronously. A concrete example is the {{newel|sys|print}} element, which simply returns the message argument on the {{chan|stdout}} channel. When Karajan starts execution, an implicit root element is created that receives arguments on the {{chan|stdout}} channel, and prints them to the console. Since the processing is done asynchronously, the appearance of {{newel|sys|print}} doing the actual work when executed is achieved. The advantage of such a mechanism is that, provided that an element does not produce any side-effects, its execution becomes equivalent to the totality of values returned (both single values, and channels). == Variables and Scope == We will not insult the reader’s intelligence by explaining what variables are. There is no explicit declaration of variables in Karajan. A variable is defined when it is assigned the first time. The scope of a variable extends to the element that it was defined in, and is pseudo-lexical. By pseudo-lexical it is meant that internally, the scoping is dynamic, but provisions are made to make it impossible to access variables outside the lexical scope. Therefore, Karajan does not support closures. On a lexical level, it is possible to read the value of a variable defined in a parent element, but setting the value of the same variable will create a new scope. In other words, Karajan uses deep access and shallow binding. The following example should make things clearer: ... {{newel|sys|set}}(v, 1) // ''v'' is '1' on stack frame ''n'' {{newel|list|list}}( // A new stack frame is created: ''n+1'' v // $v$ refers to the variable on frame ''n'' {{newel|sys|set}}(v, 2) // A new binding is made for ''v'' on frame ''n+1'' // the new binding shadows the one from frame ''n'' v // $v$ now refers to the binding on frame ''n+1'' ) // The returned list contains the values ''1'' and ''2'' {{newel|sys|print}}(v) //''v'' refers again to the binding on frame ''n'' //Therefore the printed value will be ''1'' ... ... <{{newel|sys|set}} name="v" value="1"/> <!-- $v$ is $1$ on stack frame $n$ --> <{{newel|sys|list}}> <!-- A new stack frame is created: $n+1$ --> <{{newel|kernel|variable}}>v</variable> <!-- $v$ refers to the variable on frame $n$ --> <{{newel|sys|set}} name="v" value="2"/> <!-- A new binding is made for $v$ on frame $n+1$ --> <!-- the new binding shadows the one from frame $n$ --> <{{newel|kernel|variable}}>v</variable> <!-- $v$ now refers to the binding on frame $n+1$ --> </list> <!-- The returned list contains the values $1$ and $2$ --> <{{newel|sys|print}}> <{{newel|kernel|variable}}>v</variable> <!-- $v$ refers again to the binding on frame $n$ --> </print> <!-- Therefore the printed value will be $1$ --> ... In the above example, it would be impossible for the definition of list to access variable v, since the body of the definition of list does not fall within the lexical scope of the definition of v. The reason for this kind of scoping is to reduce the ambiguity that could be introduced by not knowing the order in which child elements are executed. Please note that such ambiguity is not completely eliminated. The order in which the arguments to list are evaluated does matter, and can change the resulting list. However, the scope of the ambiguity is the same as the scope of the ambiguity in the order of evaluation of the elements. If set, list, and print are evaluated in sequence, no change in the way list evaluates its arguments can change the outcome of the execution of print(v). 1 === Global Variables === Global variables are provided for conveniently defining settings that have a global scope. Please note that in the future, global variables will be single-assignment, this approaching more the notion of constants. {{newel|sys|global}}(foo, "Foo") {{newel|sys|element}}(boo, [] {{newel|sys|print}}(foo) ) boo() === Variable Expansion === Karajan offers convenient variable expansion constructs. All pairs of curly brackets inside strings are replaced by the value of the variable with the name of the identifier inside the brackets. If no such variable exists, the element trying to access the string will fail. If the ’{’ literal is needed inside a string, it must be used twice. There is no need to escape the closing curly bracket, since it cannot be part of an identifier. If a closing bracket is part of a variable expansion expression, it will mark its end. If not, it will be interpreted as the closing curly bracket literal: {{newel|sys|set}}(a, 1) {{newel|sys|print}}("A is {a}") {{newel|sys|print}}("An opening curly bracket: {{") {{newel|sys|print}}("A closing curly bracket: }") <{{newel|sys|set}} name="a" value="1"/> <{{newel|sys|print}} message="A is {a}"/> <{{newel|sys|print}} message="An opening curly bracket: {{"/> <{{newel|sys|print}} message="A closing curly bracket: }"/> === Futures === Futures are a mechanism of binding a variable to the results of a future computation. Until the value of the computation to which the future is bound to, the future exists in an unbound state. Any attempt to use the value of an unbound future will cause the execution of the thread that tried to access the future to block until the future becomes bound. In Karajan there are two types of futures: ; single value futures : are used to hold a single value of a future computation. They are defined using the {{newel|sys|future}} element. ;future iterators : can be used to hold multiple values. However, not all the values need to be generated before the future iterator can be used. Iterating over a future iterator will cause the iteration to use as many values as are available, then block waiting for more values to be added to the future iterator. Future iterators are defined using {{newel|sys|futureIterator}}. = Modularisation = == Source Files == As mentioned in Section ??, Karajan understands two syntaxes: the native Karajan syntax and the XML syntax. The distinction between them is made using the file extension. A file with the “.k” extension will be parsed using the native parser, while a file with the “.xml” syntax will be parsed using an XML parser. {{ref|1}} == Libraries == Libraries are collections of elements grouped by the functionality they provide. A library is defined in a source file. Its functionality can be reused in other source files by using the {{newel|kernel|import}} (equivalent to {{newel|sys|include}}) element: {{newel|kernel|import}}("sys.k") It is possible to include XML libraries from native Karajan files. It is also possible to include native Karajan libraries from XML Karajan files. Consequently, the following are valid: {{newel|kernel|import}}("task.xml") <{{newel|kernel|import}} file="task.k"/> == Namespaces == Namespaces provide a way of distinguishing between elements with conflicting names in different libraries. Suppose a library “a” defines an element named {{fel|foo}}, and a library “b” also defines an element named {{fel|foo}}. Also, suppose that both libraries are included in a certain file. Namespaces make it possible to access both instances of the foo definition, without ambiguity, by prefixing the name with the namespace prefix in which the element was defined: {{fel|a:foo}} and {{fel|b:foo}}. Any reference to {{fel|foo}} without a prefix will result in an error. Nonetheless, if only one of the libraries is used, the use of {{fel|foo}} without a prefix will be allowed. Namespaces are defined using {{newel|kernel|namespace}}. = Kernel Library = The Karajan kernel contains a minimum set of elements that are required in order to get the rest of the system running. All kernel elements are automatically available in any program. == Kernel Constants == === kernel:true === ; {{newconstdef|kernel|true}}: Represents the '''true''' boolean value === kernel:false === ; {{newconstdef|kernel|false}}: Represents the '''false''' boolean value === kernel:cmdline:arguments === {{newconstdef|kernel|cmdline:arguments}} <blockquote> Holds a list with the command line arguments (if any). Arguments to the CoG Kit karajan script can be specified after the file name. They are distinct from the interpreter options, and are passed to the script as a constant, in the form of a list named "cmdline:arguments". </blockquote> === kernel:user.home === ; {{newconstdef|kernel|user.home}}: Contains the path to the user home directory === kernel:user.name === ; {{newconstdef|kernel|user.name}}: Contains the current user's name == Kernel Elements == === kernel:project === {{neweldef|kernel|project}}({{chan|stdout}}) {{Aliases|karajan}} The root element of a Karajan program. Accepts arguments on the {{chan|stdout}} channel and prints them immediately on the console. === kernel:import === {{neweldef|kernel|import}}({{arg|file}}, {{vargs}}) {{Aliases|include}} Executes the file specified by the {{arg|file}} argument. All arguments received on the default channel are considered to be element definitions (created with export), which import binds to the parent environment, such that after import completes execution, the definitions will be available for use. foo( import("file.k") ) //scope of foo() ends Import uses the library search path specified in <code>etc/karajan.properties</code>, which defaults to searching the current directory first, then the Java class path. If no file with the given name is found in the library search path, import will fail. === kernel:export === {{neweldef|kernel|export}}({{arg|name}}, {{arg|value}}) Returns the pair ({{arg|name}}, {{arg|value}}). The {{arg|value}} argument should be a lambda, which can be bound by import. export(foo, {{newel|sys|element}}([x], {{newel|sys|print}}(x))) It is also possible for {{fel|export}} to be used without arguments, but with a set of ''immediately enclosed'' definitions. In this case it would take all the definitions and export them. This makes it easier to have old code somewhat cleanly converted: export( {{newel|sys|element}}(x, [], {{newel|sys|print}}("x")) {{newel|sys|element}}(y, [], {{newel|sys|print}}("y")) ) === kernel:define === {{neweldef|kernel|define}}({{arg|name}}, {{arg|value}}) Binds a lambda, specified by the {{arg|value}} argument to the given {{arg|name}}. === kernel:namespace === {{neweldef|kernel|namespace}}({{arg|prefix}}) Allows the specification of a namespace prefix. Any elements defined in the scope of namespace will automatically have the prefix indicated by the {{arg|prefix}} argument, unless another namespace is nested. === kernel:elementdef === {{neweldef|kernel|elementdef}}({{arg|type}}, {{arg|classname}}) Used by the current implementation to map element names to Java implementation classes. === kernel:named === {{neweldef|kernel|named}}({{arg|value}}, {{oarg|name}}) Used internally by the named argument form, but can be used by the user equally well. The following are equivalent: {{newel|sys|print}}("Test", nl = {{newel|sys|false}}()) {{newel|sys|print}}("Test", kernel:named(name = nl, {{newel|sys|false}}()) However, the fact that the {{oarg|name}} argument is optional makes it impossible to completely avoid the named form. If the {{oarg|name}} argument is not present, it simply returns the value of the {{arg|value}} argument on the default channel. === kernel:number === {{neweldef|kernel|number}}({{arg|value}}) Can be used with the XML syntax to represent a number. There is no distinction between integral numbers and floating point numbers in Karajan. === kernel:string === {{neweldef|kernel|string}}({{arg|value}}) Can be used with the XML syntax to represent a string value. === kernel:variable === {{neweldef|kernel|variable}}({{arg|name}}) Used to represent the value of a variable. Example: <{{newel|sys|set}} name="n" value="5"/> <{{newel|sys|list}}> <{{newel|kernel|number}}>10</number> <{{newel|kernel|string}}>10</string> <variable>n</variable> </list> will return the list [10, “10”, 5]. === kernel:quotedlist === {{neweldef|kernel|quotedlist}}({{vargs}}) Used internally to represent a quoted list. The following two lines of code will produce the same result: [a, b, c] quotedlist(a, b, c) === kernel:cache === {{neweldef|kernel|cache}}() Caches the evaluation of the arguments. Upon subsequent executions of cache, the arguments will not be re-evaluated. Instead, the cached values will be returned. Cache does not make any checks for the invariance of the evaluation of the arguments. = System Library = '''Files:''' sys.k, sys.xml The system library contains general purpose elements that help implement the most common tasks in Karajan. == Flow Control Elements == === sys:sequential === {{neweldef|sys|sequential}}() Executes all arguments in sequence. === sys:parallel === {{neweldef|sys|parallel}}() Executes all arguments in parallel. === sys:unsynchronized === {{neweldef|sys|unsynchronized}}() Asynchronously executes all arguments in sequence. Does not return any value. If return values from asynchronous computations is required, use either {{newel|sys|future}} or {{newel|sys|futureIterator}} === sys:choice === {{neweldef|sys|choice}}() Executes arguments in succession. Terminates when one of the arguments completes successfully. If an argument fails, the next argument will have access to the following variables: ; element : Contains a reference to the element that caused the initial failure. ; error : A textual message detailing the error that occurred ; trace : A textual representation of the Karajan stack trace. ; exception : Available if a Java exception caused the failure. If no argument completes successfully choice will fail with the last failure encountered. {{fel|Choice}} exhibits a transactional behavior when it comes to return values. All single values and all channels are buffered until one argument completes successfully. If that happens then {{fel|choice}} returns the buffered values. If an argument fails, all the buffered values produced by that argument are discarded. See also: {{newel|sys|catch}} === sys:catch === {{neweldef|sys|catch}}({{arg|match}}) {{fel|Catch}} will match the error variable against the regular expression in {{arg|match}}. If successful, it will execute the rest of its arguments, otherwise it will fail with the last failure encountered. {{fel|Catch}} can be used with {{newel|sys|choice}} to selectively handle specific failures: {{newel|sys|choice}}( ... catch(".*File not found.*" {{newel|sys|print}}("File not found") ) catch(".*Connection refused.*" {{newel|sys|print}}("Connection refused") ) ) === sys:guard === {{neweldef|sys|guard}}() {{fel|Guard}} expects two sub-elements. It will execute the first, then the second, even if the first one fails. In other words, the second sub-element will always be executed. If the first element failed, after executing the second element, guard will also fail. If the second element fails, {{fel|guard}} will fail with the same error, regardless of whether the first element failed or not. {{fel|Guard}} can be used to implement clean-up actions, in a fashion similar to try/finally from C++, Java or Python: {{newel|sys|set}}(myhost, "sunny.mcs.anl.gov") {{newel|task|transfer}}(srcfile="a", desthost=myhost) guard( {{newel|sys|sequential}}( {{newel|task|execute}}(executable="/usr/bin/hammer", arguments="a", host=myhost, provider="gt2") ) {{newel|sys|sequential}}( //always clean up {{newel|task|file:remove}}(name="a", host=myhost, provider="gridftp") ) ) === sys:race === {{neweldef|sys|race}}() {{Aliases|parallelChoice}} Can be used to race a number of arguments. {{fel|Race}} will execute all its arguments in parallel, buffer their return values, and wait for the first one that completes. It will then return all the values that the winner generated. If an any argument fails before any other argument completes, then {{fel|race}} will fail. {{fel|Race}} is similar in behavior to the ''discriminator'' workflow pattern. === sys:for === {{neweldef|sys|for}}({{arg|name}}, {{arg|in}}) Can be used to iterate sequentially across a range of values. The {{arg|name}} argument is an identifier that indicates the name of the variable that will be set to the successive values of the in argument, which Karajan will try to convert to an iterator before beginning the iteration process. After evaluating {{arg|name}} and {{arg|in}}, {{fel|for}} will proceed and evaluate the rest of the arguments repeatedly, while setting the variable indicated by the {{arg|name}} argument to each value produced by the iterator ({{arg|in}}). Example: {{newel|sys|equals}}( {{newel|list|list}}( for(i, {{newel|list|range}}(1, 5), i) ) {{newel|list|list}}(1, 2, 3, 4, 5) ) will return {{newconst|kernel|true}} === sys:parallelFor === {{neweldef|sys|parallelFor}}({{arg|name}}, {{arg|in}}) Will behave in a similar way to {{newel|sys|for}} with the exception that iterations will occur in parallel. Each iteration will occur in a separate scope. Therefore variables set in one of the iterations will not be visible in the others (which is also the case with {{newel|sys|for}}). Each scope will have the variable indicated by the {{arg|name}} argument set to one of the values obtained from the {{arg|in}} argument. === sys:while === {{neweldef|sys|while}}({{chan|condition}}) Will repeatedly execute its arguments in sequence until a value of {{newconst|kernel|false}} is received on the {{chan|condition}} channel. A convenience element that returns a boolean argument on the {{chan|condition}} channel is {{newel|sys|condition}} (or {{newel|sys|?}}). While will check for a condition every time an argument completes. It is therefore possible to exit the loop after the termination of any of the arguments. Example: {{newel|sys|list}}( while( 1, 2, 3, {{newel|sys|?}}({{newconst|kernel|false}}) ) ) {{newel|sys|list}}( while( 1, {{newel|sys|?}}({{newconst|kernel|false}}), 2, 3 ) ) {{newel|sys|list}}( while( {{newel|sys|?}}({{newconst|kernel|false}}), 1, 2, 3 ) ) {{newel|sys|list}}( while( {{newel|sys|sequential}}( {{newel|sys|?}}({{newconst|kernel|false}}) 0 /* zero will make it to the list * because while will only do the check * after {{newel|sys|sequential}}() completes */ ) 1, 2, 3 ) ) will return the following lists: [1,2,3] [1] [] [0] Or, in XML: <{{newel|sys|list}}> <while> <{{newel|kernel|number}}>1</number> <{{newel|kernel|number}}>2</number> <{{newel|kernel|number}}>3</number> <{{newel|sys|condition}}> <{{newel|sys|false}}/> </condition> </while> </list> <{{newel|sys|list}}> <while> <{{newel|kernel|number}}>1</number> <{{newel|sys|condition}}> <{{newel|sys|false}}/> </condition> <{{newel|kernel|number}}>2</number> <{{newel|kernel|number}}>3</number> </while> </list> <{{newel|sys|list}}> <while> <{{newel|sys|condition}}> <{{newel|sys|false}}/> </condition> <{{newel|kernel|number}}>1</number> <{{newel|kernel|number}}>2</number> <{{newel|kernel|number}}>3</number> </while> </list> <{{newel|sys|list}}> <while> <{{newel|sys|sequential}}> <{{newel|sys|condition}}> <{{newel|sys|false}}/> </condition> <{{newel|kernel|number}}>0</number> <!-- zero will make it to the list because while will only do the check after sequential() completes --> </sequential> <{{newel|kernel|number}}>1</number> <{{newel|kernel|number}}>2</number> <{{newel|kernel|number}}>3</number> </while> </list> === sys:condition === {{neweldef|sys|condition}}({{arg|value}}) {{Aliases|?}} Evaluates the {{arg|value}} argument and returns its value on the {{chan|condition}} channel. === sys:break === {{neweldef|sys|break}}() Can be used to break out of a {{newel|sys|while}} loop. By contrast with using the {{chan|condition}} channel, break will immediately exit the loop, no matter how deep the nesting level. It does that by generating a failure, which is intercepted by the enclosing {{newel|sys|while}}. === sys:continue === {{neweldef|sys|continue}}() Can be used to skip the evaluation of the remaining arguments in an iteration in a {{newel|sys|while}} loop and jump to the next iteration. Similar to {{newel|sys|break}}, continue achieves its purpose by generating a failure which is intercepted by {{newel|sys|while}}. === sys:if === {{neweldef|sys|if}}() Executes its elements using the following scheme: # start with k = 0 # evaluate element 2 * k; # if element 2 * k is the last element, then return its return values and complete # if no value is element by argument 2 * k, fail # if value returned by element 2*k is {{newconst|kernel|true}} then evaluates element 2*k+1 returning its return values, and completes # if value returned by element 2 * k is {{newconst|kernel|false}} then continue with k = k + 1 Informally: if( <condition> <then> [<condition2> <then2> [<condition3> <then3> ...]] [<else>] ) <condition>, <then>, and <else> can be any elements. However, each <condition>, <then>, and <else> must be only one element (possibly with multiple child elements). For convenience and clarity {{newel|sys|then}} and {{newel|sys|else}} can be used. === sys:then === {{neweldef|sys|then}}() {{Aliases|else}} {{fel|Then}} and {{fel|else}} are the same as {{newel|sys|sequential}}, but can be used to make constructs using {{newel|sys|if}} more intuitive: {{newel|sys|if}}( a {{newel|sys|2===}} 1 then({{newel|sys|print}}("a is 1")) a {{newel|sys|2===}} 2 then({{newel|sys|print}}("a is 2")) else({{newel|sys|print}}("a is not 1 nor 2")) ) <{{newel|sys|if}}> <{{newel|sys|equals}}> <{{newel|kernel|number}}>1</number> <{{newel|kernel|variable}}>a</variable> </equals> <then> <{{newel|sys|print}} message = "a is 1"/> </then> <equals> <{{newel|kernel|number}}>2</number> <{{newel|kernel|variable}}>a</variable> </equals> <then> <{{newel|sys|print}} message = "a is 2"/> </then> <else> <{{newel|sys|print}} message = "a is not 1 nor 2"/> </else> </if> === sys:exclusive === {{neweldef|sys|exclusive}}() Defines a mutual exclusion block. It guarantees that at any give time, within a given execution, only one instance of this (lexically) {{fel|exclusive}} element is executing. == Elements Dealing with Variables and Arguments == === sys:set === {{neweldef|sys|set}}({{arg|names}}, {{vargs}}) Sets a variable or more to a value (or more). {{fel|Set}} tries to interpret the first argument as an identifier or a list of identifiers. If the first argument is an identifier, it is treated as being a list with one identifier. {{fel|Set}} expects the number of arguments on the default channel to be the same as the number of identifiers. It is important that a quoted list be used to specify the list of identifiers in order to avoid evaluating the variables that the identifiers represent: set(a, 1) set([a], 1) set([a, b, c], 1, 2, 3) Differences in XML: The XML variant of the {{fel|set}} element uses a slightly different set of arguments. If a single variable is assigned, the {{arg|name}} argument can be used. If multiple variables are assigned, the {{arg|names}} argument must be used. As opposed to the native syntax, the {{arg|names}} argument can be a string of comma separated identifiers which will be tokenized by {{fel|set}} <set name="a" value="1"/> <set names="a, b, c"> <{{newel|kernel|number}}>1</number> <{{newel|kernel|number}}>2</number> <{{newel|kernel|number}}>3</number> </set> === sys:default === {{neweldef|sys|default}}({{arg|name}}, {{arg|value}}) Assigns a value to a variable if no binding of that variable can be accessed within the current scope. If an accessible variable with the indicated name is already defined, then the assignment does not take place: default(a, 1) //a is assigned the value 1 {{newel|sys|set}}(b, 2) default(b, 3) //b is not assigned the value of 3 because //b is already accessible within the current scope === sys:maybe === {{neweldef|sys|maybe}}() Evaluates its arguments. If the evaluation completes successfully, it returns all the arguments. If the evaluation fails at any point, {{fel|maybe}} completes without returning anything. In particular, {{fel|maybe}} can be used in extending existing elements with optional arguments: {{newel|sys|element}}(one, [a, b, {{newel|sys|optional}}(c, d)] {{newel|sys|print}}("a = {a}", "b = {b}", maybe("c = {c}"), maybe("d = {d}")) ) {{newel|sys|element}}(two, [a, b, {{newel|sys|optional}}(c, d)] one(a = a, b = b, maybe(c = c), maybe(d = d)) ) === sys:global === {{neweldef|sys|global}}({{arg|name}}, {{arg|value}}) Sets a global variable. A global variable is a variable that has a global scope, and thus can be accessed from anywhere within the program. While the use of global variables is discouraged, it can prove useful to create constant-like definitions. Differences in XML: The XML variant of {{fel|global}} uses the same arguments as the XML variant of the {{newel|sys|set}} element. sys:...() {{Aliases|vargs}} Can be used inside an element definition to return all arguments received on the default channel. Differences in XML: In the XML syntax the '''vargs''' alias must be used, because “...” is not a valid XML element name. === channel:to === {{neweldef|channel|to}}({{arg|name}}, {{vargs}}) Returns all arguments received on the default channel on the specified channel. === channel:from === {{neweldef|channel|from}}({{arg|name}}, &lt;varies&gt;) Returns all arguments received on the specified channel on the default channel. === channel:close === {{neweldef|channel|close}}({{arg|name}}) Closes a channel. All iterations that are active on that channel and waiting for values will complete. === channel:fork === {{neweldef|channel|fork}}({{arg|name}}, {{arg|count}}) Splits a channel into a number of identical channels and returns these channels. All values written to the initial channel will be available in all the forked channels. Reading from the original channel will cause inconsistencies and should not be done. When the original channel is closed, all the forked channels will also be closed. === sys:isDefined === {{neweldef|sys|isDefined}}({{arg|name}}) Determines whether a variable is accessible within the current scope. Returns {{newconst|kernel|true}} if it is; {{newconst|kernel|false}} otherwise. === sys:quoted === {{neweldef|sys|quoted}}({{arg|name}}) Returns an identifier without evaluating the variable it might point to. === sys:discard === {{neweldef|sys|discard}}({{vargs}}) Sometimes only the side-effect of an element is needed, while ignoring the return values of the element. {{fel|Discard}} will evaluate its arguments, but avoid returning anything on the default channel. === sys:future === {{neweldef|sys|future}}({{vargs}}) Evaluates the arguments asynchronously. Returns a future representing the first return value generated by the arguments. All other arguments received on the default channel are ignored. === sys:futureIterator === {{neweldef|sys|futureIterator}}({{vargs}}) Evaluates its arguments asynchronously. Returns a future iterator representing all values received on the default channel. The particular aspect of a future iterator is that it can only be iterated over once. Every time a value is used from a future iterator, that value is removed. If access to the iterator values is needed more than once, a list can safely be created with the values. However, the process of creating the list will force synchronization with the thread that produced the values since the iterator is only closed when that thread completes. === sys:each === {{neweldef|sys|each}}({{arg|items}}) Returns all elements in {{arg|items}} as separate values. It is roughly equivalent to the following: for(i, items, i) == Element Definition Elements == === sys:element === {{neweldef|sys|element}}({{arg|name}}, {{arg|arguments}}) Allows the definition of an element. Evaluates the {{arg|name}} and {{arg|arguments}} arguments. It expects the {{arg|name}} argument to be an identifier, and {{arg|arguments}} to be a list of identifiers. The scope of the definition is the same as the scope of a variable that could be defined instead of the element. The arguments are a list of mandatory arguments. Optional arguments can also be specified using the {{newel|sys|optional}} element. If the element accepts arguments on the default channel, the {{vargs}} identifier can be used in the argument list. Other channels can be specified using the {{newel|sys|channel}} element. The rest of the arguments are not evaluated when the definition takes place, but will be evaluated whenever the element is invoked. The following example defines an element foo, which takes no arguments, and prints ’foo’ on the console: element(foo, [] {{newel|sys|print}}("foo") ) foo() In the following example, foo takes two arguments and prints them both on the screen: element(foo, [one, two] {{newel|sys|print}}(one) {{newel|sys|print}}(two) ) foo(1, 2) Arguments on the default channel can be accessed using the {{vargs}} identifier: element(foo, [one, ...] {{newel|sys|print}}(one) {{newel|sys|for}}(i, ... {{newel|sys|print}}(i) ) ) foo("one", 1, 2, 3, 4) Other channels can be used in a similar way: element(foo, [one, ..., {{newel|sys|channel}}(channelOne)] {{newel|sys|print}}(one) {{newel|sys|for}}(i, ..., {{newel|sys|print}}(i)) {{newel|sys|for}}(i, channelOne, {{newel|sys|print}}(i)) ) foo("one", 1, 2, 3, 4, {{newel|channel|to}}(channelOne, 5, 6, 7, 8)) Optional arguments can be assigned a default value using {{newel|sys|default}}: element(foo, [one, {{newel|sys|optional}}(two)] {{newel|sys|default}}(two, 2) {{newel|sys|print}}(one) {{newel|sys|print}}(two) ) foo("one") foo("one", two = "two") {{fel|Element}} can also be used to define anonymous elements. If the first argument evaluates to a list of identifiers instead of an identifier, {{fel|element}} considers that an anonymous element was instead desired, defines the element, and returns the definition, which can later be used through {{newel|sys|executeElement}}: {{newel|sys|set}}(foo, element([] {{newel|sys|print}}("Foo") ) ) {{newel|sys|executeElement}}(foo) Each definition of an element keeps a reference to the environment that was used at the time of the definition. When evaluated, elements in the body of the definition will be resolved by first searching in the local scope (eventually for elements defined by the execution of the body of this element) and, if not found, in the environment that was used at the time of the definition. In the following example, the result will be the printing of the string "a": element(foo, [] element(a, [], {{newel|sys|print}}("a")) //return an anonymous element element([] a() //this is the a() defined above ) ) {{newel|sys|set}}(b, foo()) element(a, [], {{newel|sys|print}}("b")) {{newel|sys|executeElement}}(b) This behaviour is particularly important when {{newel|kernel|import}} and {{newel|sys|export}} are used. Differences in XML: The arguments list is a string of comma separated identifiers. The XML version of element uses a different set of arguments. If an element accepts arguments on the default channel, the <code>vargs="true"</code> attribute must be used. The arguments received on the default channel will then be available in the body of the definition through the {{arg|vargs}} identifier. Optional arguments are indicated using the {{arg|optargs}} attribute. The value must be a comma separated list of identifiers. In a similar way, the channels are specified using a comma separated list of identifiers and the {{arg|channels}} attribute. <element name = "foo" arguments = "one" vargs = "true" channels = "channelOne"> <{{newel|sys|print}} message = "{one}"/> <{{newel|sys|for}} name = "i" in = "{vargs}"> <{{newel|sys|print}} message = "{i}"/> </for> <{{newel|sys|for}} name = "i" in = "{channelOne}"> <{{newel|sys|print}} message = "{i}"/> </for> </element> <foo one = "one"> <{{newel|kernel|number}}>1</number> <{{newel|kernel|number}}>2</number> <{{newel|kernel|number}}>3</number> <{{newel|kernel|number}}>4</number> <{{newel|channel|to}} name="channelOne"> <{{newel|kernel|number}}>5</number> <{{newel|kernel|number}}>6</number> <{{newel|kernel|number}}>7</number> <{{newel|kernel|number}}>8</number> </to> </foo> <element name ="foo" arguments = "one" optargs = "two"> <{{newel|sys|default}} name = "two" value = "2"/> <{{newel|sys|print}} message = "{one}"/> <{{newel|sys|print}} message = "{two}"/> </element> <foo one = "one"/> <foo one = "one" two = "two"/> === sys:parallelElement === {{neweldef|sys|parallelElement}}({{arg|name}}, {{arg|arguments}}) Like {{newel|sys|element}}, {{fel|parallelElement}} also defines an element. However, elements defined using {{fel|parallelElement}} will evaluate their arguments in parallel with their bodies. All single value arguments will automatically be futures, and all channels will automatically be future iterators. {{fel|ParallelElement}} can be used to define elements that process their arguments asynchronously. parallelElement(consumer, [...] for(i, ..., print("Received {i}") ) {{newel|sys|element}}(producer, [] {{newel|sys|for}}(i, {{newel|sys|range}}(0, 100) i {{newel|sys|print}}("Sent {i}") {{newel|sys|wait}}(delay = 100) ) ) consumer(producer()) Differences in XML: The differences for the XML syntax from {{newel|sys|element}} also apply to {{fel|parallelElement}} <parallelElement name = "consumer" vargs = "true"> <{{newel|sys|for}} name = "i" in = "{vargs}"> <{{newel|sys|print}} message = "Received {i}"/> </for> </parallelElement> <{{newel|sys|element}} name = "producer"> <{{newel|sys|for}} name = "i"> <{{newel|sys|range}} from = "0" to = "100"/> <{{newel|kernel|variable}}>i</variable> <{{newel|sys|print}} message = "Sent {i}"/> <{{newel|sys|wait}} delay = "100"/> </for> </element> <consumer> <producer/> </consumer> === sys:channel === {{neweldef|sys|channel}}({{vargs}}) Used to specify channel arguments for an element. Quotes all arguments, such that identifiers are not evaluated. Returns values that can be interpreted by {{newel|sys|element}} and {{newel|sys|parallelElement}} as representing channels. === sys:optional === {{neweldef|sys|optional}}({{vargs}}) Allows the specification of optional arguments for element definitions. Quotes all arguments and returns values that can be interpreted by {{newel|sys|element}} and {{newel|sys|parallelElement}} as representing optional arguments. === sys:self === {{neweldef|sys|self}}() {{fel|Self}} can be used to build recursive anonymous elements: {{newel|sys|set}}(f {{newel|sys|element}}([x] {{newel|sys|if}}(x == 0 1 x*self(x - 1)) ) ) {{newel|sys|print}}({{newel|sys|executeElement}}(f, 6)) == Service Interaction Elements == === sys:remote === {{neweldef|sys|remote}}({{arg|host}}) Uses the [[Karajan:Service]] that runs on the specified {{arg|host}} to evaluate the rest of the arguments. The element supports forwarding of the variable environment to the remote service{{ref|1}}, and returning values from the service. Therefore the following code will work as expected: {{newel|sys|set}}(a, 1) {{newel|sys|set}}(b remote("https://somehost:1984", a+1) ) {{newel|sys|print}}(b) //will print 2 A consequence is that if {{newel|sys|print}} is used remotely, the actual output is going to be automatically and transparently forwarded (since it is merely a return value on a particular channel) to the execution instance that initiated the remote execution. It is also possible to use this feature recursively, such that a remote code snippet in turn delegates part of the execution to other services. Please note that the code inside remote might reference entities that are sensitive to the actual location of the execution. In particular, the meaning of “localhost” if used with execute or transfer will not refer to the machine where the execution originated. Element resolution is performed separately when a script is submitted to a service. In other words, {{newel|kernel|import}}-ed files on the client side will be re-imported on the service side, if and only if they are system libraries. If imports refer to user-defined libraries not part of the system libraries, the code defining the libraries will be forwarded to the service, thus enabling the use of client-side user-defined elements on the remote side. Similarly, in-line user defined elements are also available for use on the remote side: {{newel|sys|element}}(foo, [], {{newel|sys|print}}("Foo")) remote("https://somehost:1984" foo() ) == List Manipulation Elements == === list:list === {{neweldef|list|list}}({{oarg|items}}, {{vargs}}) Constructs a list from values received on the default channel. Alternatively the {{oarg|items}} argument can be used to specify a a string with comma separated list of items. This may be particularly convenient with the XML syntax. === list:append === {{neweldef|list|append}}({{arg|list}}, {{oarg|items}}, {{vargs}}) Appends all values received on the default channel to the list indicated by the list argument. Does not return anything. Alternatively, the {{oarg|items}} argument could be used with a string of comma separated items. === list:prepend === {{neweldef|list|prepend}}({{arg|list}}, {{vargs}}) Works like append with the exception that values are added to the beginning of the list. The order of the values in the list will be the reverse of the order in which they are received by {{fel|prepend}}: {{newel|sys|set}}(l, {{newel|list|list}}(4, 5, 6)) prepend(l, 1, 2, 3) {{newel|sys|print}}(l) will print <code>[3, 2, 1, 4, 5, 6]</code> === list:join === {{neweldef|list|join}}({{vargs}}) Concatenates all lists received on the default channel and returns the resulting list. === list:size === {{neweldef|list|size}}({{arg|list}}) Returns the size of the list indicated by the {{arg|list}} argument. === list:first === {{neweldef|list|first}}({{arg|list}}) Returns the first element in a list. === list:last === {{neweldef|list|last}}({{arg|list}}) Returns the last element in a list. === list:butFirst === {{neweldef|list|butFirst}}({{arg|list}}) Returns a list composed of all but the first element in the specified list. === list:butLast === {{neweldef|list|butLast}}({{arg|list}}) Returns a list containing all elements but the last from the specified list. === list:isEmpty === {{neweldef|list|isEmpty}}({{arg|list}}) Tests whether a list is empty. Returns {{newconst|kernel|true}} if the list is empty, and {{newconst|kernel|false}} otherwise. == Map Elements == === map:map === {{neweldef|map|map}}({{vargs}}) Returns a map with the entries received on the default channel. See {{newel|map|entry}}. === map:entry === {{neweldef|map|entry}}({{arg|key}}, {{arg|value}}) Allows the specification of an entry that can be used with {{newel|map|map}} to construct a map. === map:put === {{neweldef|map|put}}({{arg|map}}, {{vargs}}) Adds the entries received on the default channel to the specified map. Existing entries with the same key are replaced. Does not return any value. === map:delete === {{neweldef|map|delete}}({{arg|map}}, {{arg|key}}) Deletes the entry with the specified key from a map. Does not return a value. === map:get === {{neweldef|map|get}}({{arg|map}}, {{arg|key}}) Returns the value corresponding to the specified key from a map. === map:size === {{neweldef|map|size}}({{arg|map}}) Returns the size of the map (the number of entries in the map). === map:contains === {{neweldef|map|contains}}({{arg|map}}, {{arg|key}}) Tests whether the map contains an entry with the specified key. == Logic Elements == Logic elements do not at this time use shortcut evaluation. They always evaluate all their arguments. === sys:and === {{neweldef|sys|and}}({{vargs}}) {{Aliases|&}} Returns the boolean '''and''' value of the arguments received on the default channel. === sys:or === {{neweldef|sys|or}}({{vargs}}) {{Aliases|<nowiki>|</nowiki>}} Returns the boolean '''or''' value of the arguments received on the default channel. === sys:not === {{neweldef|sys|not}}({{arg|value}}) Returns the boolean negation of the value in the value argument. === sys:equals === {{neweldef|sys|equals}}({{arg|value1}}, {{arg|value2}}) {{Aliases|1===}} Tests for the equality of two values. Makes a deep comparison of the arguments. === sys:true === {{neweldef|sys|true}}() Returns {{newconst|kernel|true}}. === sys:false === {{neweldef|sys|false}}() Returns {{newconst|kernel|false}}. == Numeric Elements == === math:sum === {{neweldef|math|sum}}({{vargs}}) {{Aliases|+}} Returns the sum of all the values received on the default channel. === math:product === {{neweldef|math|product}}({{vargs}}) {{Aliases|*}} Returns the product of all the values received on the default channel. === math:subtraction === {{neweldef|math|subtraction}}({{arg|from}}, {{arg|value}}) {{Aliases|-}} Returns the difference between the value specified by the {{arg|from}} argument and the value indicated by the {{arg|value}} argument. === math:quotient === {{neweldef|math|quotient}}({{arg|divisor}}, {{arg|dividend}}) {{Aliases|/}} Divides {{arg|divisor}} by {{arg|dividend}} and returns the resulting value. === math:remainder === {{neweldef|math|remainder}}({{arg|divisor}} , {{arg|dividend}}) {{Aliases|%}} Returns the remainder of the division of {{arg|divisor}} and {{arg|dividend}} === math:square === {{neweldef|math|square}}({{arg|value}}) Returns the square of a number. === math:sqrt === {{neweldef|math|sqrt}}({{arg|value}}) Returns the square root of a number. === math:equalsNumeric === {{neweldef|math|equalsNumeric}}({{arg|value1}}, {{arg|value2}}) Makes a numeric comparison of the values specified by the two arguments. A numeric comparison will try to convert string values to numbers before the comparison is performed. Like {{newel|sys|equals}}, {{fel|equalsNumeric}} performs a deep comparison. equalsNumeric(1, "1") //true equalsNumeric("2", "2.0") //true {{newel|sys|equals}}("2", 2) //false equalsNumeric([1, 2, "3"], ["1", "2", 3]) //true <equalsNumeric value1 = "1"> <{{newel|kernel|number}}>1</number> </equalsNumeric> <!-- true --> <equalsNumeric value1 = "2" value2 = "2.0"/> <!-- true --> <equals value1 = "2"> <{{newel|kernel|number}}>2</number> </equals> <!-- false --> <equalsNumeric> <{{newel|list|list}} items = "1, 2, 3"/> <{{newel|list|list}}> <{{newel|kernel|number}}>1</number> <{{newel|kernel|number}}>2</number> <{{newel|kernel|string}}>3</string> </list> </equalsNumeric> <!-- true --> === math:greaterThan === {{neweldef|math|greaterThan}}({{arg|value1}}, {{arg|value2}}) {{Aliases|&gt;}} Returns {{newconst|kernel|true}} if {{arg|value1}} is strictly larger than {{arg|value2}} === math:lessThan === {{neweldef|math|lessThan}}({{arg|value1}}, {{arg|value2}}) {{Aliases|&lt;}} Returns {{newconst|kernel|true}} if {{arg|value1}} is strictly less than {{arg|value2}} === math:greaterOrEqual === {{neweldef|math|greaterOrEqual}}({{arg|value1}}, {{arg|value2}}) {{Aliases|1=&gt;=}} Returns {{newconst|kernel|true}} if {{arg|value1}} is larger than or equal to {{arg|value2}} === math:lessOrEqual === {{neweldef|math|lessOrEqual}}({{arg|value1}}, {{arg|value2}}) {{Aliases|1=&lt;=}} Returns {{newconst|kernel|true}} if {{arg|value2}} is less than or equal to {{arg|value2}} === math:min === {{neweldef|math|min}}({{vargs}}) Returns the minimum of all numeric values on the default channel. === math:max === {{neweldef|math|max}}({{vargs}}) Returns the maximum of all numeric values on the default channel. === math:int === {{neweldef|math|int}}({{arg|value}}) Returns the integer part of the argument, where “integer part” is to be understood in the mathematical sense (also equivalent to the floor function). === math:ln === {{neweldef|math|ln}}({{arg|value}}) Returns the natural logarithm of the value argument. === math:exp === {{neweldef|math|exp}}({{arg|value}}) Returns e raised to the power of {{arg|value}}. === math:random === {{neweldef|math|random}}() Returns a pseudo-random number in the interval [0,1) with a uniform distribution{{ref|2}}. == Error Handling Elements == === sys:ignoreErrors === {{neweldef|sys|ignoreErrors}}({{oarg|match}}) Executes its arguments returning any values as they are received. If any of the arguments fails, the failure will be matched against the regular expression in {{oarg|match}} if present (otherwise it will be treated as <code>.*</code>). If the match is successful, then the error will be ignored and the next argument will be executed. If the match fails, the error will be propagated. === sys:restartOnError === {{neweldef|sys|restartOnError}}({{arg|match}}, {{arg|times}}) Evaluates its arguments in sequence. If a failure matching the regular expression in {{arg|match}} occurs, all arguments will be re-evaluated for a maximum of times indicated by the {{arg|times}} argument. === sys:generateError === {{neweldef|sys|generateError}}({{arg|error}}, {{oarg|exception}}) Allows the generation of an error. The message of the error is taken from the {{arg|error}} argument. {{oarg|Exception}} is used in the current implementation to pass a Java exception to be attached to the error. === sys:onError === {{neweldef|sys|onError}}({{arg|match}}) {{fel|OnError}} allows the definition of a custom error handler. Handlers defined with {{fel|onError}} are valid for anything executed within the scope of the parent of {{fel|onError}}. Multiple handlers can be defined within the same scope. Whenever an error occurs within the scope of an error handler, the error will be matched against error handlers starting with the inner-most handlers and ending with the outer-most handlers. If a handler matches, it is invoked. When a handler is invoked it will evaluate all its arguments except for {{arg|match}} in sequence. The execution takes place in the context of the failing element. The following variables are defined automatically to be used by the body of the error handler: ; element : The element that caused the error. If the error is corrected by the handler, the execution of the element can be re-started using {{newel|sys|executeElement}}. ; error : The message of the error that occurred. ; trace : A textual representation of the Karajan stack trace. ; exception : In the current implementation exception can either contain a Java exception or the message “No exception available”. Error handlers are not re-entrant. If an unhandled error occurs within the body of the handler, the handler will fail immediately. == String Elements == === str:concat === {{neweldef|str|concat}}({{vargs}}) Concatenates all arguments received on {{vargs}} and returns the resulting string value. === str:split === {{neweldef|str|split}}({{arg|string}}, {{arg|separator}}) Returns a list obtained by splitting {{arg|string}} in tokens separated by {{arg|separator}}. The separator will not be part of the tokens. === str:strip === {{neweldef|str|strip}}({{arg|value}}) Strips all leading and trailing whitespace characters from {{arg|value}}. === str:matches === {{neweldef|str|matches}}({{arg|string}}, {{arg|regexp}}) Returns {{newconst|kernel|true}} if {{arg|string}} matches the regular expression specified by {{arg|regexp}}, and {{newconst|kernel|false}} otherwise. === str:nl === {{neweldef|str|nl}}() Returns the new-line separator. === str:chr === {{neweldef|str|chr}}({{arg|code}}) Returns the character whose code is represented by {{arg|code}}. == Miscellaneous Elements == === sys:print === {{neweldef|sys|print}}({{arg|message}}, {{oarg|nl}}) Returns the value in the message argument on the {{chan|stdout}} channel. If the {{oarg|nl}} argument is not present or set to {{newconst|kernel|true}}, it appends a new line character to the message argument before returning it. The {{newel|kernel|project}} (root element) automatically prints all argument received on the {{chan|stdout}} channel on the console. === sys:echo === {{neweldef|sys|echo}}({{arg|message}}, {{oarg|nl}}, {{oarg|stream}}) Immediately prints the value in the message argument to the console. If the {{oarg|nl}} argument is not present or set to {{newconst|kernel|true}}, it also prints a new line character. If the {{oarg|stream}} argument is present, {{fel|echo}} instead tries to print the message on the specified output stream. The difference between {{newel|sys|print}} and {{fel|echo}} is that {{newel|sys|print}} does not rely on a side-effect to print values. Therefore the evaluation of {{newel|sys|print}} cannot be distinguished from returning the value generated by {{newel|sys|print}}. It is recommended that {{newel|sys|print}} be used instead of {{fel|echo}} if possible. {{neweldef|sys|checkpoint}}({{oarg|file}}, {{oarg|automatic}}, {{oarg|interval}}, {{oarg|timestamped}}, {{oarg|now}}) === sys:checkpoint === Allows the configuration of checkpointing. If the {{oarg|now}} arguments is present and set to {{newconst|kernel|true}}, creates a checkpoint of the state at the time of the evaluation of checkpoint, and writes it to the file indicated by the {{oarg|file}} argument. The {{oarg|automatic}} argument, if set to {{newconst|kernel|true}}, indicates that automatic checkpoints should be created at the interval specified by the {{oarg|interval}} argument (in seconds). If the {{oarg|timestamped}} argument is also present and set to {{newconst|kernel|true}}, the file names in which the checkpoint is saved will have a date and time appended in the '''YYYYMMDDhhmm''' format. === sys:wait === {{neweldef|sys|wait}}({{oarg|delay}}, {{oarg|until}}) When evaluated waits the number of milliseconds specified by the {{oarg|delay}} argument or until the date in the {{oarg|until}} argument before completing. === sys:time === {{neweldef|sys|time}}() Evaluates all arguments and returns the total time elapsed, in milliseconds. {{newel|sys|set}}(t time( {{newel|task|execute}}(executable="/bin/wait", arguments="10", ...) ) ) {{newel|sys|print}}("Job done in ", t/1000, " seconds.") === sys:file === {{neweldef|sys|file:execute}}({{arg|file}}) {{Aliases|executeFile}} Parses and executes a file. The difference between {{newel|kernel|import}} and {{fel|file:execute}} is that {{fel|file:execute}} always parses and executes the file when evaluated, unlike {{newel|kernel|import}} which only parses the file once. {{fel|File:execute}} can therefore be used to execute files which change over time, or to execute different files based on a certain context. === sys:executeElement === {{neweldef|sys|executeElement}}({{arg|element}}, {{arg|args}}, {{vargs}}) Executes an element optionally passing the single value arguments indicated by the {{arg|args}} argument, and the {{vargs}} on the default channel. {{arg|Args}} must be a {{newel|sys|map}} in which the keys are argument names and the values are argument values. It is also possible to pass named arguments directly using the named form. However, this does not allow passing of arguments named {{arg|element}} or {{arg|args}}. If {{arg|args}} is not present, {{vargs}} will be mapped to named arguments according to the rules in Argument Mapping === sys:elementList === {{neweldef|sys|elementList}}() Returns a list containing the arguments to {{fel|elementList}} but in non-evaluated form. The elements in the resulting list can then be evaluated using {{newel|sys|executeElement}} === sys:cacheOn === {{neweldef|sys|cacheOn}}({{arg|value}}) Caches all return values of the rest of its arguments based on the value of the {{arg|value}} argument. Subsequent evaluations of this element in which the value argument will have the same value will not re-evaluate the rest of the arguments, but return the cached values instead. The cache is bound to this static instance of the {{fel|cacheOn}} element. In other words, if another {{fel|cacheOn}} element exists, it will not use the values cached by this element, irrespective of the value of the {{arg|value}} argument. Caching is not guaranteed. It is a mechanism that could help improve performance, but it should not be relied on to guarantee that certain elements are only evaluated once. Also, elements that rely on side-effects to perform their function will not be able to perform those functions if their cached valued is used. {{newel|sys|Echo}} will, for example, not do anything if cached. However, {{newel|sys|print}} will, because it does not rely on a side-effect to print values to the console. === sys:numberFormat === {{neweldef|sys|numberFormat}}({{arg|pattern}}, {{arg|value}}) Allows the formatting of a decimal number. The pattern argument indicates the pattern to be used for formatting (as used by the <code>java.text.DecimalFormat</code> class). The {{arg|value}} argument holds the decimal value that is to be formatted. In short, the following characters can be used in patterns: ; # : Digit; zero not shown ; 0 : Digit; zero is shown ; . : Decimal separator ; , : Grouping separator ; E : Scientific notation separator See also: http://java.sun.com/j2se/1.4.2/docs/api/java/text/DecimalFormat.html === sys:file === {{neweldef|sys|file:contains}}({{arg|file}}, {{arg|value}}) {{Aliases|contains}} {{fel|File:contains}} determines whether a file contains a specific sequence of characters. The {{arg|file}} argument points to the file to be checked, while the {{arg|value}} argument specifies the value to be searched. === sys:uid === {{neweldef|sys|uid}}({{oarg|prefix}}, {{oarg|suffix}}) Returns a string with a unique ID. The {{oarg|prefix}} and {{oarg|suffix}} arguments can be used to specify a prefix and a suffix respectively. In the current implementation, the uniqueness of the returned string is relative to the instance of the interpreter. === sys:file === {{neweldef|sys|file:read}}({{arg|name}}) {{Aliases|readFile}} {{fel|File:read}} reads the contents of a file, pointed to by the {{arg|name}} argument. This is intended for short text files that may possibly hold things like error messages or exit codes. The file is completely read into memory; therefore this element would not be suitable for manipulation of large files. === sys:file === {{neweldef|sys|file:write}}({{arg|name}}, {{oarg|append}}, {{vargs}}) {{fel|File:write}} writes all arguments received on the default channel to the file with the given {{arg|name}}. The file is truncated first, unless the {{oarg|append}} argument is {{newconst|kernel|true}}. When the element terminates, either successfully or not, the file is closed. === sys:outputStream === {{neweldef|sys|outputStream}}({{arg|type}}, {{oarg|file}}) Returns an output stream which can be used for writing values to. The type argument can be one of “stdout”, “stderr”, or “file”. If the type is “file”, the {{oarg|file}} argument must be present and indicate a valid file name. === sys:closeStream === {{neweldef|sys|closeStream}}({{arg|stream}}) Closes a stream opened with {{newel|sys|outputStream}}. === sys:sort === {{neweldef|sys|sort}}({{oarg|descending}}, {{vargs}}) Returns all arguments in sorted order. The values are sorted in ascending order, unless the {{oarg|descending}} argument is set to {{newconst|kernel|true}}. === sys:dot === {{neweldef|sys|dot}}({{vargs}}) Returns the "dot product" of all the arguments, which are expected to be some form of vectors (lists or channels). {{fel|Dot}} returns the result asynchronously if any its arguments is a channel that is not closed. The returned values are lists with a value extracted from each of the vectors. === sys:cross === {{neweldef|sys|cross}}({{vargs}}) Returns the "cross product" of its vector arguments. Each value returned is a list with a value from each vector. {{fel|Cross}} does not work asynchronously. === sys:stats === {{neweldef|sys|stats}}({{oarg|asmap}}) If {{oarg|asmap}} is {{newconst|kernel|false}} then it returns a string summarizing information such as the amount of used and free memory, the number of CPUs, and the current number of active threads. If the {{oarg|asmap}} argument is set to {{newconst|kernel|true}}, it returns the information as a map with the following keys: * memused * heapsize * heapmax * cpucount * threadcount === sys:filter === {{neweldef|sys|filter}}({{oarg|regexp}}, {{oarg|invert}}, {{vargs}}) Filters arguments based on a regular expression, specified by {{oarg|regexp}}. If {{oarg|invert}} is {{newconst|kernel|true}}, matches will be inverted (values that do not match are returned). If only one argument is received on the default channel, and that argument is a list, a list is returned with the values of the argument filtered. If more than one argument is received on the default channel, each argument will be matched against {{oarg|regexp}}. === sys:info === {{neweldef|sys|info}}({{oarg|prefix}}, {{oarg|name}}) Prints information (such as name and arguments) about elements. If the {{oarg|prefix}} argument is present, it will print information about all elements currently defined within that namespace prefix. If the {{oarg|name}} arguments is present it will only print information about the element with the given name. If none of the two possible arguments are present, {{fel|info}} will print information about all currently defined elements (within the scope of {{fel|info}}). == Notes == {{note|1}} At the time of this writing, futures are not properly forwarded if unbound at the time of the remote invocation, possibly causing errors or the remote execution to hang indefinitely. {{note|2}} The uniform distribution relies in the current implementation on the properties of the Java <code>Math.random()</code> function = Task Library = '''Files''': task.k, task.xml The task library interfaces with the Java CoG Kit abstraction classes, allowing the use of services for job submission and file operations. The tasks in this library can function in two modes: scheduled or unscheduled. When scheduled, remote tasks are not executed directly. They are rather passed to a scheduler which can handle issues such as throttling, resource allocation, and task-to-resource mapping. == Task Elements == === task:scheduler === {{neweldef|task|scheduler}}({{arg|type}}, {{arg|resources}}, {{arg|handlers}}, {{oarg|properties}}) Defines the a {{fel|scheduler}} to be used. A scheduler in Karajan has the role of managing resources and assigning abstract tasks (such as {{newel|task|execute}} and {{newel|task|transfer}}) to concrete resources. More details about the role of Karajan schedulers can be found in [[Karajan:The role of schedulers]]. The type describes the particular type of scheduler that is desired. The resources that can be used by the scheduler are passed in the resources argument and can be defined using {{newel|task|resources}}. Each {{fel|scheduler}} will also require a list of task handlers, specified using handlers with the help of the handler element. Each type a {{fel|scheduler}} may support an optional set of properties. The {{oarg|properties}} argument, if specified, must be a {{newel|map|map}} containing string keys and values. The following schedulers are currently available: <ol> <li> '''Default''' The “default” {{fel|scheduler}} uses a round-robin scheduling policy. However it also performs lookahead matching. This means that if a certain host has reached its maximum allowable number of tasks, it will be skipped. Also, if a suitable host is not found for the next task in the queue, other tasks may be scheduled. The {{fel|scheduler}} will use the handlers in the order they were specified in the handlers list, with the first handler having the highest priority. The default scheduler supports the following properties: ;jobsPerCpu: Sets the maximum number of tasks that the scheduler will allocate for one CPU. ;maxSimultaneousJobs: Sets the total maximum number of remote tasks that the scheduler will allow at any given time. ;showTaskList: If set to {{newconst|kernel|true}} the scheduler will pop-up a window providing a lists of tasks that are being executed, and additional task and memory statistics. </li> <li> '''Weighted''' The weighted scheduler is an experimental adaptive scheduler that maintains a "performance" history of all the hosts that it manages. Each host starts with a score of 1. If a task fails on a host, its score is decreased by a certain factor, if a task succeeds, the score is increased by a certain factor, and so on. Scores are also temporarily decreased with each job running on a host. Periodically, a normalization of the scores is performed. The normalization process involves multiplying each score with the same factor such that the geometric average of the scores after the normalization is 1. The weighted scheduler has two host selection strategies: "random" and "best". The "best" strategy means that the host with the highest score at the time of the submission of a task will be chosen for that task. By contrast the "random" strategy uses a weighted random choice, which gives higher chances for a host with a higher score. With a weighted random policy, every host, assuming that scoreLowCap > 0, and given a sufficiently large number of tasks, will eventually get a chance to be used (and thus possibly increase its score). The following properties are available for the weighted scheduler (default values are listed in parentheses, after the property name; factors are values with which the score for a host is multiplied in a certain event): ;connectionRefusedFactor: (0.1) factor for connection refused exceptions while submitting to a host ;connectionTimeoutFactor: (0.05) factor for connection timeout expcetions while submitting to a host ;jobSubmissionTaskLoadFactor: (0.9) used when a job is submitted successfully; reversed when the job completes (either successfully or in failure) ;transferTaskLoadFactor: (0.9) used when a task transfer is started; reversed when the transfer completes ;fileOperationTaskLoadFactor: (0.95) used when a file operation is started; reversed when the operation completes ;successFactor: (1.2) factor used upon successful completion of a task ;failureFactor: (0.9) used when a task fails ;scoreHighCap: (100) maximum value for a score ;scoreLowCap: (0.001) minimum value for a score ;renormalizationDelay: (100) number of tasks submitted after which a normalization occurs ;policy: ("random") use either a weighted "random" host selection policy or a "best" score host selection policy. </li> </ol> The scope of the {{fel|scheduler}} is similar to a deeply accessible variable. In fact, the current implementation works by setting a hidden variable (named "#scheduler") in the current scope, and which would be visible in any dynamic sub-scope. The following example shows a typical scheduler definition: {{newel|kernel|import}}("sys.k") {{newel|kernel|import}}("task.k") scheduler("default" {{newel|task|resources}}( {{newel|task|host}}("host1", cpus = 256 {{newel|task|service}}("execution", provider = "gt2", jobManager = "PBS", uri = "host1.example.net:2119") {{newel|task|service}}("file", provider = "gsiftp", uri = "host1.example.net:2911") ) {{newel|task|host}}("host2", cpus = 2 {{newel|task|service}}("execution", provider = "gt2", uri = "host2.example.net:2119") {{newel|task|service}}("file", provider = "gsiftp", uri = "host2.example.net:2911") ) ) handlers = {{newel|sys|list}}( {{newel|task|handler}}("execution", "gt2") {{newel|task|handler}}("execution", "gt4") {{newel|task|handler}}("file", "gsiftp") {{newel|task|handler}}("file-transfer", "ssh") ) properties = {{newel|map|map}}( {{newel|map|entry}}("jobsPerCpu", "1") ) ) Or in XML: <{{newel|kernel|import}} file="sys.xml"/> <{{newel|kernel|import}} file="task.xml"/> <scheduler type = "default"> <{{newel|task|resources}}> <{{newel|task|host}} name="host1" cpus="256"> <{{newel|task|service}} type="execution" provider="gt2" jobManager="PBS" uri = "host1.example.net:2119"/> <{{newel|task|service}} type="file" provider="gsiftp" uri="host1.example.net:2911"/> </{{newel|task|host}}> <{{newel|task|host}} name="host2" cpus="2"> <{{newel|task|service}} type="execution" provider="gt2" uri="host2.example.net:2119"/> <{{newel|task|service}} type="file" provider="gsiftp" uri="host2.example.net:2911"/> </{{newel|task|host}}> </{{newel|task|resources}}> <{{newel|kernel|argument}} name="handlers"> <{{newel|sys|list}}> <{{newel|task|handler}} type="execution" provider="gt2"/> <{{newel|task|handler}} type="execution" provider="gt4"/> <{{newel|task|handler}} type="file" provider="gsiftp"/> <{{newel|task|handler}} type="file-transfer" provider="ssh"/> </{{newel|sys|list}}> </{{newel|kernel|argument}}> <{{newel|kernel|argument}} name="properties"> <{{newel|map|map}}> <{{newel|map|entry}} key="jobsPerCpu" value="1"/> </{{newel|map|map}}> </{{newel|kernel|argument}}> </scheduler> === task:handler === {{neweldef|task|handler}}({{arg|type}}, {{arg|provider}}) A handler specifies a Java CoG Kit Abstraction handler. A handler is used to submit tasks. Type indicates the type of handler. They type is string and can have one of the following values: “execution”, “file”, and “file-transfer”. Execution handlers are used for submitting jobs. File handlers are used for file operations (such as renaming, deleting, and listing of files). File transfer handlers are used only for transferring files. It is possible to transfer files using file handlers, but it is not possible to delete a file using a file transfer handler. The provider argument indicates the provider to be used for the handler. For a list of currently supported providers please see the abstractions guide. === task:resources === {{neweldef|task|resources}}({{vargs}}) Encapsulates a set of hosts which can be specified using host. === task:host === {{neweldef|task|host}}({{arg|name}} , {{oarg|cpus}} , {{vargs}}) Returns a host definition. The name argument indicates the host name or IP address. The number of CPUs of the host can be specified using the *cpus argument. A set of services can also be specified on the default channel. === task:service === {{neweldef|task|service}}({{arg|type}}, {{arg|provider}}, {{oarg|uri}}, {{oarg|project}}, {{oarg|jobManager}}, {{oarg|securityContext}}) Returns a service definition. The type of the service can be one of “execution”, “file”, or “file-transfer”. Provider indicates The Java CoG Kit abstraction provider for the service. For a list of currently supported providers please see the abstractions guide. The {{oarg|uri}} argument can be used to specify a URI for the service. If missing the host name of the {{newel|task|host}} containing the service will be used. The {{oarg|project}} argument can be used to automatically bind a queuing system project to the service in order to alleviate the need to do it with the execute element. The {{oarg|jobManager}} argument can be used to specify a [[job manager]] different from the default. Examples of job managers include ''Fork'', ''PBS'', and ''Condor''. A non-default [[security context]] can be specified using the {{oarg|securityContext}} argument. === task:securityContext === {{neweldef|task|securityContext}}({{arg|provider}}, {{arg|credentials}}) Returns a Java CoG Kit abstraction security context. The returned context will be instantiated for the specified provider. The credentials argument can be used to pass a specific set of credentials to security context. === task:allocateHost === {{neweldef|task|allocateHost}}({{arg|name}}) Allows tasks to be grouped on one host. By default, the scheduler assigns a different host to each task. {{fel|AllocateHost}} can be used to reserve a host from the scheduler until it completes. The {{arg|name}} indicates the name of the variable to be set with the allocated host, and is automatically quoted. //Define a scheduler {{newel|task|scheduler}}( ... ) allocateHost(host1 {{newel|task|execute}}("/bin/date", stdout="date", host=host1) {{newel|task|transfer}}(srcfile="date", srchost=host1, desthost="localhost") ) Or, in XML: <!-- Define a scheduler --> <{{newel|task|scheduler}}> ... </scheduler> <allocateHost name="host1"> <{{newel|task|execute}} executable="/bin/date" stdout="date" host="{host1}"/> <{{newel|task|transfer}} srcfile="date" srchost="{host1}" desthost="localhost"/> </allocateHost> The default scheduler uses a late binding mechanism with {{fel|allocateHost}}. It generates a virtual host that is only bound to an actual host when the first task using it is submitted to the scheduler. This removes the limitation on the number of parallel {{fel|allocateHost}} that can be running, and allows contained jobs to be submitted to the scheduler, which will later handle the throttling issues. Multiple {{fel|allocateHost}} can be nested allowing the grouping of tasks on multiple dependent hosts. === task:host === {{neweldef|task|host|hasService}}({{arg|host}}, {{arg|type}}, {{arg|provider}}) Checks if a host, specified with the {{newel|task|host}} element contains a service of the specified {{arg|type}} and with the specified {{arg|provider}}. Returns {{newconst|kernel|true}} if such a service exists, and {{newconst|kernel|false}} otherwise. === task:execute === {{neweldef|task|execute}}({{arg|executable}}, {{arg|arguments}}, {{oarg|directory}}, {{oarg|stdout}}, {{oarg|stderr}}, {{oarg|stdin}}, {{oarg|redirect}}, {{oarg|provider}}, {{oarg|host}}, {{oarg|count}}, {{oarg|jobtype}}, {{oarg|maxtime}}, {{oarg|maxwalltime}}, {{oarg|maxcputime}}, {{oarg|environment}}, {{oarg|queue}}, {{oarg|project}}, {{oarg|minmemory}}, {{oarg|maxmemory}}, {{oarg|nativespec}}, {{oarg|delegation}}) Executes a remote job. {{arg|Executable}} indicates the executable to be run. Arguments can be passed to the executable using {{arg|arguments}}. If present, the {{oarg|directory}} argument specifies the remote directory in which the job will be executed. {{oarg|Stdout}} and {{oarg|stderr}} allow the redirection of the output and error streams to a remote file. {{oarg|Stdin}} allows the redirection of the standard input from a remote file. If {{oarg|redirect}} is set to {{newconst|kernel|true}} the standard output and standard error of the remote job is redirected to the local console. The {{oarg|host}} argument allows the job to be executed on a specific host, and the {{oarg|provider}} argument allows the job to be executed using a specific provider. The {{oarg|delegation}} can be used to enable credential delegation with providers which support it. Credential delegation is disabled by default. The rest of the arguments are passed to the underlying provider. A native specification (such as a classic GRAM RSL, or WS-GRAM RSL) can be passed to the provider using the {{oarg|nativespec}} argument. === task:transfer === {{neweldef|task|transfer}}({{oarg|srcfile}}, {{oarg|srcdir}}, {{oarg|srchost}}, {{oarg|destfile}}, {{oarg|destdir}}, {{oarg|desthost}}, {{oarg|provider}}, {{oarg|srcprovider}}, {{oarg|destprovider}}, {{oarg|thirdparty}}, {{oarg|srcOffset}}, {{oarg|length}}, {{oarg|destOffset}}) Transfers a file. The file can be transfered between the local machine and a remote machine, or between two remote machines. The name of the source file is specified by the {{oarg|srcfile}} argument. If present, {{oarg|destfile}} specifies the name of the target file, otherwise the source file name is used. The {{oarg|srcdir}} argument indicates the directory on the source machine where the source file can be found. If the {{oarg|srcdir}} argument is missing, the default directory will be assumed (provider dependent). The {{oarg|destdir}} argument indicates the directory on the target machine where the file will be copied. If the {{oarg|destdir}} argument is missing, the default directory will be assumed (provider dependent). {{oarg|Srchost}} and {{oarg|desthost}} indicate the source and the target machines respectively, while the {{oarg|provider}} argument can be used to force the scheduler to use a specific provider, or in the event a scheduler is not used. If the source and the destination use distinct providers, the {{oarg|srcprovider}} and {{oarg|destprovider}} arguments can be used. The {{oarg|thirdparty}} can be used to indicate that a direct transfer between two machines, none of which are the local host, is requested. At this time, only GridFTP supports third party transfers. By default, the Java CoG Kit Abstractions will use simulated third party transfers (routed through the local host) even if both the source and destination are different from the local host. '''Partial transfers''' can be achieved using {{oarg|srcOffset}}, {{oarg|length}}, and {{oarg|destOffset}}. Currently these are only supported with GridFTP 3rd party transfers. === task:file:list === {{neweldef|task|file:list}}({{arg|dir}}, {{oarg|host}}, {{oarg|provider}}) Returns a list of files in a directory specified by {{arg|dir}}, on the {{oarg|host}} machine. The {{oarg|provider}} argument can be used to select a specific provider for the operation. {{oarg|Provider}} defaults to the local provider. === task:file:remove === {{neweldef|task|file:remove}}({{arg|name}}, {{oarg|host}}, {{oarg|provider}}) Removes a file specified by {{arg|name}}, on the {{oarg|host}} machine. The {{oarg|provider}} argument can be used to select a specific provider for the operation. {{oarg|Provider}} defaults to the local provider. === task:file:exists === {{neweldef|task|file:exists}}({{arg|name}}, {{oarg|host}}, {{oarg|provider}}) Returns {{newconst|kernel|true}} if the file specified by {{arg|name}} exists on the {{oarg|host}} machine. The {{oarg|provider}} argument can be used to select a specific provider for the operation. {{oarg|Provider}} defaults to the local provider. === task:dir:make === {{neweldef|task|dir:make}}({{arg|name}}, {{oarg|host}}, {{oarg|provider}}) Creates a directory specified by {{arg|name}}, on the {{oarg|host}} machine. The {{oarg|provider}} argument can be used to select a specific provider for the operation. {{oarg|Provider}} defaults to the local provider. === task:dir:remove === {{neweldef|task|dir:remove}}({{arg|name}}, {{oarg|host}}, {{oarg|provider}}) Removes an empty directory. === task:file:isDirectory === {{neweldef|task|file:isDirectory}}({{arg|name}}, {{arg|host}}, {{arg|provider}}) Returns {{newconst|kernel|true}} if the file specified by {{arg|name}} exists on the {{oarg|host}} machine and it is a directory. The {{oarg|provider}} argument can be used to select a specific provider for the operation. {{oarg|Provider}} defaults to the local provider. === task:file:chmod === {{neweldef|task|file:chmod}}({{arg|name}}, {{arg|mode}}, {{oarg|host}}, {{oarg||provider}}) Changes the permissions on the file specified with the {{arg|name}} argument to the mode string indicated by the {{arg|mode}} argument. If {{oarg|host}} and {{oarg|provider}} are present, the operation is done remotely using the respective provider. === task:file:rename === {{neweldef|task|file:rename}}({{arg|from}}, {{arg|to}}, {{oarg|host}}, {{oarg|provider}}) Renames a file. The source and target name are specified using the {{arg|from}} and {{arg|to}} arguments. If {{oarg|host}} and {{oarg|provider}} are present, the operation is done remotely. === task:SSHSecurityContext === {{neweldef|task|SSHSecurityContext}}({{arg|credentials}}) Instantiates a SSH security context. This is simply a convenience function for <code>securityContext(”ssh”, credentials)</code>. === task:InteractiveSSHSecurityContext === {{neweldef|task|InteractiveSSHSecurityContext}}({{oarg|username}}, {{oarg|privateKey}}, {{oarg|nogui}}) Instantiates a SSH security context which will lazily display a dialog window allowing the user to input a user-name/password pair or a user-name/private key/passphrase set. The dialog will only be displayed once per each instance of an interactive SSH security context. If {{oarg|username}} and/or {{oarg|privateKey}} are specified, the values are used to pre-fill the corresponding dialog fields. The {{fel|InteractiveSSHSecurityContext}} makes use of a class present in the SSH provider of the Java CoG Kit. This class will try to determine whether a GUI can be displayed or not (by checking <code>GraphicsEnvironment.isHeadless())</code>. If a Swing dialog cannot be displayed, a text-mode interface is used instead. The {{oarg|nogui}} argument can be used to force the use of the text-mode interface (by setting it to {{newconst|kernel|true}}). === task:passwordAuthentication === {{neweldef|task|passwordAuthentication}}({{arg|username}}, {{arg|password}}) Returns a {{arg|username}}/{{arg|password}} pair suitable to be used as a credential for a {{newel|task|securityContext}}. === task:publicKeyAuthentication === {{neweldef|task|publicKeyAuthentication}}({{arg|username}}, {{arg|privatekey}}, {{arg|passphrase}}) Returns a {{arg|username}}/{{arg|privatekey}}/{{arg|passphrase}} set which can be used as credentials for {{newel|task|securityContext}}. The {{arg|privatekey}} argument must point to a file containing the private key. = Java Library = '''Files:''' java.k, java.xml The Java library allows limited interfacing with Java classes and objects from Karajan. == Java Elements == === java:new === {{neweldef|java|new}}({{arg|classname}}, {{oarg|types}}, {{vargs}}) Instantiates a new Java object and returns it. {{arg|Classname}} represents the fully qualified name of the class. The {{oarg|types}} argument is a list of fully qualified class names used to search for a constructor. The arguments on the default channel are passed to the constructor after performing a conversion based on the {{oarg|types}} argument. The {{oarg|types}} argument is not always necessary, but should be used if Karajan cannot determine the types of the arguments that need to be passed to the constructor. {{newel|sys|set}}(x {{newel|java|new}}("java.lang.Double", "1.0") /* The types argument is not necessary since * the string argument is automatically mapped * to java.lang.String */ ) {{newel|sys|set}}(j {{newel|java|new}}("java.lang.Integer", types = ["int"], 1) /* The types argument is required since the numeric * type used by Karajan cannot be mapped automatically * to a specific Java numeric type. */ ) Primitive Java types are represented by their corresponding keywords: <code>boolean, byte, char, int, long, float,</code> and <code>double</code>. === java:invokeMethod === {{neweldef|java|invokeMethod}}({{arg|method}}, {{oarg|static}}, {{oarg|classname}}, {{oarg|object}}, {{oarg|types}}, {{vargs}}) Invokes a method on a Java object or a static method on a Java class. The invocation is static if {{oarg|static}} is set to {{newconst|kernel|true}} or {{oarg|classname}} is present. Otherwise the value of {{oarg|object}} is taken to be a Java object and the invocation is done on a virtual method of the object. {{arg|Method}} indicates the name of the method. Unless the {{oarg|types}} argument is present, Karajan will try to determine the method signature from the types of {{vargs}}. A method is searched for in the inheritance hierarchy of the object. If one is found, the method is invoked and, if its return value is not void the returned value is returned on the default channel. The format of the type argument is identical to the one for {{newel|java|new}}. === java:executeMain === {{neweldef|java|executeMain}}({{arg|class}}, {{vargs}}) Invokes <code>static void main(String[] args)</code> on a class. The qualified class name should be passed in the {{arg|class}} argument. The arguments on the default channel are converted to strings and passed as the args to the main method. === java:getField === {{neweldef|java|getField}}({{arg|field}}, {{oarg|object}}, {{oarg|classname}}) Returns the value of a field from an object or class. The filed name is passed in {{arg|field}}. If {{oarg|object}} is present, the value of the instance field of the object is retrieved. If {{oarg|classname}} is present, the class field (static field) of the specified class is retrieved. Arguments {{oarg|object}} and {{oarg|classname}} are mutually exclusive. === java:waitForEvent === {{neweldef|java|waitForEvent}}({{vargs}}) {{fel|WaitForEvent}} is a rather obscure and incomplete element. It waits for a Java event such as a button being pressed, or a window being closed. Each argument is a list composed by three elements: * A return value which will be returned when the associated event occurs * The type of the event to wait for. Currently the following types are supported: ;<code>java.awt.events.ActionEvent</code>: Can be used to wait for a button being pressed (or any other actions that have a addActionListener(java.awt.events.ActionListener) method. ;<code>java.awt.events.WindowEvent</code>: Can be used to listen for the windowClosing notification on a window. * The source object to be used When the event occurs, {{fel|waitForEvent}} completes returning the return value specified as the first item in the list corresponding to the event that the list represents. === java:classOf === {{neweldef|java|classOf}}({{arg|object}}) Returns the Java class name of the specified {{arg|object}}. === java:null === {{neweldef|java|null}}() Karajan has no explicit <code>null</code> value. However, it may be necessary that a <code>null</code> value be used when invoking Java methods. {{fel|Null}} provides that. = HTML Library = '''Files:''' html.k, html.xml The HTML library can be used to produce HTML output from Karajan scripts. It is incomplete, but able to generate simple HTML pages. It is listed here in the hope that it will be useful. However, not many guarantees are made about it. The library elements utilize the {{chan|html}} channel. test jump to html:write {{newel|html|write}} == HTML Elements == === html:write === {{neweldef|html|write}}({{arg|file}}, {{chan|html}}) Writes all arguments received on the {{chan|html}} channel to the specified file. It is roughly equivalent to the following: {{newel|sys|file:write}}(name=file {{newel|channel|from}}(html ... ) ) The following code produces a simple HTML file: {{newel|kernel|import}}("html.k") write(file="out.html" {{newel|html|html}}( {{newel|html|head}}( {{newel|html|title}}("Title") ) {{newel|html|body}}( {{newel|html|h1}}("Heading 1") {{newel|html|text}}("Some text") ) ) ) The XML syntax may appear more convenient for using the HTML library: <{{newel|kernel|import}} name="html.xml"/> <write file="out.html"> <{{newel|html|html}}> <{{newel|html|head}}> <{{newel|html|title}}>Title</title> </head> <{{newel|html|body}}> <{{newel|html|h1}}>Heading 1</h1> <{{newel|html|text}}>Some text</text> </body> </html> </write> === html:quickstart === {{neweldef|html|quickstart}}({{arg|title}}, {{chan|html}}) Produces an HTML skeleton output with the given title, and the arguments on the html channel substituted inside the <code><body></code> tag. === html:html === {{neweldef|html|html}}({{chan|html}}) Generates the <code><html></html></code> tag pairs with the {{chan|html}} arguments substituted inside. === html:head === {{neweldef|html|head}}({{chan|html}}) === html:title === {{neweldef|html|title}}({{arg|title}}) === html:body === {{neweldef|html|body}}({{oarg|bgcolor}}, {{chan|html}}) === html:table === {{neweldef|html|table}}({{oarg|width}}, {{oarg|height}}, {{oarg|border}}, {{oarg|cellpadding}}, {{oarg|rules}}, {{oarg|bgcolor}}, {{oarg|class}}, {{oarg|style}}, {{chan|html}}) === html:tr === {{neweldef|html|tr}}({{oarg|width}}, {{oarg|height}}, {{chan|html}}) === html:td === {{neweldef|html|td}}({{arg|text}}, {{oarg|width}}, {{oarg|height}}, {{oarg|colspan}}, {{oarg|bgcolor}}, {{oarg|align}}, {{oarg|valign}}, {{chan|html}}) A note that needs to be made about {{fel|td}} is that the {{arg|text}} argument is mandatory, which means that it needs to always be present, even if a HTML nested element is used. The following example shows the correct usage for a table cell containing just an image: ... {{newel|html|table}}( ... {{newel|html|tr}}( td("", {[el|html|img}}(src="image.jpg")) ) ... ) ... or ... <{{newel|html|table}}> ... <{{newel|html|tr}}> <td text=""> <{{newel|html|img}} src="image.jpg"/> &lt;/td> &lt;/tr> ... </table> ... === html:th === {{neweldef|html|th}}({{arg|text}}, {{oarg|width}}, {{oarg|height}}, {{oarg|colspan}}, {{chan|html}}) See also: {{newel|html|td}} === html:h1 === {{neweldef|html|h1}}({{arg|text}}, {{chan|html}}) === html:h2 === {{neweldef|html|h2}}({{arg|text}}, {{chan|html}}) === html:h3 === {{neweldef|html|h3}}({{arg|text}}, {{chan|html}}) === html:h4 === {{neweldef|html|h4}}({{arg|text}}, {{chan|html}}) === html:h5 === {{neweldef|html|h5}}({{arg|text}}, {{chan|html}}) === html:h6 === {{neweldef|html|h6}}({{arg|text}}, {{chan|html}}) === html:ul === {{neweldef|html|ul}}({{chan|html}}) === html:pre === {{neweldef|html|pre}}(text , {{chan|html}}) === html:br === {{neweldef|html|br}}() === html:li === {{neweldef|html|li}}() === html:a === {{neweldef|html|a}}({{arg|text}}, {{oarg|href}}, {{oarg|style}}, {{chan|html}}) === html:anchor === {{neweldef|html|anchor}}({{arg|name}}) Anchor is used to define a HTML anchor since the a element does not support this functionality. It effectively returns <code><a name="<nowiki>{</nowiki>{{arg|name}}<nowiki>}</nowiki>"/></code> on the {{chan|html}} channel. === html:img === {{neweldef|html|img}}({{arg|src}}, {{oarg|border}}) === html:text === {{neweldef|html|text}}({{arg|text}}) Returns the value of the {{arg|text}} argument on the {{chan|html}} channel. = Forms Library = == Concepts == '''Files:''' forms.k, forms.xml The forms library can be used to build and gather information from GUI forms. The current implementation uses the Java Swing library. === Component ID === Each component whose state is user-modifiable (such as button, textField, etc.) must have an ID, which can be used to retrieve the state of the component when using the form element. === Component Layout === The layout of components is done using {{newel|form|hbox}} and {{newel|form|vbox}}. === Component Alignment === Components support alignment using the {{oarg|halign}} and {{oarg|valign}} arguments. {{oarg|Halign}} and {{oarg|valign}} have values between 0.0 and 1.0, with 0.0 representing left/top alignment, and 1.0 representing right/bottom alignment. By default, both {{oarg|halign}} and {{oarg|valign}} are set to 0.5 (centered). == Form Elements == === form:form === {{neweldef|form|form}}({{arg|id}}, {{arg|title}}, {{arg|waiton}}) This element represents the main form element. It creates a window based on the specification received in {{vargs}}, and displays it. It will then wait for a control with the '''id''' that matches the {{arg|waiton}} value, and terminates returning a map pairing control '''id'''s and their values. The layout of the controls in the window is performed by recursive nesting of vertical and horizontal containers ({{newel|form|vbox}} and {{newel|form|hbox}} respectively). === form:hbox === {{neweldef|form|hbox}}({{oarg|homogeneous}}, {{vargs}}) Represents a horizontal container. All controls specified by {{vargs}} are laid out one after another horizontally. By default, all controls are assigned an area of equal height, but the width varies according to the natural dimensions of the control (the total width is proportionally divided between the components according to their size). The {{oarg|homogenous}} argument can be used to force all cells to have an equal width. The various possibilities are illustrated in Figure 1, Figure 2, and Figure 3. [[Image:workflow0x.gif]] '''Figure 1''': A non-homogenous hbox [[Image:workflow1x.gif]] '''Figure 2''': A scaled non-homogenous hbox [[Image:workflow2x.gif]] '''Figure 3''': A homogenous hbox === form:vbox === {{neweldef|form|vbox}}({{oarg|homogenous}}, {{vargs}}) A {{fel|vbox}} is similar to a {{newel|form|hbox}} but with the sub-components laid vertically. === form:label === {{neweldef|form|label}}({{arg|text}}, {{oarg|halign}}, {{oarg|align}}) Represents a text label. The text content is described by the {{arg|text}} argument. See also: [[Karajan:Forms_Library#Component_Alignment|Component Alignment]] === form:button === {{neweldef|form|button}}({{arg|id}}, {{arg|caption}}, {{oarg|halign}}, {{oarg|valign}}) Represents a push-button. The text used for the label is specified with the caption argument. The ID is specified with the {{arg|id}} argument. See also: [[Karajan:Forms_Library#Component_Alignment|Component Alignment]] === form:checkBox === {{neweldef|form|checkBox}}({{arg|id}}, {{oarg|caption}}, {{oarg|checked}}, {{oarg|halign}} , {{oarg|valign}}) Constructs a check-box, whose initial state can be set using the {{oarg|checked}} argument. By default, the check-box is not checked. The label of the checkbox is specified using the {{arg|caption}} argument. See also: [[Karajan:Forms_Library#Component_Alignment|Component Alignment]] === form:radioBox === {{neweldef|form|radioBox}}({{arg|id}}, {{arg|caption}}, {{vargs}}) Allows the construction of a radio-button group. The group will have a titled border with the text defined by the {{arg|caption}} argument. RadioBox expects radio button definitions on the default channel ({{vargs}}). === form:radioButton === {{neweldef|form|radioButton}}({{arg|id}}, {{arg|caption}}, {{oarg|selected}}, {{oarg|halign}}, {{oarg|valign}}) Defines a radio button, with the text set by the {{arg|caption}} argument. By default, the first button in a {{fel|radioBox}} is selected, unless the {{oarg|selected}} argument is used on a different radio button. See also: [[Karajan:Forms_Library#Component_Alignment|Component Alignment]] === form:textField === {{neweldef|form|textField}}({{arg|id}}, {{oarg|columns}}, {{oarg|halign}}, {{oarg|valign}}) Describes a text field, used for text input. A text field has only one line of text. The number of columns of the text field is indicated using the {{oarg|columns}} argument. See also: [[Karajan:Forms_Library#Component_Alignment|Component Alignment]] === form:passwordField === {{neweldef|form|passwordField}}({{arg|id}}, {{oarg|columns}}, {{oarg|halign}}, {{oarg|valign}}) A {{fel|passwordField}} is similar to a {{newel|form|textField}}, with the difference that the characters entered are visually represented by asterisks (*). See also: {{newel|form|textField}} === form:comboBox === {{neweldef|form|comboBox}}({{arg|id}}, {{oarg|halign}}, {{oarg|valign}}, {{vargs}}) Constructs a combo-box, which can be used to select from multiple items in a drop-down list. The items are specified as {{vargs}}, using {{newel|form|comboItem}}. See also: [[Karajan:Forms_Library#Component_Alignment|Component Alignment]] === form:comboItem === {{neweldef|form|comboItem}}({{oarg|text}}, {{oarg|selected}}) Specifies a {{newel|form|comboBox}} item. The text of the item is indicated by the {{arg|text}} argument. By default, the first item in a {{newel|form|comboBox}} is selected, unless the {{oarg|selected}} argument is used on a different item. === form:HSeparator === {{neweldef|form|HSeparator}}() Constructs a horizontal separator component. === form:VSeparator === {{neweldef|form|VSeparator}}() Constructs a vertical separator component. === form:filler === {{neweldef|form|filler}}({{oarg|width}}, {{oarg|height}}) Defines a filler component. A filler component serves no functional purpose, but can be used to influence the layout of a form by providing a visually invisible component of the specified {{oarg|width}} and {{oarg|height}} (in pixels). === form:messageDialog === {{neweldef|form|messageDialog}}({{arg|message}}, {{arg|title}}) Used to display a popup window displaying the specified {{arg|message}} and having the specified {{arg|title}}. The element completes when the popup message is closed by the user, either by using the window controls or the '''OK''' button. = Restart Library = The Restart Log Library provides fault tolerance in a style similar to that of [[Condor DAGMan]]. The execution of certain operations is recorded into a log file on the disk. In case of a failure the execution can be resumed using the information saved in the log file. The operations that previously completed successfully will not be re-executed. The library defines two elements: {{newel|rlog|restartLog}} and {{newel|rlog|logged}}. This mechanism offers no guarantees of semantic consistency after a restart if the control flow is influenced by factors that change between the original execution and the resumed execution. It is generally safe to use this mechanism with {{newel|sys|for}} and {{newel|sys|parallelFor}} if the iteration values are fixed. Additionally, return values of {{newel|rlog|logged}} elements are not recorded. Consequently a resumed {{newel|rlog|logged}} element will not return anything. Usage example: {{newel|kernel|import}}("sys.k") {{newel|kernel|import}}("task.k") {{newel|kernel|import}}("rlog.k") {{newel|rlog|logged}}( {{newel|task|transfer}}(srcfile="a.txt", desthost="host.example.org", provider="gt2") ) {{newel|sys|parallel}}( {{newel|rlog|logged}}( {{newel|task|execute}}(executable="/bin/cat", arguments="a.txt", stdout="b.txt", host="host.example.org", provider="gt2") ) {{newel|rlog|logged}}( {{newel|task|execute}}(executable="/bin/cat", arguments="a.txt", stdout="c.txt", host="host.example.org", provider="gt2") ) ) {{newel|sys|parallelFor}}(out, {{newel|sys|list}}("b.txt", "c.txt") {{newel|rlog|logged}}( {{newel|task|transfer}}(srcfile=out, srchost="host.example.org", provider="gt2") ) ) Resuming: cog-workflow workflow.k -rlog:resume=workflow.0.rlog == rlog:restartLog == {{neweldef|rlog|restartLog}}({{oarg|resume}}, {{oarg|name}}, {{chan|restartlog}}) {{fel|RestartLog}} performs the following functions: # Opens a log file. The prefix of the log file name is taken from the {{oarg|name}} argument or, if the {{oarg|name}} argument is missing, from the file name of the current script being executed. The actual file name is obtained by appending a dot character ("."), a unique numeric identifier, and the ".rlog" extension. {{fel|RestartLog}} will attempt to successively use increasing numeric identifiers, starting from 0 (zero). If a log file with that identifier already exists or if an exclusive lock on the file cannot be obtained, the next number is used. An exclusive lock is acquired on the log file such that other processes will not attempt to use the same file. # Accepts arguments on the {{chan|restartlog}} channel and writes them to the log file. After writing each value, the file buffers are immediately flushed to the disk using the [http://java.sun.com/j2se/1.4.2/docs/api/java/io/FileDescriptor.html#sync() FileDescriptor.sync()] method. # In the case of a restart it also parses a previous log and builds the necessary data in a way that the {{newel|rlog|logged}} elements can use. If during a restart {{fel|restartLog}} cannot acquire an exclusive lock on the log file, it will not attempt to use another log file, but fail instead. # Upon successful completion, it closes and deletes the log file. Restarts can be triggered in two ways: # Using the {{oarg|resume}} argument with the full file name of a log file # By specifying the -rlog:resume=<logname> command line argument to the script (not to the interpreter). == rlog:logged == {{neweldef|rlog|logged}}() {{fel|Logged}} elements execute their child elements and, upon termination, they return, on the {{chan|restartlog}} channel an identifier that uniquely identifies a {{fel|logged}} element within a given execution, and a thread id, which is used to differentiate between concurrent runs of the same element. The same (element id; thread id) pair can occur multiple times in a log file, but it only reflects successive executions of an element within the same thread. If a restart is in effect, {{fel|logged}} elements will analyze data parsed from the log, and if there exists an entry with the same (element id; thread id) pair which has a count larger than 0, the count will be decreased and {{fel|logged}} will complete without executing its arguments/child elements. It will consequently not return any values whatsoever, not even values that were returned by its child elements in a previous run. It is therefore recommended that {{fel|logged}} only wrap elements that do not return values. = Service = Karajan features a service which can be used to execute parts of a script remotely. Some of the features of the service are listed below: ; Security : The service and client use GSI security enabling strong authentication and encryption of data ; Configurable communication channels : The Karajan networking infrastructure consists of configurable communication sub-systems, which separate the high level messaging protocols from the low level implementation details. The current implementation uses SSL sockets over TCP/IP, and can be configured to use persistent connections, or callbacks, or polling (or a combination of them), and the configurations can be different for each host or for a specific domain. This provides firewall transparency when needed, yet can minimize resource consumption and maximize performance when firewalls are not involved. ; Semantic transparency : Remote execution can be seamlessly integrated with local execution, while preserving most of the system semantics, such as return values, definitions, etc. == Using the Service == The Karajan service can be accessed from Karajan scripts using the {{newel|sys|remote}} element. == Shared or Personal == The Karajan service can be used in two modes: ''personal'' or ''shared''. In ''personal'' mode, the service runs under a user’s credential, loaded from a proxy certificate. It is therefore necessary that a valid proxy exists before the service is started in ''personal'' mode. In this mode, all libraries are available for use without restrictions, but the connections are limited to clients using the same credential as the one used for starting the service. In ''shared'' mode, the service requires a host credential. In this mode, connections can be initiated by multiple users, and authorization is performed based on a [[grid-map]] file. Access to certain libraries or functions that could be used to access files, resources, or other information belonging to other users using the service are restricted. Any attempt to execute such functions will result in the execution failing. The following list enumerates libraries and elements whose use is restricted in shared mode: * The {{Klib|java}} * Elements not in the {{Klib|task}} which can be used to access local files: {{newel|sys|file:contains}}, {{newel|sys|file:read}}, {{newel|sys|outputStream}}, {{newel|sys|closeStream}}, {{newel|sys|checkpoint}}, {{newel|sys|file:execute}}, and {{newel|sys|file:write}}. In addition, the {{Klib|task}} requires a [[Karajan:Service#SLocal Provider|special local provider]], which can use operating system services to execute local tasks under non-shared privilege, achieved through [[grid-mapfile]] mapping. The service will refuse to run in ''shared'' mode if the normal local provider is detected. Details about building and configuring the secure local provider are available In the Service section. When running in ''shared'' mode the service should be started under a non-priviledged account. '''Please do not run the service from the root account or any other administrative/priviledged account!''' == Limiting Access to Resources in ''Shared'' Mode == Using the service in ''shared'' mode has a number of security requirements: * Access to resources should be restricted based on user identities. * A user must not be able to access any other user’s data/resources. Consequently, this requirement is divided into: *# Restricting access to the shared environment, such that privilege escalation cannot be done in order to override any of the security measures in place *# Delegating certain required privileged operations (such as execution and file access) to local security domains (running privileged operations under specific user accounts) Access restriction is achieved using a ”all-or-nothing” access control list: the [[grid-mapfile]]. A given identity (materialized in a GSI/X509 certificate) is only able to use the service if there exists an entry for that identity in the [[grid-mapfile]]. Shared environment access restrictions is done in two ways:{{ref|1}} # Execution of Karajan elements that could be used for escalating privileges or accessing other users’ data/resources is prohibited. Attempts to use these will result in an error instead. For example, allowing arbitrary Java method invocations inside the shared interpreter environment (the service JVM) is prohibited. Similarly, elements that can be used to access files belonging to the account under which the shared interpreter is running is also prohibited, and their use will result in an error. # Instantiation of arbitrary Java objects, regardless of the means, is restricted to known “safe” object types. A subset of the problem is presented below: The Karajan service allows passing various arguments from the client to the service by means of serialization and de-serialization. However this process does not guarantee the invariance of all objects when copied to a different resource. It is possible, that a certain object exists within the libraries of both the client and the service, which could, through the simple process of de-serialization, be initialized with privileged data, which can later be retrieved through other means. For example, let us consider the following Java class: public class Foo implements Serializable { private File file; private transient String contents; public Foo(File file) { this.file = file; this.contents = ''read contents of file'' } public String toString() { return contents; } } The transfer of an instance of <code>Foo</code> from the client to the service, by means of serialization/de-serialization will not preserve the local meaning of the file attribute. Instead, the mere instantiation of Foo combined with a seemingly unprivileged operation ({{newel|sys|print}}<code>(”foo is {foo}”)</code>) could allow access to arbitrary files in the service shared environment. Another possible scenario is objects whose constructors implement side effects that execute privileged operations. Another <code>Foo</code>, should illustrate this: public class Foo implements Serializable { public Foo() { ''something that amounts to "rm -rf /"'' } } It is clear that de-serializing an instance of the latter <code>Foo</code> in the service environment is undesirable, to say the least. Surely, both <code>Foo</code> versions are somewhat extreme, and somewhat unlikely to exist in the libraries of both the client and the service, but there is no certainty, and auditing all classes in the libraries would be very tedious if at all possible. The chosen solution to the problem is to have a set of allowed classes/packages which can be safely migrated between client and server, while excluding anything that is not explicitly in the set. A configuration file (<code>etc/karajan-restricted-classes.properties</code>) defines the de-serialization policies. The file allows two types of entries: ; package.allow : Define a package which is allowed. All classes in the specified package and all its sub-packages are considered safe to be de-serialized. ; class.disallow : Explicitly disallow a class from being de-serialized, even if it is contained in an allowed package. == Communication Layer Configuration == The communication layer supports three basic modes of operation, described below: ; Persistent : In this mode, connections are kept open for as long as needed. All client&rarr;server and server&rarr;client communications will re-use the same connection, even if multiple requests are sent concurrently from the client. It is also possible to keep connections open after all current communications have terminated, in the event that future communications will be needed. This mode can be safely used if the client is behind a firewall, but it may keep resources allocated unnecessarily if no communication happens between the server and the client. ; Callback : The callback mode can be used to instruct the service to initiate a connection to the client if the service needs to send any messages to the client. This type of configuration will most likely not work if a client is behind a firewall. ; Polling : Polling is somewhat similar to persistent connections in that it will safely work if the client is behind a firewall. It instructs the client to disconnect idle connections, but re-establish them at specific intervals. It is therefore more conservative in terms of resource usage, but it may cause unwanted delays. These modes of operation can be combined. For example, a combination between using callbacks and polling will ensure that the system will work whether the client is behind a firewall or not, but it will provide better performance in the best case scenario (no firewall). The communication layer is conservative, in the sense that before a connection is initiated, it will try to search for any other existing connection that can be used to transmit a certain bit of information, irrespective of the initiator of the connection, and only if such a search fails, will it consider establishing a new connection. The configuration can be changed by editing <code>etc/remote.properties</code>. The file contains a set of pairs of domain expressions and connection properties. The entries are considered in the reverse order from that in which they appear in the file. In other words, the first entry will be considered last, only if no other matching entry can be found. The domain expression is a regular expression which is matched against the domain name of the service host. The connection properties are a set of comma separated options. These options are described in the following table: ; keepalive(timeout) : Indicates that the connection is to be kept alive. The optional timeout indicates that the connection should be also kept alive for the specified number of seconds, even if no actual data is sent through. ; reconnect : This option instructs the system to re-initiate a connection in the event that a connection loss occurs. By default, failed connections will not be re-established. ; callback : Instructs services to connect to the client if no existing connection can be used for sending data. A client will have started a local service before the first connection with the service for which a callback configuration exists is established. ; poll(interval) : Instructs the client to poll the service for updates. The service will buffer all data that it needs to send to the client, and commit the buffered data when the client initiates a polling run. The interval indicates, in seconds, the interval at which the client should initiate the polls. An example configuration is listed below: #default to persistent connections #this one is reached if no other entry matches ".*" keepalive(120), reconnect #callbacks can safely be used within our own domain ".*mydomain.com" callback #there's a firewall between mydomain.com and otherdomain.com, so #we poll every 2 minutes ".*otherdomain.com" poll(120) #for the sake of this example, a combination of callback and polling #if callbacks don't work, polling will pick things up ".*thirddomain.org" callback, poll(120) == The Secure (Grid-Mapped) Local Provider == === Pre-built Packages === Binary packages for the secure local provider are available in the following formats: [[http://www.cogkit.org/release/4_1_4/cog-4_1_4-provider-slocal-bin.tar.gz| .tar.gz]] [[http://www.cogkit.org/release/4_1_4/cog-4_1_4-provider-slocal-bin.zip| .zp]] Simply unpack either one of them in the <tt>cog-4_1_4</tt> directory, overwriting any exiting files. === Building From Source === In order to build the secure local provider, the following steps must be taken: * If a previous build was performed, run <code>ant distclean</code> * Edit the abstractions dependency file in <code>cog/modules/abstraction/dependencies.xml</code> * Change the dependency on the ''provider-local'' to ''provider-slocal'': <ant antfile="${main.buildfile}" target="dep"> <property name="module" value="provider-slocal"/> </ant> * Re-compile with ant dist '''At this time, the secure local provider only implements job execution (execute). File operations are not yet implemented.''' The secure local provider uses a [[gridmap]] file, which describes a maping of GSS distinguished names to local user accounts. For details about gridmap files, take a look at [http://globus.org/toolkit/docs/4.0/admin/docbook/ch05.html#id2516656 Specifying Identity Mapping Information] The actual mapping is done using ''sudo'' and a simple job wrapper. ''Sudo'' must be configured to allow the execution of the wrapper under target user accounts without a password. An example ''sudo'' configuration is shown below: <shared_username> ALL=(<target_usernames_list>) NOPASSWD: <cog_path>/libexec/job-wrapper Where <code><shared_username></code> is the username of the account under which the shared service is running, <code><target_usernames_list></code> is a comma separated list of user-names which the provider can map to (these must also be correctly configured in the [[gridmap]] file), and <code><cog_path></code> is the absolute path to the CoG/Karajan installation. If the target user account is the same as the shared user account, the secure local provider can be configured to bypass ''sudo'' (which it does by default). The provider configuration file (<code>etc/provider-slocal.properties</code>), supports the following properties: ; grid.mapfile : Indicates the location of the gridmap file. The default is <code>/etc/grid-security/grid-mapfile</code>. ; sudo : Points to the location of ''sudo''. ; nosudo : If the target user is the same as the user under which the shared service is running, then the value of this property will be used as the path to a program used to start the wrapper. ; job.wrapper : The location of the wrapper. It defaults to <code><cog_path>/libexec/job-wrapper</code>, therefore it should not be necessary to specify this property unless a non-standard configuration is used. An example configuration file is shown below: grid.mapfile=/etc/grid-security/grid-mapfile sudo=/usr/bin/sudo nosudo=/bin/bash job.wrapper=/home/karajan/cog/libexec/job-wrapper = Embeding Karajan into Java = This document provides details on the two modes through which Karajan scripts can be used from Java code The first possibility is to use the Karajan parser to build an internal representation of the code, and execute it. Example: try { ElementTree tree = Loader.loadFromString("include(\"sys.k\"), print(\"Hello world!\")"); ExecutionContext ec = new ExecutionContext(tree); ec.start(); ec.waitFor(); } catch (Exception e) { e.printStackTrace(); } An alternative, seemingly easier way to do the same is: KarajanWorkflow workflow = new KarajanWorkflow(); workflow.setSpecification("..."); workflow.start(); workflow.waitFor(); if (workflow.isFailed()) { System.err.println("Failed:"); worklfow.getFailure().printStackTrace(); } The other possibility is to build an ElementTree without parsing: try { ElementTree tree = new ElementTree(); Sequential s = new Sequential(); Print p1 = new Print(); p1.setProperty("message", "Hello"); p1.setProperty("nl", Boolean.FALSE); Print p2 = new Print(); p2.setProperty("message", " world!"); s.addElement(p1); s.addElement(p2); tree.setRoot(s); ExecutionContext ec = new ExecutionContext(tree); ec.start(); ec.waitFor(); } catch (Exception e) { e.printStackTrace() } This would be roughly equivalent to: sequential( print(message = "Hello", nl = false) print(message = " world!") ) It is worth noting that it is not necessary to import the {{klib|sys}} because there is no need to resolve names of elements to implementations, since the implementations are specified directly. More details about the mapping of element names to implementations can be found by consulting the actual library definitions found in the [http://svn.sourceforge.net/viewvc/cogkit/trunk/current/src/cog/modules/karajan/resources resource] directry. Listed below are a few common libraries with links to their sources (from which the mapping can be infered): * [http://svn.sourceforge.net/viewvc/cogkit/trunk/current/src/cog/modules/karajan/resources/sys-common.xml?view=markup sys-common.xml] *[http://svn.sourceforge.net/viewvc/cogkit/trunk/current/src/cog/modules/karajan/resources/sys.xml?view=markup sys.xml] *[http://svn.sourceforge.net/viewvc/cogkit/trunk/current/src/cog/modules/karajan/resources/tsk.xml?view=markup task.xml] = Notes = The current implementation will first translate the native syntax to XML, then parse the resulting XML file = References = All CoG Kit references can be found [[http://wiki.mcs.anl.gov/gregor/index.php/Gregor_von_Laszewski#PUBLICATIONS_AND_PRESENTATIONS here]] {| {{bib|las06karajan}} '''Java CoG Kit Workflow''' in to be published, Gregor von Laszewski, Mihael Hategan, and Deepti Kodeboyina, Argonne National Laboratory, Argonne IL, 60430, USA gregor@mcs.anl.gov, 2006. {{bib|las05workflow-jgc}} '''Java CoG Kit Workflow Concepts''', Gregor von Laszewski and Mihael Hategan, in Journal of Grid Computing, http://www.mcs.anl.gov/~gregor/papers/vonLaszewski-workflow-jgc.pdf. Electronic Publication Link, Jan. 2006. http://dx.doi.org/10.1007/s10723-005-9013-5 {{------}} |}