Java CoG Kit SRB Integration

From Java CoG Kit

Description

The goal is to bridge the gap between computational and data grids by integrating GT4 with the Storage Resource Broker (SRB). This project could compliment the existing GridFTP SRB interface by Bill Allcock.

We aim to create clones for: globus-url-copy, RFT, and perhaps support for meta data queries using the MDS.

[edit]

Phase I

Create a clone of the GlobusUrlCopy class: org.globus.cog.srb.GlobusUrlCopy with support for SRB URIs to enable transfers to/from SRB and GSIFtp servers.

Implementation

Implementation turned out to be pretty darn easy. SRB already supports URI input streams and GSI. Two new Classes have beed created, and two old ones have minor changes

Support for GSI

SRB can be compiled with GSI support, and thus the URI protocol has been divided in srb:// and gsisrb://, analog to ftp:// and gsiftp://

New Classes

org.globus.cog.io.streams.SRBInputStream: Provides a Globus input stream to srb.
org.globus.cog.io.streams.SRBOutputStream: Provides a Globus output stream to srb.

Classes with minor changes

GlobusRulCopy: Added changes to the help text
UrlCopy: Added support for the SRB input streams

New Libraries

SRB Jargon Java API: http://www.sdsc.edu/srb/jargon/index.html

Here are some example calls

To transfer the file /tmp/gtk24.zip from a GSIFTP server to a vanilla SRB under user vsilva, pwd secret:
- globus-url-copy -debug gsiftp://vm-rhl9:2811//tmp/gtk24.zip srb://vsilva.dev:secret@192.168.64.128:5544//A/home/vsilva.dev/gtk24.zip

To transfer the file /tmp/gtk24.zip from a GSIFTP server to a GSI enabled SRB under user vsilva
- globus-url-copy -debug gsiftp://vm-rhl9:2811//tmp/gtk24.zip gsisrb://192.168.64.128:5544//A/home/vsilva.dev/gtk24.zip

[edit]

Phase II

Changes to the RFT service to support SRB URIs have been scrubbed.

[edit]

Phase III

WS Service to query a GSI Enabled SRB Server

This service is loosely based on the counter sample and allows a client to query a GSI enabled srb server thru a GT4 container.

Sample Calls

To create and end point reference (EPR) to the WS service on a given container.

ws-srb-create -s https://134.67.66.53:8443/wsrf/services/SRBQueryService

The result would be something like:

<ns1:QueryReference xsi:type="ns2:EndpointReferenceType" xmlns:ns1="http://srb.c
om" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:ns2="http://sche
mas.xmlsoap.org/ws/2004/03/addressing">
 <ns2:Address xsi:type="ns2:AttributedURI">https://134.67.66.53:8443/wsrf/servic
es/SRBQueryService</ns2:Address>
 <ns2:ReferenceProperties xsi:type="ns2:ReferencePropertiesType">
  <ns1:QueryKey>21539046</ns1:QueryKey>
 </ns2:ReferenceProperties>
 <ns2:ReferenceParameters xsi:type="ns2:ReferenceParametersType"/>
</ns1:QueryReference>

To retrieve files that belong to user vsilva that are larger than 10 bytes (Using the EPR created on step 1).

ws-srb-query -e srb.epr "user name = vsilva, size > 10"

To query user defined metadata files for project Foo whose owner has the string silva:

ws-srb-query -e srb.epr "Project Name = Foo, user like silva"

Output

Output is returned in XML format of the form:

<MetaDataRecordList len="9">
        <Record rank="0" fields="7">
                <Field type="3">
                        <Name>file name</Name>
                        <Desc>data name</Desc>
                        <Value>foo.txt</Value>
                </Field>
                <Field type="3">
                        <Name>user group name</Name>
                        <Desc>name of user group</Desc>
                        <Value>vsilva</Value>
                </Field>
                <Field type="3">
                        <Name>user name</Name>
                        <Desc>user name</Desc>
                        <Value>vsilva</Value>
                </Field>
                <Field type="3">
                        <Name>user domain</Name>
                        <Desc>user domain name</Desc>
                        <Value>NCC</Value>
                </Field>
                <Field type="0">
                        <Name>size</Name>
                        <Desc>size of data</Desc>
                        <Value>10</Value>
                </Field>
                <Field type="3">
                        <Name>file comments</Name>
                        <Desc>comments on data</Desc>
                        <Value></Value>
                </Field>
                <Field type="3">
                        <Name>owner domain</Name>
                        <Desc>domain of data creator</Desc>
                        <Value>NCC</Value>
                </Field>
        </Record>
        ...
</MetaDataRecordList>

Boolean Expression Notes

The SRB Java API (Jargon) does not seem support complex boolean queries (e.g OR, NOT, etc).

[edit]

Contact

Vladimir Silva

[edit]

Status

Phase I is complete and ready for test. Code changes have been submitted for review.
Phase II has been scrubbed
Phase III is being tested

[edit]

Package Name

We propose the following package names for the project

org.globus.cog.srb
org.globus.cog.rft
org.globus.cog.mds

globus-url-copy exists for rft as a c client in the trunk:

Along with globus-data-management bundle, you should be building ws-transfer bundle to get -rft switch with g-u-c

[edit]

License

apache jargon license

[edit]

Download

[edit]

Integration in CoG Kit

[edit]

Refences

SRB 100% Java client API called Jargon: http://www.sdsc.edu/srb/jargon/index.html
- License: http://www.sdsc.edu/srb/jargon/LICENSE.txt
SDSC Storage Resourrce Broker: http://www.sdsc.edu/DICE/SRB/
SRB OGCE portlets (expected availability Feb 1, 2006)

[edit]

SRB COG Kit Integration

Vladimir Silva and Gregor von Laszewski

We aim to create modified versions of:

globus-url-copy
RFT, and
support for meta data queries using WSRF.

[edit]

Support for GSI

SRB can be compiled with GSI support, and thus the URI protocol has been divided in

srb://
gsisrb://

which is analog to ftp:// and gsiftp://

We have added the following new classes to the CoG Kit

org.globus.cog.io.streams.SRBInputStream: Provides a Globus input stream to multiple protocols: ftp, http, etc.
org.globus.cog.io.streams.SRBOutputStream: Provides a Globus output stream to multiple protocols: ftp, http, etc.

In addition we have modified the following classes and programs slightly to reflect tha ability to integrate with SRB

GlobusUrlCopy: Changes to the help text
UrlCopy: Changed the methods get(In/Out)putStream methods to include support for the new SRB input streams

[edit]

New JARS

We need the following additional Jar files:

Jargon, http://www.sdsc.edu/srb/jargon/

[edit]

Setup

Set COG_INSTALL_PATH: export COG_INSTALL_PATH={PATH2COG}
Add the bin directory and Jargon JARS to your classpath:

export CLASSPATH=[PATH2JARGON]:[PATH2BINFOLDER]

[edit]

Examples

To transfer the file /tmp/gtk24.zip from a GSIFTP server to a vanilla SRB under user silva, pwd secret:

globus-url-copy -debug
 gsiftp://vm-rhl9:2811//tmp/gtk24.zip
 srb://vsilva.dev:secret@192.168.64.128:5544//home/silva/gtk24.zip

To transfer the file /tmp/gtk24.zip from a GSIFTP server to a GSI enabled SRB under user silva

globus-url-copy -debug
  gsiftp://vm-rhl9:2811//tmp/gtk24.zip 
    gsisrb://192.168.64.128:5544//home/silva/gtk24.zip

To trasnfer the local file c:\tmp\epr.txt to the GSI FT server vm-rhl9 under the directory /tmp/testfile.txt

 globus-url-copy -debug file:///c:/tmp/epr.txt 
    gsiftp://vm-rhl9:2811//tmp/testfile.txt

To trasnfer the local file c:\tmp\epr.txt to a SRB serverunder user vsilva, password secret

 globus-url-copy -debug file:///c:/tmp/epr.txt     
    srb://vsilva:secret@192.168.64.128:5544//home/vsilva/testfile.txt

To trasnfer the file gtk24-dev.zip from the SRB home collection vsilva to an GSI FTP server

 globus-url-copy -debug srb://vsilva:secret@192.168.64.128:5544//A/home/vsilva/gtk24-dev.zip  
    gsiftp://vm-rhl9:2811//tmp/gtk24.zip

[edit]

Troubleshooting

[edit]

Invalid passwords

Invalid passwords throw this execption:

GlobusUrlCopy error: UrlCopy transfer failed. [Root error message: read() -- couldn't read complete packet] org.globus.io.urlcopy.UrlCopyException: UrlCopy transfer failed. [Root error message: read() -- couldn't read complete packet]. Root exception is java.io.IOException: read() -- couldn't read complete packet

at edu.sdsc.grid.io.srb.SRBCommands.read(SRBCommands.java:1002)
at edu.sdsc.grid.io.srb.SRBCommands.commandStatus(SRBCommands.java:1430)
at edu.sdsc.grid.io.srb.SRBCommands.srbGetSvrVersion(SRBCommands.java:7777)

[edit]

Incomplete host names

A URL with an incomplete host name may throw the error Invalid zone name:

To: srb://srbAdmin.NCC:secret@ebony:5544//nccZone/home/srbAdmin.NCC/foo.txt
Protocol    : srb
Host name   : ebony
Port number : 5544
Url path    : /nccZone/home/srbAdmin.NCC/foo.txt
User        : srbAdmin.NCC
Pwd         : secret

UrlCopy transfer failed. [Root error message: Invalid zone name.] org.globus.io.urlcopy.UrlCopyException: UrlCopy transfer failed. [Root error message: Invalid zone name.]. Root exception is java.net.ConnectException: Invalid zone name.

at edu.sdsc.grid.io.srb.SRBFile.uriInitialAccount(SRBFile.java:590)
at edu.sdsc.grid.io.srb.SRBFile.<init>(SRBFile.java:473)

Retrieved from "http://wiki.cogkit.org/index.php/Java_CoG_Kit_SRB_Integration"

Java CoG Kit SRB Integration

From Java CoG Kit

Description

Phase I

Phase II

Phase III

Contact

Status

Package Name

License

Download

Integration in CoG Kit

Refences

SRB COG Kit Integration

Support for GSI

New JARS

Setup

Examples

Troubleshooting

Invalid passwords

Incomplete host names

Views

Personal tools

SC06

Collaboration and Jobs

CoG Kits

Java CoG Kit

Community

Wiki

Search

Toolbox