Gerald Guo CGL Report

Hadoop Log

2010-12-26T18:01:00.001-05:00

When Hadoop is started, it sets hadoop.log.dir using -Dhadoop.log.dir=$HADOOP_LOG_DIR.
If you don't set environment variable HADOOP_LOG_DIR explicitly, it will be $HADOOP_HOME/logs. If you don't specify HADOOP_HOME, Hadoop will guess it by using path of the script that you use to start Hadoop. So if you install Hadoop to dir <hadoop_dir>, and HADOOP_LOG_DIR is not set, then the log dir is <hadoop_dir>/logs.

If you want to change root log dir, change file 'conf/hadoop-env.sh'. Add a line similar to

export HADOOP_LOG_DIR=/your/local/log/dir

In following table, you should replace those variables which are enclosed in angle brackets.
<jobid>: id of a job
<username>: username of the user who starts up Hadoop.
<host>: host name of the node which runs the process.

Direcotory	Description	Related config parameters
<hadoop.log.dir>	Log of various daemons
hadoop-<username>-jobtracker-<host>.log	Log of jobtracker daemon
hadoop-<username>-namenode-<host>.log	Log of namenode daemon
hadoop-<username>-secondarynamenode-<host>.log	Log of secondarynamenode daemon
hadoop-<username>-tasktracker-<host>.log	Log of tasktracker daemon
hadoop-<username>-datanode-<host>.log	Log of datanode daemon
job_<jobid>_conf.xml	Configuration file of a job	Only exists when the job is running.

<hadoop.log.dir>/history		mapreduce.jobtracker.jobhistory.location
job_<jobid>_conf.xml		Only exists when the job is running.
job_<jobid>_<username>		Only exists when the job is running.

<hadoop.log.dir>/done	log of completed jobs	mapreduce.jobtracker.jobhistory.completed.location
job_<jobid>_<username>	Event logging. It includes all events of the job (e.g. job started, task started).
job_<jobid>_conf.xml	Job conf file. It includes all configurations of the job.

<hadoop.log.dir>/userlogs	Log of attempts. Stored on each task tracker.
job_<jobid>	Each directory contains log of all attempts of the job.

/jobtracker/jobsInfo (in HDFS)	Job Status Store	mapreduce.jobtracker.persist.jobstatus.active mapreduce.jobtracker.persist.jobstatus.hours mapreduce.jobtracker.persist.jobstatus.dir
<jobId>.info	job status of a job

Job logs in <hadoop.log.dir>/history/done directory are kept for mapreduce.jobtracker.jobhistory.maxage. Default value is 1 week.

Schedulers in supercomputers

2010-11-29T11:57:00.000-05:00

LoadLeveler

http://www.ccs.ornl.gov/eagle/LL.html
http://www.bu.edu/tech/research/training/scv-software-packages/loadleveler/
http://kb.iu.edu/data/azvs.html


Cluster status	llstatus
List jobs in the queues/classes	llq
List information of classes/queues	llclass
Reservation check	llqres	This may or may not work. Probably you should command showres

PBS

pbsnodes

Moab

http://www.clusterresources.com/products/mwm/docs/a.gcommandoverview.shtml
http://www.clusterresources.com/products/maui/docs/16.1simulationoverview.shtml

mshow -a # show available resources

IOMeter

2010-11-27T15:41:00.000-05:00

Document: http://iometer.cvs.sourceforge.net/viewvc/iometer/iometer/Docs/Iometer.pdf

Windows: install IOMeter (it includes both dynamo and iometer GUI)

Linux: the linux package only includes dynamo (the runtime engine). It does not include GUI. So you need to connect dynamo to the Windows box on which IOMeter is installed. Use following command to start dynamo

sudo ./dynamo -i iometer_computer_host -n manager_name -m manager_computer_host

Parameters:
iometer_computer_host: host/ip where IOMeter GUI is run (Windows box)
manager_computer_host: host/ip where dynamo is run
manager_name: name of the dynamo manager (can be arbitrary. used to distinguish different dynamo managers connecting to the same IOMeter server)

Several options that are confusing

Disk Targets > # of Outstanding I/Os
"specifies the maximum number of outstanding asynchronous I/O operations per disk the selected worker(s) will attempt to have active at one time."
"This control can be overridden by the # of Outstanding I/Os control group in the Test
Setup tab (depending on the Cycling Options)."

Test Setup > Cycling Options
"The Cycling Options control group specifies the series of tests that is run for each access specification."

Number of workers per manager
Workers are added in the order they are shown in the Topology panel
Number of targets per worker
selected targets are added to each worker in the order they are shown in the Disk Targets or Network Targets tab for that worker.
Number of outstanding I/O operations per target (disk workers only).

Cycling Options

Normal - run all selected targets for all workers.
In all managers, all works are active and each work uses all its selected targets. "The number of outstanding I/Os per disk is specified by the # of Outstanding I/Os field in the Disk Targets tab."
Cycle Workers -- add step workers using all selected targets at a time
"This test type increases the number of workers for each manager in each test.". In each test, the first N (depending on iteration and step) workers listed in Topology panel are used.
Cycle Targets -- add step targets for all workers at a time.
This test type increases the number of targets for each worker in each test.

Increment Targets Parallel -- add step targets to all managers at a time.

Test #	Manager 1		Manager 2		Targets per Manager
Test #	Worker 1	Worker 2	Worker 1	Worker 2	Targets per Manager
1	1 target	0 targets	1 target	0 targets	1 target
2	2 targets	0 targets	2 targets	0 targets	2 targets
3	2 targets	1 target	2 targets	1 target	3 targets
4	2 targets	2 targets	2 targets	2 targets	4 targets

Increment Targets Serial -- add step targets at a time.

Test #	Manager 1		Manager 2		Total Targets
Test #	Worker 1	Worker 2	Worker 1	Worker 2
1	1 target	0 targets	0 targets	0 targets	1 target
2	2 targets	0 targets	0 targets	0 targets	2 targets
3	2 targets	1 targets	0 targets	0 targets	3 targets
4	2 targets	2 targets	0 targets	0 targets	4 targets
5	2 targets	2 targets	1 target	0 targets	5 targets
6	2 targets	2 targets	2 targets	0 targets	6 targets
7	2 targets	2 targets	2 targets	1 target	7 targets
8	2 targets	2 targets	2 targets	2 targets	8 targets

Cycle Targets and Workers -- add step targets at a time spread across workers

Test #	Worker 1	Worker 2	Worker 3	Targets per Manager
1	1 target	0 targets	0 targets	1 target
2	2 targets	0 targets	0 targets	2 targets
3	1 target	1 target	0 targets	2 targets
4	2 targets	2 targets	0 targets	4 targets
5	1 target	1 target	1 target	3 targets
6	2 targets	2 targets	2 targets	6 targets

Cycle # Outstanding I/Os -- run step outstanding I/Os on all disks at a time.
Cycle # Outstanding I/Os and Targets -- run step outstanding I/Os on step targets at a time

Not all managers have the same number of works. Not all works have the same number of targets. Read following paragraph to see how IOMeter handles it.

"The ending value for each sequence is determined differently for each quantity. For the number of workers per manager, it is determined by the manager with the most workers in the Topology panel. For the number of targets per worker, it is determined by the worker with the most targets selected in the Disk Targets or Network Targets tab. For the outstanding I/Os per target, it is determined by the # of Outstanding I/Os control group in the Test Setup tab.
If not all managers have the same number of workers, or not all workers have the same number of targets, those managers/workers with less than the maximum number of workers/targets will repeat at their highest value while other managers/workers continue to increase. "

In Disk Targets Tab, "A yellow icon with a red slash through it means that the drive needs to be prepared before the test starts".

"Yellow disk icons represent logical drives (i.e. those with a drive letter). They are shown only if they are writable. Disk workers access logical drives by reading and writing a file called iobw.tst in the root directory of the drive. If this file exists, the drive is shown with a plain yellow icon; if the file does not exist, the drive is shown with a red slash through the icon. (If this file exists but is not writable, the drive is considered read-only and is not shown at all.)
If you select a drive that does not have an iobw.tst file, Iometer will begin the test by creating this file and expanding it until the drive is full (this is shown as “Preparing Drives” in the status bar). You can change the size of this file to control how much of the disk Iometer can use by setting the Maximum Disk Size control to the desired size (in sectors).
Blue disk icons represent physical drives. They are shown only if they contain nothing but free space (no defined partitions). Disk workers access physical drives by writing direct to the raw disk. Physical drives do not use an iobw.tst file. Running with physical drives is recommended."

JsUnit Maven Plugin

2010-11-12T14:27:00.000-05:00

Document is at http://jsunit.berlios.de/maven2.html. It's too brief, especially following paragraph:

The type of the test suite, one of the following values:

ALLTESTS
Looks for a class AllTests derived from TestSuite and runs its suite.
TESTSUITES
Looks for all classes ending with TestSuite and that are derived from TestSuite and run their suites.
TESTCASES
Looks for all classes ending with TestCase and that are derived from TestCase and runs them (the default).

The problem is what "derived from" means and how to to that. I will show in detail how to use JsUnit plugin.

1) sample test file

Following is a dummy test file. It should be put into src/test/js.

var dummyobj = dummyobj || {};

function DummyTest(name) {
  TestCase.call(this, name);
};

DummyTest.inherits(TestCase);

DummyTest.prototype.setUp = function() {
    dummyobj.name = "Gerald";
};

DummyTest.prototype.tearDown = function() {
    delete dummyobj.name;
};

DummyTest.prototype.testDummy = function() {
  this.assertEquals('Gerald', dummyobj.name);
};

DummyTest is the test case.
It "inherits" from class TestCase. All of its functions whose name start with "test" will be tested.

2) Inherit implementation

Following is implementation of inherit borrowedfrom Shindig code. It can be put in a file inherit_implementation.js under directory src/main/js.

Function.prototype.inherits = function(parentCtor) {
    function tempCtor() {};
    tempCtor.prototype = parentCtor.prototype;
    this.superClass_ = parentCtor.prototype;
    this.prototype = new tempCtor();
    this.prototype.constructor = this;
};

3) Pom.xml

<plugin>
    <groupId>de.berlios.jsunit</groupId>
    <artifactId>jsunit-maven2-plugin</artifactId>
    <executions>
        <execution>
            <id>test</id>
            <configuration>
                <sourceDirectory>${basedir}/src/main/js</sourceDirectory>
                <sources>
                    <source>inherit_implementation.js</source> 
                    <source>file_to_be_tested.js</source>
                </sources>
                <testSourceDirectory>${basedir}/src/test/js</testSourceDirectory>
                <testSuites>
                    <testSuite>
                        <name>SampleSuite</name>
                        <type>TESTCASES</type>
                        <includes>
                            <include>*/*test.js</include>
                        </includes>
                    </testSuite>
                </testSuites>
            </configuration>
            <goals>
                <goal>jsunit-test</goal>
            </goals>
        </execution>
    </executions>
</plugin>

Resources

For some tests, you need to provide fake window object, XmlHttpRequest object, DOM objects, etc. Project env-js (site) is exactly designed for this purpose. It provides a simulated browser environment.

How to run/debug Shindig in Eclipse

2010-11-08T13:56:00.000-05:00

Instructions here (http://shindig.apache.org/developers/java/build.html#Setting_up_an_Eclipse_project_to_build_Apache_Shindig) are for old versions of Eclipse.

I am using Eclipse Helios. You should install Maven plugin m2eclipse before following following instructions.

Download or check out shindig code.
File –> Import
Maven –> Existing Maven Projects
Specify the root directory of shindig code.
Right click top level imported project (*-project): Debug As –> Debug Configuration
Create a new configuration for “Maven Build”.
Base directory: root directory of shindig code
Profiles: run
Unselect “Skip Tests”.
If you want to use a port number rather than 8080, add a parameter “jetty.port”.
Clicek “Debug”.
Jetty server should run successfully. Look at your console for possible error messages.

Add breakpoints, then send a request to Jetty server. Eclipse complains that it cannot find source code for the debugged app, and prompt you to add source lookup directories.

OAuth in OGCE gadget container

2010-09-15T23:34:00.001-04:00

OGCE gadget container makes support of OAuth as easy as possible.

Follow two files are the most important related to OAuth configuration.

config/shindig/shindig.properties
Most important config parameter is shindig.signing.global-callback-url. It has been rewritten correctly so the end users don’t need to worry about it.
config/shindig/oauth.json
This files contains information of all consumers of which the OGCE gadget container is a proxy.

If you want to write OAuth gadget, follow instructions showed on this page: http://code.google.com/apis/gadgets/docs/oauth.html.
More complicated cases: http://sites.google.com/site/oauthgoog/oauth-proxy/social-oauthproxy

Tomcat SSL configuration. Import certificate chain to keystore (emit error “keytool error: java.lang.Exception: Input not an X.509 certificate”)

2010-01-06T19:07:00.002-05:00

I seldom use mutual authentication in the context of SSL. Recently, in our derived project (integration with MyOSG), we need to enforce use of mutual SSL authentication.

Enable client authentication in Tomcat server

At first, I enabled it in tomcat server configuration file

An important option is "clientAuth”.
Note: before this step, I have generated and imported a certificate for tomcat server.

Import user certificate

This is done by the end users who wish to access the protected services. They need to

import received certificate from CA to their browser
Actually, both the private key and certificate need to be imported.
For Firefox, only pkcs#12 format is supported. If the private key has been imported, you just need to import certificate whose format can be PEM, binary, etc.
import server’s certificate into trusted ca repository in browser
The aim is to make the browser trust the certificate received from remote service. It’s useful when the service certificate is not issued by a well-known top-level CA.

After those two steps, I directed my browser to the service url. Unfortunately, I got the following error:

After digging a little bit, I found out the cause was that tomcat server does not trust certificate sent by my browser. So the solution is simple: add certificate chain related to my certificate to tomcat keystore.

I got the certificate chain from the issuer of my certificate. It’s in PKCS#7 format and contains two certificates in the file. You can view PKCS7-formatted file using following commands:
1) openssl pkcs7 -print_certs -text < pkcs7_cert_chain.pem
2) keytool -printcert –file pkcs7_cert_chain.pem

When I tried to import it into keystore using following command
    keytool -importcert -file pkcs7_cert_chain.pem -keystore keystore -alias test-cert –trustcacerts
I got the following error
    keytool error: java.lang.Exception: Input not an X.509 certificate
I am sure keytool can recognize the file because the following command prints out information in the file correctly.
    keytool -printcert –file pkcs7_cert_chain.pem

Solution

After reading keytool manual carefully, I found following statements:

Importing a New Trusted Certificate

    Before adding the certificate to the keystore, keytool tries to verify it by attempting to construct
    a chain of trust from that certificate to a self-signed certificate (belonging to a root CA), using
    trusted certificates that are already available in the keystore.

Importing a Certificate Reply

    ……

      o If the reply is a PKCS#7 formatted certificate chain, the chain is first ordered (with the user
        certificate first and the self-signed root CA certificate last), before keytool attempts to
        match the root CA certificate provided in the reply with any of the trusted certificates in the
        keystore or the "cacerts" keystore file (if the -trustcacerts option was specified). If no match
        can be found, the information of the root CA certificate is printed out, and the user is
        prompted to verify it, e.g., by comparing the displayed certificate fingerprints with the fin-
        gerprints obtained from some other (trusted) source of information, which might be the root CA
        itself. The user then has the option of aborting the import operation. If the -noprompt option
        is given, however, there will be no interaction with the user.

So, what I was doing is to import a trusted certificate chain. This is not allowed directly. keytool just accepts cert file that includes a single certificate.

So I extracted two certificates into two files, and fed them into keytool one by one. Details:
Use command
openssl pkcs7 -print_certs < pkcs7_cert_chain.pem
to display the two certificates in the original pkcs7 file. And then copied and pasted each cert into an individual file.

Note: when you are importing a certificate reply from a CA, certificate chain can be imported directly into keystore. However, before doing that, you must guarantee that the corresponding private key has already been imported in to the same keystore.

Resources about OAuth support in Gadget

2009-11-13T13:07:00.000-05:00

http://sites.google.com/site/oauthgoog/oauth-proxy

http://code.google.com/apis/gadgets/docs/oauth.html

http://code.google.com/apis/accounts/docs/OAuth.html

http://groups.google.com/group/oauth/browse_thread/thread/5dea93b44dbbb628

OAuth test client: http://term.ie/oauth/example/client.php

Google OAuth Playground: http://googlecodesamples.com/oauth_playground/ (just for Google’s APIs)

A bug in Shindig code (for revision 832845)

2009-11-05T00:00:00.001-05:00

Recently, I tried to deploy cyberaide filetransfer gadget (http://lily01.rit.edu/cyberaideFiletransfer.xml) and jobman gadget (http://lily01.rit.edu/cyberaideJobman.xml). They did NOT work in shindig server built from revision 832845. Those two gadgets use ‘tab’ feature which is not implemented well in shindig code.

Problem description
See source code of tab feature at http://svn.apache.org/repos/asf/incubator/shindig/trunk/features/src/main/javascript/features/tabs/tabs.js?p=832845.
In this file, function ‘html_sanitize’ is used. It is a function from caja javascript library.
Shindig website does not provide much useful information about integration of caja. http://incubator.apache.org/shindig/developers/java/build.html#Running_with_Caja
I found a useful wiki page in caja project: http://code.google.com/p/google-caja/wiki/GettingStarted.
Basically the user needs to add following text to ModulePrefs element of the gadget xml file
<Require feature="caja"/>
After I changed the gadget xml file, it still didn’t work. It seems that after caja library sanitizes the rendered html/javascript code, some useful code is removed.

My solution
To make it simple, I disabled caja. I just removed invocations of functions belonging to caja library. Concretely, I removed invocations of function ‘html_sanitize’ in file ‘tab.js’

Also, feature ‘minimessage’ also uses caja functions to sanitize html/javascript. If your gadgets use that feature, corresponding changes must be made as well.

X connection to localhost:xx.x broken (explicit kill or server shutdown)

2009-11-02T16:48:00.000-05:00

When I run tomcat server, I got this weird problem. It seems that tomcat tries to access GUI, which is contrary to my thought that tomcat is a background daemon.

A simple google search brought me lots of posts about this problem: http://www.google.com/search?hl=en&client=firefox-a&rls=org.mozilla%3Aen-US%3Aofficial&hs=ugP&q=tomcat+connection+to+localhost%3A+broken+%28explicit+kill+or+server+shutdown%29&aq=f&oq=&aqi=

According to suggestion from this post http://bugs.mysql.com/bug.php?id=24226, I just unset environment variable *DISPLAY* in *startup.sh*.
In bash, use *unset DISPLAY*

Enable and disable Shindig caching

2009-11-02T16:39:00.000-05:00

Shindig implements a complex caching hierarchy including referred external resources, locale messages, gadget specification files, etc.
During development phase, the developers want to disable caching to make modifications take effect immediately.

Build-Time Solution
It is according to this post https://studio.atlassian.com/browse/AG-145.
Download source code tree and build the war file using command “mvn –Pdebug package”
Run-time solution
HttpCache: caches http request sent out from shindig server. During gadget rendering, this is controlled by “nocache” parameter. So when you render a gadget, add parameter ‘nocache’ with value 1.
I don’t know how to do it at server side.

Transition from _IG_* APIs to gadgets.* APIs

2009-10-31T23:58:00.000-04:00

iGoogle announced that legacy gadget APIs (_IG_*) are deprecated:
http://igoogledeveloper.blogspot.com/2009/09/more-things-change-more-they-stay-same.html
http://groups.google.com/group/Google-Gadgets-API/browse_thread/thread/7f1aa77dcf90064c

Migration guide:
http://code.google.com/apis/gadgets/docs/migration.html

It would be great if there is a migration tool that automate the process.

Trial of Apache Sling

2009-09-24T17:02:00.001-04:00

Introduction

Sling is based on Felix which is an implementation of OGSi. It includes Felix webconsole (http://felix.apache.org/site/apache-felix-web-console.html) bundle to make it easy to inspect OSGi framework. Also Sling integrates Jackrabbit and wraps it as an OSGi bundle. To update Java Content Repository, the client app can send regular HTTP POST requests. In other words, it exposes Java Content Repository in a REST way.
It can be run either independently or in a servlet container (e.g. Apache Tomcat).

Resources

Sling: http://sling.apache.org/site/index.html
Useful Sling help: http://felix.apache.org/site/apache-felix-framework-usage-documentation.html
Install and upgrade bundles in Felix web console: http://sling.apache.org/site/installing-and-upgrading-bundles.html
Address of Felix Management Web Console: http://ip_address:port/system/console/

When I clicked “Configuration” link on the top, Null Pointer Exception was thrown out.
It turns out that this is problem of Felix webconsole. See
http://issues.apache.org/jira/browse/FELIX-1135
http://issues.apache.org/jira/browse/FELIX-1028

This post contains some useful information http://groups.google.com/group/sakai-kernel/web/building-3akai-sling.

Solution

Download sling (standalone distribution) and unzip the tarball:
http://mirror.cc.columbia.edu/pub/software/apache/sling/org.apache.sling.launchpad.app-5-incubator-bin.tar.gz
Download new version of felix webconsole:
https://issues.apache.org/jira/secure/attachment/12407768/org.apache.felix.webconsole-1.2.9-SNAPSHOT.jar
Assume the new version of webconsole is downloaded to dir <DIR>

Execute following commands in the directory where sling tarball is unzipped.

jar xf org.apache.sling.launchpad.app-5-incubator.jar 
cd resources/bundles/5/ 
rm org.apache.felix.webconsole-1.2.8.jar 
cp <dir>/org.apache.felix.webconsole-1.2.9-SNAPSHOT.jar ./ 
jar cMf org.apache.sling.launchpad.app-5-incubator.jar resources META-INF org

Run the server:

java -jar org.apache.felix.webconsole-1.2.8.jar -p 4040

Note: you CANNOT use arbitrary new version of felix webconsole without updating other components of felix package. The reason is that different versions of felix console requires different versions of felix core/OSGi.

Misc.

If you are using both JCR Installer and Felix web console to update a bundle, you will get into trouble.
JCR installer:
    http://sling.apache.org/site/jcr-installer-jcrjcrinstall-and-osgiinstaller.html
    JCR installer tries to install OSGi bundles/modules found in Java Content Repository.
Bug
   “JCR Install prevents update of bundle through other channels like the web console”
    http://issues.apache.org/jira/browse/SLING-1106

Java Conent Repository - Jackrabbit

2009-09-23T23:37:00.000-04:00

Resources

Wiki: http://en.wikipedia.org/wiki/Content_repository_API_for_Java
JSR 170: http://jcp.org/en/jsr/detail?id=170
JSR 170 API: http://www.day.com/maven/jsr170/javadocs/jcr-1.0/index.html
JSR 283 (JCR 2.0): http://jcp.org/en/jsr/detail?id=283 (in progress)
Jackrabbit: http://jackrabbit.apache.org/
List of Jackrabbit components: http://jackrabbit.apache.org/jackrabbit-components.html

Introduction

Jackrabbit implements JSR 170 and JSR 283.

Wrap JCR to make it a WebDAV server so that WebDAC client can access it.
Actually it is a fully featured WebDAV server backed by the JCR repository.
Address for default repository: http://your_ip:your_port/repository/default/
It seems that this method only exposes the file and directories in JCR. If you want more fine-grained access, use JCR remoting instead.
Also see: http://jackrabbit.apache.org/standalone-server.html#StandaloneServer-WebDAVaccess
http://jackrabbit.apache.org/jackrabbit-web-application.html
JCR remoting
Provides more fine-grained content access.
Address for all workspaces of JCR repository: http://your_ip:your_port/server
Also see: http://jackrabbit.apache.org/jackrabbit-web-application.html
RMI access
http://jackrabbit.apache.org/standalone-server.html#StandaloneServer-RMIaccess
local access
Through JNDI or servlet context.

Installation problem

When I tried to install Jackrabbit on gridfarm machine which uses NFS V3, I got problem described in this bug report (http://issues.apache.org/jira/browse/JCR-1605). Basically it says Jackrabbit needs some features that are not implemented in NFS prior to V4.

Dynamic-height in Shindig

2009-08-28T14:27:00.001-04:00

Gadget specification provides a feature called dynamic-height to dynamically adjust height of a gadget.
If gadget server and gadget container are in the same domain, this can be done using javascript.
However, if gadget server and gadget container are in different domains, this can not be achieved by directly using Javascript and DOM because of same-origin policy imposed by modern browsers.
This email http://markmail.org/message/53r5dwhr56lby733#query:dynamic%20height%20shindig+page:1+mid:53r5dwhr56lby733+state:results includes some information about how to make it work.
I am going to give more details here.

Gadget container

Download
    http://svn.apache.org/repos/asf/incubator/shindig/trunk/javascript/container/cookies.js
    http://svn.apache.org/repos/asf/incubator/shindig/trunk/javascript/container/gadgets.js
    http://svn.apache.org/repos/asf/incubator/shindig/trunk/javascript/container/util.js
    http://your_shindig_deploy/gadgets/js/rpc.js?c=1&debug=1 (//this file actually includes rpc framework and all possible communication mechanism, e.g. ifpc, wpm...)
And put them into your gadget container server.

Include following code in your container web page to make them point to those javascript files you downloaded in step 1).

<script type="text/javascript" src="rpc.js"></script>
<script type="text/javascript" src="cookies.js"></script>
<script type="text/javascript" src="util.js"></script>
<script type="text/javascript" src="gadgets.js"></script>

Download
rpc_relay.html : http://svn.apache.org/repos/asf/incubator/shindig/trunk/javascript/container/rpc_relay.html
or
rpc_relay_uncompressed.html: http://svn.apache.org/repos/asf/incubator/shindig/trunk/javascript/container/rpc_relay.uncompressed.html

Gadget server/renderer

Check out Shindig source code. See http://incubator.apache.org/shindig/source-repository.html

Change source code of file ifpc.transport.js

Original:
// The RPC mechanism supports two formats for IFPC (legacy and current).
var src = null;
if (rpc.l) {

New

// The RPC mechanism supports two formats for IFPC (legacy and current).
var src = null;
if (false) {

This disables use of legacy data format which is buggy.

Change configuration in file config/container.js
    Original: "parentRelayUrl" : "/gadgets/files/container/rpc_relay.html"
    New:       "parentRelayUrl" : "rpc_relay file address on your gadget container" //it is download in this step
Also you may want to use ifpc instead of browser-specific communication mechanisms.
    Original: "useLegacyProtocol" : false    //gadget tries browser-specific communication mechanisms first. If they fail, fall back to ifpc.
    New:      "useLegacyProtocol" : true           //use ifpc directly.
Note: the current wire format will be used instead of legacy format because we have disabled legacy format.
This option is not used in the way described by comment in the source code. The shindig developers want users to use this option to control whether legacy format is used. I use it to configure whether ifpc is used. If ifpc is used, only current wire format is utilized.
This can also be set after you deploy shindig. The configuration file is located at WEB-INF/classes/containers/default/.
Rebuild shindig.
Copy generated .war file in directory server/target to your servlet container.

Resources

Detailed discussion: http://docs.google.com/View?id=dmxthpg_125ckh55fff
Cross-Domain Communication with IFrames: http://softwareas.com/cross-domain-communication-with-iframes

Google GData API usage

2009-08-21T00:39:00.001-04:00

Recently, I am using Google GData APIs to access Google Picasa and Google Calendar Services.
There is a Java library which is hosted at http://code.google.com/p/gdata-java-client/. Currently, the code is maintained using ant. We use maven2 to build our project, so we manually uploaded those jars to our maven repository and added necessary pom files. Dependencies of gdata java library are described at http://code.google.com/p/gdata-java-client/wiki/Dependencies.
Google Picasa doc: http://code.google.com/apis/picasaweb/docs/2.0/developers_guide.html
Google Calendar doc: http://code.google.com/apis/calendar/docs/2.0/developers_guide.html

During using of Google Calendar APIs, one thing I want to do is publish a new event to a specific non-primary calendar. According to http://code.google.com/apis/calendar/docs/2.0/developers_guide_protocol.html#CreatingSingle, POST requests should be sent to http://www.google.com/calendar/feeds/default/private/full. However that results in putting new event to primary calendar. The document does not tell us how to put a new event to non-primary calendar.

Solution (How to publish a new event to a non-primary calendar)

Steps:

Get a list of all calendars owned by the user
Send GET request to http://www.google.com/calendar/feeds/default/owncalendars/full
In the response feed, each calendar has an entry.
Each entry should has a link element like this:
    <link rel="alternate"
             type="application/atom+xml"
             href="http://www.google.com/calendar/feeds/pnk4d66rpmiho1pl0m4t6dnki0%40group.calendar.google.com/private/full"/>
Value of attribute href is the URL to which the POST request should be sent to create a new event in the corresponding calendar.
Send POST request to the URL specified by attribute "href" found in step 2)
Example of POST data is shown http://code.google.com/apis/calendar/docs/2.0/developers_guide_protocol.html#CreatingSingle

MySQL Connection Timeout Error in Hibernate

2009-08-10T13:04:00.001-04:00

Problem

I am using Hibernate and MySQL in myour project. After our application is started, it just works well. However, if the application is not used for some time (like one day), it throws connection exception when it is used again. Trace of the exception:

com.mysql.jdbc.CommunicationsException: 
The last packet successfully received from the server was seconds ago.
The last packet sent successfully to the server was 407270 seconds ago, which  is longer than the server configured value of 'wait_timeout'. 
You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.

I googled this problem, and it seems that many other persons also suffer this problem. I found some really nice blog posts about how to solve this problem.

According to the error message, I set the option autoReconnect (http://dev.mysql.com/doc/refman/5.0/en/connector-j-reference-configuration-properties.html) to true. But it still does not work. I searched in google and it shows this option works in some platforms while it doesn't work in other platforms.

Solution

Basically, the solution is 1) use a connection manager 2) tweak parameters of the connection manager.

C3P0 is a free JDBC connection manager.
Some C3p0 configuration parameters are mapped to Hibernate config. Those mapped configuration parameters are listed at http://www.mchange.com/projects/c3p0/index.html#hibernate-specific.

"Please note that configuration parameters that ARE mapped into hibernate's config file MUST be set within hibernate's configuration, or else they will be overridden by hibernate-specified defaults.
[The hibernate-mapped c3p0 parameters are minPoolSize, maxPoolSize, maxIdleTime, maxStatements, acquireIncrement, testConnectionOnCheckout, and idleConnectionTestPeriod. These map to the fllowing hibernate parameters: hibernate.c3p0.min_size, hibernate.c3p0.max_size, hibernate.c3p0.timeout, hibernate.c3p0.max_statements, hibernate.c3p0.acquire_increment, hibernate.c3p0.validate, and hibernate.c3p0.idle_test_period. DataSources configured in Tomcat should always use c3p0-native parameter names. But pools constructed under the covers by Hibernate must be configured in hibernate's config file with hibernate-defined parameter names where applicable.] "

For those C3p0 parameters that are NOT mapped, they should be specified in file WEB-INF/classes/c3p0.properties.
Note: C3P0 configuration parameter <c3p0.testConnectionsOnCheckout> is mapped to <hibernate.c3p0.validate> only in hibernate 2.x!
Also the configuration parameter c3p0.testConnectionsOnCheckout is really expensive. Users can use c3p0.maxIdleTime (it is mapped to hibernate.c3p0.timeout in hibernate) instead.

My hibernate.cfg.xml includes following configuration:

        <property name="connection.provider_class">org.hibernate.connection.C3P0ConnectionProvider</property>
        <property name="c3p0.min_size">5</property>
        <property name="c3p0.max_size">30</property>
        <property name="c3p0.timeout">600</property>
        <property name="c3p0.max_statements">0</property>
        <property name="c3p0.acquire_increment">5</property>
        <property name="c3p0.idle_test_period">60</property>

The idle test operation can be optimized by specifying option preferrredTestQuery.
I am using MySQL as underlying RDB and putting c3p0.preferredTestQuery = SELECT 1 into c3p0.properties works fine (this is stolen from post comment http://www.databasesandlife.com/automatic-reconnect-from-hibernate-to-mysql/#comment-3156).

Resources

A really good article about how to reconnect from Hibernate to MySQL: http://www.databasesandlife.com/automatic-reconnect-from-hibernate-to-mysql/
C3P0 (Connection manager): http://www.mchange.com/projects/c3p0/index.html
How to configure C3P0 in Hibernate: https://www.hibernate.org/214.html
A forum thread about C3P0 configuration in Hibernate: https://forum.hibernate.org/viewtopic.php?t=934779&highlight=c3p0

Key generation using OpenSSL

2009-06-15T16:01:00.001-04:00

Generate RSA public/private key pair

Generate private key:

openssl genrsa -out private_key.pem 1024

Convert private key to PKCS8 format:

openssl pkcs8 -topk8 -in privkey.pem -nocrypt

Generate public key:

openssl rsa -pubout -in private_key.pem -out public_key.pem

Read Base64 Encoded private key

openssl rsa -text -in private_key.pem

Generate Self-Signed certificate

openssl req \
  -x509 -nodes -days 365 \
  -newkey rsa:1024 -keyout private_key.pem -out cert.pem

2-legged OAuth System Tutorial

2009-06-04T17:12:00.000-04:00

Use Cases:

(1) A user wants to grant data access priviledges to a third-party application.
Here three parties are involved:
User
    the end user
Third-Party application
    It wants to manipulate user's data on behalf of the user.
Service Provider
    a remote application that provides the real service.

The procedure of granting priviledges:
(I) complete steps described in section
(Provider side) > (For regular users)
(II) complete steps described in section
(Consumer side) > (For regular users)
Note: the public key uploaded in step (I) and private key uploaded in
step (II) must match.

(2) A user wants to develop a third-party application for our service.
(I) Register the application by completing steps described in section
(Provider side) > (For thrid-party application developers)
Now, the application can use the service.

(II) To make use of built-in third-party application, complete steps
described in section
(Consumer side) > (For application manager)

(3) A user wants to use a deployed and configured third-party application.
(I) Go to http://yourhost:yourport/oauthConsumer-0.1
(II) Login with existing account and you will be directed to page 'account.jsp'
(III) All supported third-party applications are listed under section
(Available service providers). Click a link to use the corresponding
application.

Consumer side

For regular users

Steps
(1) Go to http://yourhost:yourport/oauthConsumer-0.1
(2) Register a new account or log in with existing account
(3) Go to account.jsp (you should be automatically directed to this page)
(4) If it is the first time for the user to request an OAuth protected
      service via client application, the user should do some configurations.
      Click the link "Private Key Binding And Service Provider Setting"
(5) On this page, user can do some configurations
    (5.1) UserId binding
      This setting binds local user Id to user Id used by remote service.
      For example, if a user wants to access remote Yahoo flickr service
      through a third-party app, the local user Id is the user id used by the
      third-party app and it is bound to the user Id used by Yahoo Flickr.
    (5.2) Private key setting
      Possession of private key makes the third-party app be able to access
      remote service on behalf of the user.

For application manager

Actually, only administrators should have the priviledge to manage third-party
application registry. But currently user management system is simple and all
users have the same priviledges.
Steps
(1) Go to http://yourhost:yourport/oauthConsumer-0.1
(2) Register a new account or log in with existing account
(3) Go to account.jsp (you should be automatically directed to this page)
(4) Click the link "Attributes of available service providers"
(5) Now, the user can register a new third-party app, remove an existing
registration and modify existing registration information.

Provider side

Currently, only one service is offered - echo service.

For regular users

Steps
(1) Go to http://yourhost:yourport/oauthServer-0.1
(2) Register a new account or log in with existing account
(3) Go to account.jsp (you should be automatically directed to this page)
(4) Click the link "Certificate Setting"
(5) On this page, user can do some configurations
    (5.1) Third-party app priviledge setting
      Users can specify which third-party applications are allowed to access
      their data.
    (5.2) Certificate upload
      Upload your X.509 certificate (of PEM format).

For thrid-party application developers

Steps
(1) Go to http://yourhost:yourport/oauthServer-0.1
(2) Register a new account or log in with existing account
(3) Go to account.jsp (you should be automatically directed to this page)
(4) Click the link "Developer Account"
(5) Type information of the third-party application that is to be registered.
A consumer key/secret pair is generated as well.

Use of Public/Private key pair

There are two types of private/public key pair:
(1)
    One is used to guarantee integrity of messages transmissioned between
third-party application and service provider.
    So for service provider, the public key is published. If a third-party
application wants to access the service, it should use the published public
key to sign outbound messages.
(2)
    The other is owned and deployed by end user. The goal is to provide a way so
that third-party application can use backend service on behalf of user
without user involvement.
    So this pair is owned by end user. To deploy the key pair:
(*) upload the public key to service provider
(*) upload the private key to third-party application
    Then the third-party application uses the private key to sign part of the
message (The whole message is signed using pub/priv key pair described in
bullet 1).

OAuth message

    ___________________________________________________________
   |                                                           |
   |     _________________________________________________     |
   |    |   ___________________________________________   |    |
   |    |  |  ______________       _________________   |  |    |
   |   P|  | |              |     |                 |  |  |    |
   |   a|  | | Payload 2    |     |  Signature (2)  |  |  |    |
   |   y|  | |______________|     |_________________|  |  |    |
   |   l|  |___________________________________________|  |    |
   |   o|                                                 |    |
   |   a|   ___________________________________________   |    |
   |   d|  |                                           |  |    |
   |    |  |                Other parts                |  |    |
   |   1|  |___________________________________________|  |    |
   |    |_________________________________________________|    |
   |     _________________________________________________     |
   |    |                                                 |    |   
   |    |       Signature (1)                             |    |
   |    |_________________________________________________|    |
   |                                                           |
   |___________________________________________________________|

In above figure, <Signature (1)> is calculated using key described in bullet 1. <Signature (2)> is calculated using key described in bullet 2.

H2 Exception "Database may be already in use: Locked by another process"

2009-06-02T23:21:00.001-04:00

I wrote two Java web servlet apps (oauth consumer app, and oauth server app) both of which rely on H2 as database engine. I want H2 to be run in embedded mode instead of server mode. According to the official document, following statement should be added to web.xml

<listener>
    <listener-class>org.h2.server.web.DbStarter</listener-class>
</listener>

This class has methods that listen to the servlet context event (context initialization, context destroyed).

When I deploy just one of the two web apps to tomcat, it works fine. When I deploy both of them, following exception is thrown

Exception in thread "main" org.h2.jdbc.JdbcSQLException: Database may be already in use: Locked by another process. Possible solutions: close all other connecti on(s); use the server mode [90020-63]

I am 100% sure that those two apps use different databases. To make myself 200% sure that my code does not create or use additional databases unintentionaly, I rechecked my code many many times. But my code looks just fine. So I doubted it may be because of the H2 database engine.
Then I used command 'lsof' to list opened files by tomcat java process. Surprisingly, a database named 'test' is created under my home directory.
I reread the document http://www.h2database.com/html/tutorial.html#web_applications, and I got what I missed before.

For details on how to access the database, see the file DbStarter.java. By default the DbStarter listener opens an embedded connection using the database URL 'jdbc:h2:~/test', user name 'sa', and password 'sa'.

Code of class DbStarter can be accessed at http://code.google.com/p/h2database/source/browse/trunk/h2/src/main/org/h2/server/web/DbStarter.java.
I am just curious about why a default database is created by class DbStarter.

Solutions
(1) remove the listener declaration in web.xml
In other words, remove
<listener> <listener-class>org.h2.server.web.DbStarter</listener-class> </listener>
(2) change url of created database.

  <context-param>
    <param-name>db.url</param-name>
    <param-value>jdbc:h2:your_new_db_url</param-value>
  </context-param>

Change ROOT app in Tomcat

2009-05-17T17:48:00.001-04:00

Recently, I tried new version of shindig - shindig-1.1-SNAPSHOT-java (http://incubator.apache.org/shindig/download/index.html). It is a reference implementation for OpenSocial 0.9 spec.

The old problem (described at http://cglreport.zhenhua.info/2009/02/shindig-server-on-tomcat.html) has not been solved.
At first I wanted to try it by deploying the war as root application in tomcat.

One easy solution is to rename the war file to ROOT.war.
I used another solution.
Steps:
Deployment:
(1) create directory <tomcat_root>/myapps
(2) put the war file (shindig-server-1.1-SNAPSHOT.war) to the directory created in step 1.
(3) rename the original directory ROOT. (This step may be overlooked sometimes)
Configuration:
(1) Add file <tomcat_root>/conf/Catalina/localhost/ROOT.xml
(2) Content of the file is

<?xml version="1.0" encoding="UTF-8"?>
<Context
    path=""
    docBase="/globalhome/zhguo/servers/apache-tomcat-6.0.18/myapps/shindig-server-1.1-SNAPSHOT.war"
    debug="0"
    reloadable="true">
</Context>

(3) restart tomcat

Implementation of 2-Legged OAuth

2009-05-06T01:59:00.000-04:00

2-Legged OAuth

I did not change the process flow of original 3-legged OAuth to support 2-legged OAuth. I made a small change to user authorization step described at http://oauth.net/core/1.0/#auth_step2. Two additional parameters are added to support 2-legged OAuth: user_id and cgl_oauth_signagure. See http://zhenhua-guo.blogspot.com/2009/04/oauth-myproxy-integration-prototype.html for more information.

Integrity check:

check trust relationship between service provider and client app.
Currently, just check whether the clien app has been registered at service provider with correct information.
check pre-assigned priviledges to client app by user (this only applies to 2-legged OAuth)
A client app must have been granted access priviledge before it can access user data stored at service provider

Implementation

Client

Client App

Most important information about a client application includes:

name
description
consumer key
See http://oauth.net/core/1.0/#anchor6
consumer secret
See http://oauth.net/core/1.0/#anchor6
callback URL
The address to which user is redirected after data owner accepts access request.
signature method
HMAC-SHA1 or RSA-SHA1. Currently, only use RSA-SHA1.
private key
Used to generate signature
request token URL
See http://oauth.net/core/1.0/#auth_step1
authorization URL
See http://oauth.net/core/1.0/#auth_step2
access token url
See http://oauth.net/core/1.0/#auth_step3

A list of client applications is listed and users can add or remove client applications.
Snapshot:

Users can set userid that is used during invocation of remote service. Also to make it possible for client app to access user data at service provider, the user should upload private key to client app so that correct signature can be generated during OAuth process.

Tables

Four tables in database:

oauth_consumer_users: This table stores basic user account information. Currently, username and password are the only two attributes.
oauth_consumer_settting: This table stores private key belonging to each user. The private key is used during OAuth authorization process.
oauth_consumer_spbinding: This table stores binding relationships between users and service providers. So the information in this table is used when client app invokes remote service on behalf of user. Currently the main information is the user id that is used by service provider. In other words, a mapping from local user id to remote user id is established.
oauth_consumers: This table stores information of client applications.

Service Provider

Service provider provides interfaces that can be used by third party applications to access users' data.

For application developer

Client application registration

To register a client app, developers need to follow these instructions:

apply for a consumer key/secret pair.
type informatoin about the clien app including
- app name
- app description
- app callback URL
- public key

In my implementation, the registration page can be accessed at <baseurl>/oauthServer-0.1/developerAcc.jsp after our system is installed.
Screenshot:

For common users

Some third-party applications may want to access users' data stored at service provider. In 3-legged OAuth, the service provider redirects the user to a web page where the access request can be accepted or denied. In 2-legged, no user intervenetion is involved, so the authorization setting must be preset.
In our system, we provide a way for common users to grant priviledge to and revoke priviledge from client (3rd party) applications. Also the user must upload a public key against which signature of requests receieved from client app can be verified.
Snapshot

Tables

oauth_server_users: Basic user account information. Currently, just includes user name and password.
oauth_server_setting: stores public key extracted from myproxy server or direct upload.
oauth_server_dev: Stores information of registered client applications.
oauth_server_userappauth: Stores information of priviledge authorization set by each user.

Port attribute in HTTP Cookie

2009-04-08T10:54:00.001-04:00

Recently, I have encountered a http session problem in jetty.
I start two Jetty instances on the same machine. These two jetty instances share the same base URL, but port numbers are different.
http://example.com:8000/
http://example.com:9000/
When I visit http://example.com:8000/, a cookie (called JSESSIONID) is set by the server automatically.
After that I visit the other URL http://example.com:9000/, the cookie set for http://example.com:8000/ is sent to the server by user agent. The the server gets confused :-(

After debugging it, I found the reason is that the server does NOT include port number in the Set-Cookie header. According to section 3.3.1 in RFC 2965, if the port attribute is missing in the Set-Cookie header, the user agent would react based on following description:

   Port    The default behavior is that a cookie MAY be returned to any
           request-port.

That means if a cookie is set for address A, the cookie also matches those addresses which share all the same URL components with A but port number.

For tomcat, I searched its email list
http://www.google.com/search?hl=en&client=firefox-a&rls=org.mozilla%3Aen-US%3Aofficial&q=cookie+port+site%3Ahttp%3A%2F%2Fmail-archives.apache.org%2Fmod_mbox%2Ftomcat-users%2F&btnG=Search
It seems that port number currently is not supported in cookie management.

Solution
(1) use different domain names which map to the same IP
(2) use different paths.
E.g.
http://example.com:8000/webapp1
http://example.com:9000/webapp2

OAuth + MyProxy integration prototype

2009-04-03T02:38:00.001-04:00

Client

Client is the consumer of backend services.
Functionalities

A simple user management system
Users can create new accounts, log in the system and log out the system.
Each user can upload her/his private key
For each backend service, the user can set his/her user id which is bound to the backend service.

In above figure, the end user logins the client app as user "john". The client support accessing of three backend services. Then the user sets the binding of user id to each backend service. When the user wants to invoke backend service 1, the client app would set the user id to "mark" in the requests.

Service Provider

Functionalities of the service provider:

A simple user management system
Each user can bind a MyProxy account to his/her local account.
The needed information of MyProxy account includes username, password, MyProxy server host and port number.
Then the server would automatically retrieve a certificate chain from the specified MyProxy server. After parsing the certificate chain, public key of the original certificate that belongs to the user is extracted.
Note: we don't use public key included the proxy certificate. There is just one original certificate stored in the MyProxy server while many proxy certificates can be acquired. Each time a new proxy certificate is requested, a new public/private key is created and the public key is sent to MyProxy server. Usually, this process is done by system automatically.
Each service could be accessed by various client applications. Users can set which client apps are allowed to access their data.

OAuth Integration

In OAuth authorization procedure, there are three main steps:

Get a request token
Ask user to authorize the request
Exchange request token with an access token

Then the access token acquired in step (3) is used to access protected resources.
Trust relationship between client app and service provider is built out of band. Two mechanisms are supported: shared secret and RSA.
In my solution, step (2) is extended with step (1) and (3) staying the same. Two additional parameters are added: user_id and cgl_oauth_signagure. Parameter user_id is described here. The cgl_oauth_signature is calculated against user_id using the user's private key. After service provider receives the request, the corresponding public key (retrieved from MyProxy server) is used to verify the signature.

Future Work

Revoke the authorized privileges
1. Users can set an expiration time after which the access token will become invalid. Then the whole OAuth authorization process must be executed once again.
2. Provides a way for user to revoke authorized privileges at any time they like.
Trust relationship establishment.
Developers of client apps go to specified web page of the service provider to apply for development account. Then one of the following two things happens:
- A shared secret would be generated automatically and copied to the corresponding client app.
- The developers of the client apps upload their certificate to backend service.

Deploy standalone shindig on Quarry and an OAuth gadget demo

2009-02-28T19:48:00.000-05:00

I got a virtual machine which I can log in as root. The virtual machine is pretty clean without any more packages besides CentOS.
I installed vim using yum install vim-enhanced. I also installed jdk1.5 and maven2.

Then I checked out the source code from shindig svn trunk. See instructions here http://incubator.apache.org/shindig/#tab-building.
Unfortunately, the test failed when I compiled and installed shindig. This kind of things always happens because of instability of trunk code. So I copied my previous checkout of shindig code which worked well from another machine. Then the compilation and installation succeeded.

OAuth gadget demo to retrieve google contact list

Now the sample demo is here:
http://gw11.quarry.iu.teragrid.org:9999/gadgets/files/samplecontainer/samplecontainer.html

Set field "Displaying gadget" to http://156.56.104.196:8888/oauth/gadgets/os_gadget.xml, Uncheck "use cache".
Click button "reset all". If everything works fine, you would see following in main panel:

Clieck "Personalize this gadget", a popup window would appear which prompts you to login to your google account. Then you can choose grant or deny the access request. If you grant the request, the popup window would be closed automatically, and the main panel would be populated with your google contact list.

Manipulate IFrame in Javascript

2009-02-21T22:57:00.001-05:00

Create a iframe and set the content directly:

var frame = document.createElement('iframe');
parentElement.appendChild(frame);
frame.contentDocument.write(result);

According to Dom level 2 HTML specification, contentDocument is a property of HTMLIFrameElement.

Following code may also work. However I have not found the contentWindow property in W3C DOM specification.

var frame = document.createElement('iframe');
parentElement.appendChild(frame);
frame.contentWindow.document.write(result);

Integration of OpenSocial containers via OAuth

2009-02-14T23:58:00.001-05:00

For OpenSocial gadgets, OpenSocial containers are necessary to hold users' data (friends, profile, ...). In Javascript API, class opensocial.DataRequest can be used to get users' data from the container which renders the gadget.

But to integrate users' data from third-party OpenSocial containers, some additional steps are needed. opensocial.DataRequest does not allow gadget developers to specify which OpenSocial container would be used to serve the data. Also the gadgets can not send requests directly to third-party containers because of Same-Origin policy imposed by browsers.

Method gadgets.io.makeRequest can be used to send arbitrary HTTP GET/POST requests. These requests are sent to the original container first. The original container does some processing and relays the requests to the destination address.
http://code.google.com/apis/opensocial/docs/0.8/reference/gadgets/#gadgets.io.makeRequest
http://code.google.com/apis/opensocial/articles/makerequest-0.8.html
http://code.google.com/apis/gadgets/docs/remote-content.html

Both REST and RPC protocol specifications requires that compliant containers must be OAuth service provider:
http://www.opensocial.org/Technical-Resources/opensocial-spec-v081/restful-protocol (Section 4)
http://www.opensocial.org/Technical-Resources/opensocial-spec-v081/rpc-protocol
Also I found OAuth Consumer Request 1.0 Draft 1 (hosted at googlecode, not oauth.net) which standardizes two-legged authorization process without a User involvement. It is useful in server-to-server interaction.

How to use OAuth in gadgets
http://code.google.com/apis/gadgets/docs/oauth.html (Very useful)
http://sites.google.com/site/oauthgoog/oauth-proxy/social-oauthproxy

OpenSocial container list:
http://wiki.opensocial.org/index.php?title=Main_Page#Container_Information

Extensions:
Also, I found many OAuth extension drafts which are not listed in OAuth.net. I guess these extensions are implemented in Shindig and evaluated.
http://oauth.googlecode.com/svn/spec/ext/

Enable/Disabl Directory Listing in Tomcat

2009-02-07T16:44:00.000-05:00

By default, directory listing (list content of a directory instead of render a web page) is forbidden in Tomcat.

To enable it globally, modified file <tomcat>/conf/web.xml.
Default setting for default servlet is:

    <servlet>
        <servlet-name>default</servlet-name>
        <servlet-class>org.apache.catalina.servlets.DefaultServlet</servlet-class>
        <init-param>
            <param-name>debug</param-name>
            <param-value>0</param-value>
        </init-param>
        <init-param>
            <param-name>listings</param-name>
            <param-value>false</param-value>
        </init-param>
        <load-on-startup>1</load-on-startup>
    </servlet>

Change the param-value for parameter listings to true.

To enable it for a specific web application, add following snippet to the web.xml of the application

    <servlet>
        <servlet-name>default_new</servlet-name>
        <servlet-class>
          org.apache.catalina.servlets.DefaultServlet
        </servlet-class>
        <init-param>
            <param-name>debug</param-name>
            <param-value>0</param-value>
        </init-param>
        <init-param>
            <param-name>listings</param-name>
            <param-value>true</param-value>
        </init-param>
        <load-on-startup>1</load-on-startup>
    </servlet>
    <!-- The mapping for the default servlet -->
    <servlet-mapping>
        <servlet-name>default_new</servlet-name>
        <url-pattern>/</url-pattern>
    </servlet-mapping>

Note: servet-name can not be default because it is name of tomcat's default servlet . So you need to use a different name. Also servlet-mapping element is necessary to make it work.

Shindig server on Tomcat

2009-02-07T16:01:00.000-05:00

Shindig can be run with built-in jetty server without problems. But after I deployed the war to Tomcat, there are some problems. I got a blank page when I tried to render a gadget. I searched for the problem, some other guys got this problem as well.

Finally I figured out how to make shindig work on Tomcat.
Why does not it work?
It ignores the context path. In other words, the implementation only considers the situation that the application is deployed as root application.

Two possible Solutions
(1) make shindig war the root application
Two alternative ways:
(*) rename it to ROOT.war and deploy it to tomcat. (The original ROOT application cannot be used)
(*) change conf/context.xml to set shindig application as root application
(2) Modify shindig configuration files <shindig>/java/common/conf/shindig.properties and <shindig>/config/container.js to add context path in all related urls.
It seems to work well. I am not sure whether there is any potential problem.

Maven cheat sheet

2008-11-08T02:21:00.000-05:00

Download and install maven: http://maven.apache.org/download.html.

Running Maven
http://maven.apache.org/guides/getting-started/maven-in-five-minutes.html
http://maven.apache.org/guides/getting-started/index.html
Generally the local repository is provided in USER_HOME/.m2/repository.

Configuration
http://maven.apache.org/guides/mini/guide-configuring-maven.html
Three levels:

Project - most static configuration occurs in pom.xml.
There should be a project configuration for every project. The configuration in pom.xml overrides the same configuration in other two levels. A user does not need to specify every aspect of the configuration. Those unspecified aspects are inherited from Maven's default Super POM.
Introduction to POM: http://maven.apache.org/guides/introduction/introduction-to-the-pom.html.
POM reference: http://maven.apache.org/pom.html
Installation - this is configuration added once for a Maven installation
This configuration file is generally located in M2_HOME/conf/settings.xml.
NOTE: This location can be overridden with the system property:
-Dorg.apache.maven.global-settings=/path/to/global/settings.xml
Guide: http://maven.apache.org/guides/mini/guide-configuring-maven.html
Reference: http://maven.apache.org/ref/2.0.8/maven-settings/settings.html
Detailed reference: http://maven.apache.org/settings.html
User - this is configuration specific to a particular user
This configuration file is generally located in USER_HOME/.m2/settings.xml.
NOTE: This location can be overridden with the system property:
-Dorg.apache.maven.user-settings=/path/to/user/settings.xml

Build your own private/internal repository:
This article introduces how to create a repository using Artifactory: http://www.theserverside.com/tt/articles/article.tss?l=SettingUpMavenRepository. In addition, the author also compares some mainstream maven remote repository managers including Standard maven proxy, Dead simple Maven Proxy, Proximity and Artifactory.
In my case, I also use Artifactory and deploy it to tomcat. It has a nice web-based interface. Artifactory uses database(derby I think) to store various repository data so a user can not know the repository content by directly looking at the directory.

Deploy your artifacts to remote repository by using maven-deploy plugin:
http://maven.apache.org/plugins/maven-deploy-plugin/usage.html
(1) If the artifacts are built by using Maven, you should use deploy:deploy Mojo.
In your pom.xml, element <distributionManagement/> should be inserted to tell Maven how to deploy current package. If your repository is secured, you may also want to configure your settings.xml file to define corresponding <server/> entries which provides authentication information.
Command: mvn deploy.
(2) If the artifacts are NOT built by using Maven, you should use deploy:deploy-file Mojo.
Sample command:
mvn deploy:deploy-file -Dpackaging=jar -Durl=file:/grids/c2/www/htdocs/maven2
-Dfile=./junit.jar -DgroupId=gridshib -DartifactId=junit -Dversion=GTLAB

FAQ:
(1) What does maven standard directory layout look like?
http://maven.apache.org/guides/introduction/introduction-to-the-standard-directory-layout.html
(1) How to specify parent artifact in pom.xml?
Read http://maven.apache.org/guides/introduction/introduction-to-the-pom.html.
(2) If a dependent package can not be download from central Maven repository, three methods can be used to deal with it:

"
Install the dependency locally using the install plugin. The method is the simplest recommended method. For example:
mvn install:install-file -Dfile=non-maven-proj.jar -DgroupId=some.group -DartifactId=non-maven-proj -Dversion=1
Notice that an address is still required, only this time you use the command line and the install plugin will create a POM for you with the given address.
Create your own repository and deploy it there. This is a favorite method for companies with an intranet and need to be able to keep everyone in synch. There is a Maven goal called deploy:deploy-file which is similar to the install:install-file goal (read the plugin's goal page for more information).
Set the dependency scope to system and define a systemPath. This is not recommended, however, but leads us to explaining the following elements:
"

(2) How to add new repository?
Put following code snippet into pom.xml or settings.xml.

<repository>
  <id>your-new-repository-id</id>
  <name>New Maven Repository </name>
  <layout>default</layout>
  <url>Address of the new repository</url>
  <snapshots>
    <enabled>enable-it?</enabled>
  </snapshots>
  <releases>
    <enabled>enable-it?</enabled>
  </releases>
</repository>

(3) How to disable default central maven repository?
Put following snippet into your pom.xml.

<repository>
  <id>central</id>
  <name>Maven Repository Switchboard</name>
  <layout>default</layout>
  <url>http://repo1.maven.org/maven2</url>
  <snapshots>
    <enabled>false</enabled>
  </snapshots>
  <releases>
    <enabled>false</enabled>
  </releases>
</repository>

(4) How can I package source code without run test?
Feed parameter -Dmaven.test.skip=true into the command line.
Note this property is defined by maven plugin surefire.
(5) Why does "mvn clean" delete my source code?
In your pom.xml, if content of element <directory> nested in element <build> is "./", "mvn clean" will delete all content in current directory including the src directory.
There are two more elements which can be used to specify locations of compiled classes.
outputDirectory: The directory where compiled application classes are placed.
testOutputDirectory: The directory where compiled test classes are placed.
(6) How to add resources into built package?
http://maven.apache.org/guides/getting-started/index.html#How_do_I_add_resources_to_my_JAR.
http://maven.apache.org/guides/getting-started/index.html#How_do_I_filter_resource_files

Automatically adjust height of IFrame using Javascript

2008-10-10T18:09:00.001-04:00

A simple and natural requirement, but unfortunately it may or may not work as you expect.
What matters here is security. If src of an iframe does not access external address, it works well. Otherwise, nothing will solve the problem except that the user lowers the security level. Javascript can not access information of an IFrame whose content is hosted by a third-party site.

Let us assume that src of an iframe does not point to an external resource, following code can be used:

var ele =document.getElementById(frameid); 
ele.style.height = ele.contentWindow.document.body.scrollHeight

You may wonder why document.frames[frame_name] is not used here. Anyway, I tried it on Firefox 3, it did not work. As you know, compatibility is a real pain for web developers.

One possible solution is to set up a proxy server which relays the requests/responses.

Some notes about tomcat/hibernate/servlet

2008-09-10T13:22:00.001-04:00

Tomcat

For tomcat applications, resources (configuration files...) should be put in directory WEB-INF/classes. The files cannot be symbolic links. They MUST be regular files in order for tomcat to handle them correctly. Of course, you can embed resources into your .jar files.
For Tomcat, the regular CLASSPATH environment variable is IGNORED. The resources (jars, configuration files...) must be put into your tomcat app directory. Read this post to see what is under the hood.
Resource/class search order for web apps:

Bootstrap classes of your JVM
System class loader classses (described above)
/WEB-INF/classes of your web application
/WEB-INF/lib/*.jar of your web application
$CATALINA_HOME/lib
$CATALINA_HOME/lib/*.jar

The System class loader is the only place where CLASSPATH can be parsed and used. However, tomcat startup script(bin/catalina.sh) totally ignore environment variable CLASSPATH. You will figure it out if you read file bin/setclasspath.sh. In my case, first several lines are :

#  Set CLASSPATH and Java options
# 
#  $Id: setclasspath.sh 589060 2007-10-27 08:19:24Z jfclere $
# -----------------------------------------------------------------------------

# First clear out the user classpath
CLASSPATH=

Now, you know the root of the problem right? If you want to include CLASSPATH, you can modify the bash file. I have not tried it. It should work.

Hibernate

For hibernate app, corresponding mapping tables must be created before you can run your app. Hibernate will NOT create those tables automatically for you. For the primary key property/column, If you use generator "native", you must set property of the primary key column correctly in order to let database generate primary key values correctly. For example, you can use AUTO_INCREMENT to make database automatically generate primary key values in MySQL. You can choose to use generator "assigned" to assign primary key values in your programs.
For database operations in Hibernate, you should always EXPLICITLY start and commit a transaction.

Servlet/JSP

Method Forward declared in RequestDispatcher should be understood correctly. It is an INTERNAL control transfer. It means this control transfer is TRANSPARENT to end users. One consequence is that the links in the target page, to which you forward control, MUST use absolute path instead of relative path.
For example, first a user sends request to http://domain/users/gerald.jsp. At server side, your servlet internally transfers control to /users/profiles/g/gerald.jsp. In the target page, if you have links with relative path, the paths are relative to the original request path (/users/ in this example) instead of path of the target page (/users/profiles/g/ in this example).
Your servlet should not have generated any response ( in output stream to request sender ) when redirection or forwarding is carried out.

Hibernate installation (run on MySQL)

2008-09-07T16:59:00.000-04:00

Recently, I tried to install Hibernate on gridfarm machine. The version I used was 3.3.0 SP1.
Configuration document can be found here. However, the document seems not to be updated.
(1) Configuration file hibernate.properties vs. hibernate.cfg.xml.
In the document, either one can be used. If both are used, configurations in hibernate.cfg.xml would override configurations in hibernate.properties. The document says using hibernate.properties is an easy way to configure Hibernate. So I chose it. However, after trial I found that in Hibernate 3.3.0 SP1 hibernate.cfg.xml. In other words, you MUST have a configuration file "hibernate.cfg.xml". Of course, you can use programmatical configuration which is not covered in this post.
(2) Logger
In documents, I saw "Hibernate logs various events using Apache commons-logging.". However, when I tried to run my Hibernate app, it complained that logger class can not be found. After inspection, I figured out that Hibernate 3.3.0 does NOT use Apache common-logging any more. Instead it uses slf4j. But in Hibernate 3.3.0 SP1 distribution only slf4j-api jar is included while an implementation is needed to run Hibernate(If you turn off logging, I don't know whether it will still complain). In other words, to use logging functionality, you must download an implementation of logging service supported by slf4j. I used slf4j-simple.
(3) JNDI bindings (for session factory and data source).
Because I just wanted to write a simple prototype without deploying in J2EE server, I chose not to use JNDI bindings for data source or session factory. After several hour trial, I could not get it to work!! I always got the error "org.hibernate.impl.SessionFactoryObjectFactory - Could not bind factory to JNDI".It seemed that Hibernate always was always trying to use JNDI to bind Session Factory. However, the official document says

"A JNDI bound Hibernate SessionFactory can simplify the lookup of the factory and the creation of new Sessions. Note that this is not related to a JNDI bound Datasource, both simply use the same registry!

If you wish to have the SessionFactory bound to a JNDI namespace, specify a name (eg. java:hibernate/SessionFactory) using the property hibernate.session_factory_name. If this property is omitted, the SessionFactory will not be bound to JNDI. (This is especially useful in environments with a read-only JNDI default implementation, e.g. Tomcat.)

When binding the SessionFactory to JNDI, Hibernate will use the values of hibernate.jndi.url, hibernate.jndi.class to instantiate an initial context. If they are not specified, the default InitialContext will be used.

Hibernate will automatically place the SessionFactory in JNDI after you call cfg.buildSessionFactory(). This means you will at least have this call in some startup code (or utility class) in your application, unless you use JMX deployment with the HibernateService (discussed later).

If you use a JNDI SessionFactory, an EJB or any other class may obtain the SessionFactory using a JNDI lookup.

We recommend that you bind the SessionFactory to JNDI in a managend environment and use a static singleton otherwise. To shield your application code from these details, we also recommend to hide the actual lookup code for a SessionFactory in a helper class, such as HibernateUtil.getSessionFactory(). Note that such a class is also a convenient way to startup Hibernate—see chapter 1. "

It turned out that a property called "hibernate.current_session_context_class" must be set.
My hibernate.cfg.xml looks like:

<!DOCTYPE hibernate-configuration PUBLIC
    "-//Hibernate/Hibernate Configuration DTD 3.0//EN"
    "http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd">

<hibernate-configuration>
    <session-factory>
        <property name="show_sql">true</property>
        <property name="current_session_context_class">thread</property>
        <property name="connection.username">username</property>
        <property name="connection.password">password</property>
        <property name="connection.url">jdbc:mysql://localhost/hibernate</property>
        <property name="connection.driver_class">com.mysql.jdbc.Driver</property>
        <property name="dialect">org.hibernate.dialect.MySQLDialect</property>
        <mapping resource="User.hbm.xml"/> 
    </session-factory>
</hibernate-configuration>

Then, I tried a very very simple read-only JNDI service provider - simple-jndi. My simple-jndi configuration file (jndi.properties which should be put into root of your classpath) looks like:

#The first (required) parameter, org.osjava.sj.root, is the location 
#of your simple-jndi root, which is the location in which simple-jndi
#looks for values when code asks for them.

org.osjava.sj.root=/home/zhguo/share/simple-JNDI/config
java.naming.factory.initial=org.osjava.sj.SimpleContextFactory

Under directory pointed to by org.osjava.sj.root property, I have a jndi data file called hibernate_jndi.properties:

mysql.type=javax.sql.DataSource
mysql.driver=com.mysql.jdbc.Driver
mysql.url=jdbc:mysql://localhost/hibernate
mysql.user=username
mysql.password=password

So, my new version of hibernate.cfg.xml looks like:

<hibernate-configuration>
    <session-factory>
        <property name="show_sql">true</property>
        <property name="current_session_context_class">thread</property>
        <property name="connection.datasource">hibernate_jndi.mysql</property>
        <property name="dialect">org.hibernate.dialect.MySQLDialect</property>
        <mapping resource="User.hbm.xml"/> 
    </session-factory>
</hibernate-configuration>

Note: in the file, session factory name is not set! This is because we are using a read-only JNDI. If you use a read-write JNDI, you can set the session factory name in one of the following ways:
(*) In hibernate.properties file:
hibernate.session_factory_name hibernate/session_factory
(*) In hibernate.cfg.xml file:
<session-factory name="your name goes here">

Hadoop and HBase port usage

2008-08-27T01:18:00.000-04:00

Hadoop port usage

On name node
50090: org.apache.hadoop.dfs.SecondaryNameNode
33220: ditto.(This port is not fixed and may be changed when hadoop is restarted)

50070: org.apache.hadoop.dfs.NameNode
9000:ditto
46684: ditto(This port is not fixed and may be changed when hadoop is restarted)

9001: org.apache.hadoop.mapred.JobTracker

50030: ditto
60502: ditto(This port is not fixed and may be changed when hadoop is restarted)

On data node
50075: org.apache.hadoop.dfs.DataNode
50010: ditto
45868: ditto(This port is not fixed and may be changed when hadoop is restarted)

50060: org.apache.hadoop.mapred.TaskTracker
55027: ditto(This port is not fixed and may be changed when hadoop is restarted)

HBase port usage
On master
60000: org.apache.hadoop.hbase.master.HMaster start
60010: ditto

On data node
60030: org.apache.hadoop.hbase.regionserver.HRegionServer
60020: ditto

Insert pubchem data into HBase

2008-08-25T15:04:00.000-04:00

HBase shell
HBase provides a shell utility which lets users to execute simple commands. The shell can be started up using:
${HBASE_HOME}/bin/hbase shell
Then input command help to get help document which describes usage of various supported commands. These commands can be used to manipulate data stored in HBase. E.g. command list can be used to list all tables in hbase. Command get can be used to get row or cell contents from hbase. Command put can be used to store data into a cell.

Data insertion
Data source is ftp://ftp.ncbi.nlm.nih.gov/pubchem/. I modified python scripts and C source code given by Rajarshi.
Data retrieval and processing steps:

Download all information about compounds from ftp://ftp.ncbi.nlm.nih.gov/pubchem/Compound/CURRENT-Full/SDF. I finally got 123 GB.
Decompress those files
Extract information from these .sdf files and write it to .txt files. Library openbabel is used to compile the C++ code.
Combine those .txt files generated in step 3 into one big .dat file
Write a ruby script to insert all data in the .dat file into HBase.
Command is like this: ${HBASE_HOME}/bin/hbase org.jruby.Main rubyscript.rb

Why did I write Ruby script instead of Java program in step 5?
HBase is written in Java and so provides Java API. However, to compile Java programs is kind of cumbersome -- set lengthy CLASSPATH ...
So I chose to write scripts which can be executed directly by HBase shell. I found useful information on this page. There is a section called "scripting" in that post. But the information there is far from complete. It does not tell readers how to write the scripts. At first, I wrote a script which included some shell commands, one command per line, and then fed it to hbase shell. Unfortunately, it didn't work. After enumerous trials, I found that Ruby scripts could be fed to shell. Ruby scripts cannot make use of existing shell commands directly. Ruby binding of original Java APIs must be used.

I have not learnt Ruby at all before. So I must teach myself to grasp basic knowledge about Ruby. Ruby is sort of different in terms of syntactic flexibility. It supports so many shorthands to improve productivity. Anyway, "Ruby is easy to learn, but hard to master". By the way, Ruby documents seem not to be abundant compared with Python, Perl...

How to write Ruby scripts for HBase?
This site http://wiki.apache.org/hadoop/Hbase/JRuby contains related information. But I could not run the sample script successfully because of errors in the script!!! Damn it! I wonder whether the author had tested the code before he released it. Some errors are so obvious.
After the ruby script is completed, it can be executed using:
${HBASE_HOME}/bin/hbase org.jruby.Main rubyscript.rb
Java API:
http://hadoop.apache.org/hbase/docs/current/api/index.html

My ruby script:

#!/usr/bin/ruby -w

include Java
import org.apache.hadoop.hbase.HBaseConfiguration
import org.apache.hadoop.hbase.HColumnDescriptor
import org.apache.hadoop.hbase.HTableDescriptor
import org.apache.hadoop.hbase.client.HBaseAdmin
import org.apache.hadoop.hbase.client.HTable
import org.apache.hadoop.hbase.io.BatchUpdate
import org.apache.hadoop.io.Text

pubchem_compound_fields = [
    'cid',
    'iupac_openeye_name',
    'iupac_cas_name',
    'iupac_name',
    'iupac_systematic_name',
    'iupac_traditional_name',
    'nist_inchi',
    'cactvs_xlogp',
    'cactvs_exact_mass',
    'openeye_mw',
    'openeye_can_smiles',
    'openeye_iso_smiles',
    'cactvs_tpsa',
    'total_charge',
    'heavy_atom_count']

compound_table_name = 'compound'

numfields = pubchem_compound_fields.length

path = "/home/zhguo/BigTable/BigTable-Pubchem/data/"
filename = "#{path}compound.dat"
file = File.new(filename, 'r')
counter = 0

conf = HBaseConfiguration.new
tablename = compound_table_name
tablename_text = Text.new(tablename)
desc = HTableDescriptor.new(tablename)
coltextarr = Array.new
pubchem_compound_fields.each_with_index do |v, i|
    if (i == 0) then next; end
    desc.addFamily(HColumnDescriptor.new("#{v}:"))
    coltextarr << Text.new("#{v}:")
end

admin = HBaseAdmin.new(conf)
if !admin.tableExists(tablename_text) then
    admin.createTable(desc)
=begin
    puts "deleting table #{tablename_text}"
    admin.disableTable(tablename_text)
    admin.deleteTable(tablename_text)
    puts "deleted table #{tablename_text} successfully"
=end
end

#admin.createTable(desc)
table = HTable.new(conf, tablename_text)

startind = 1641500 #from which line should we start.This
                   #is useful when you don't want to start
                   #from the beginning of the data file.

nlines = `cat #{filename} | wc -l`

logfilename = 'updatedb.log'
logfile = File.new(logfilename, "a")
while (line = file.gets) #&& (counter < 20)
    counter += 1
    if (counter < startind) then
        next
    end
    msg = "processing line #{counter}/#{nlines}"
    logfile.puts msg
    if counter%100 == 0 then
        print  msg
        STDOUT.flush
        logfile.flush
    end

    arr = line.split("\t")
    len = arr.length
        if (numfields != len) then
        next
    end
    rowindex = 0
    rowname = arr[rowindex]
    arr.delete_at(rowindex)
    row = Text.new(rowname)
    b = BatchUpdate.new(row)

    arr.each_with_index do |v, i|
        str = java.lang.String.new(v)
        b.put(coltextarr[i], str.getBytes("UTF-8"))
    end
    table.commit(b)
end

Installation and configuration of Hadoop and Hbase

2008-08-24T18:33:00.001-04:00

Hadoop

Installation
Hadoop installation instructions: http://hadoop.apache.org/core/docs/current/quickstart.html and http://hadoop.apache.org/core/docs/current/cluster_setup.html.
To set up hadoop cluster, generally two configuration files should be modified:
hadoop-site.xml and slaves.
(1) My hadoop-site.xml looks like:

<configuration>
  <property>
    <name>fs.default.name</name>
    <value>pg3:9000</value>
  </property>
  <property>
    <name>mapred.job.tracker</name>
    <value>pg3:9001</value>
  </property>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
</configuration>

Read file hadoop-default.xml for all available options.
(2) My slaves file looks like:

localhost
pg1
pg2

I need to install Hadoop on three machines now and I use rsync to make these machines synchronized with eath other in terms of configuration.

Commands
(*) Format a new file system: hadoop namenode -format
(*) Start/stop Hadoop
start-dfs.sh/stop-dfs.sh
start up the distributed file system (HDFS)
start-mapred.sh/stop-mapred.sh
start up map reduce service.
start-all.sh/stop-all.sh
start up both HDFS and map reduce service

Hadoop reads content in file slaves to get all nodes and then starts up all these nodes.

Check status of the services
HDFS: http://domain:50070/
MapReduce: http://domain:50030/

HBase

Installation instructions: http://hadoop.apache.org/hbase/docs/current/api/overview-summary.html#overview_description
The configuration file is hbase-site.xml. My hbase-site.xml looks like

<configuration>
  <property>
    <name>hbase.master</name>
    <value>pg3:60000</value>
    <description>The host and port that the HBase master runs at.</description>
  </property>

  <property>
    <name>hbase.rootdir</name>
    <value>hdfs://pg3.ucs.indiana.edu:9000/hbase</value>
    <description>The directory shared by region servers.</description>
  </property>
</configuration>

Commands
start-hbase.sh starts up hbase service
stop-hbase.sh stop hbase service

Note: hbase bases its functionalities on hadoop. Sometimes it is necessary for hbase to know the configuration of hadoop. Following statements are excerpted from hbase document which I think is important:

"Of note, if you have made HDFS client configuration on your hadoop cluster, hbase will not see this configuration unless you do one of the following:

Add a pointer to your HADOOP_CONF_DIR to CLASSPATH in hbase-env.sh
Add a copy of hadoop-site.xml to ${HBASE_HOME}/conf, or
If only a small set of HDFS client configurations, add them to hbase-site.xml
An example of such an HDFS client configuration is dfs.replication. If for example, you want to run with a replication factor of 5, hbase will create files with the default of 3 unless you do the above to make the configuration available to hbase. "

Mysql error "InnoDB: Unable to lock ./ibdata1, error: 11"

2008-08-18T15:06:00.000-04:00

Recently, there was a power outage in the lab. I did not shut down my machines before the power outage. After I restarted my Ubuntu, I could not start up Mysql. The error was:

InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Error in opening ./ibdata1

But I am 100% sure that no another mysqld process was running. After I searched on line, I found that I was not the only one encountering this error. See this post.
This post has an insight into this problem. It seems to be caused by NFS. Mysql is not installed on a local file system. It is installed on a remote file system which is mounted to other file systems by using NFS.

Solution
make a copy of the original files (ibdata1, ib_logfile0, ib_logfile1...).

mv ibdata1 ibdata1.bak
cp -a ibdata1.bak ibdata1
......

Social Networking on Drupal

2008-07-29T22:01:00.001-04:00

Previously, I tried several open source social networking frameworks including elgg, aroundme, peopleaggregator... However, few of them are mature and stable. So I also tried Content Management Systems. I delved into Drupal.
Drupal is an excellent CMS which has many powerful modules. Some modules provide social networking support. There exists a drupal group which promotes social networking support in Drupal.
Here I list some useful modules which may be used in social networking:
Organic Groups
"Enable users to create and manage their own groups. Each group can have subscribers, and maintains a group home page. Groups get their own theme, language, taxonomy, and so on."

Buddylist
Allows users to add other users to buddy list, organize buddies into buddy group and monitor buddies' content.

Invite
Adds an 'Invite a friend' feature that allows users to send and track invitations to join a web site.

FOAF
Functionalities:
"1. automatically import/synchronize profile information between any Drupal-powered sites
2. import profile information from external FOAF files
3. export a FOAF file based on a user's profile"

Privatemsg
Allows users to send private messages to each other.

User Relationships
Allow users to create named relationships between each other.

Wiki
interwiki
This filter provides a simplified, wiki-like syntax for linking to articles or search results on many commonly-used internet reference websites
Liquid Wiki
The Liquid Wiki Engine Project aims to build a module providing wiki funktionality to Drupal.
MediaWiki
Integrate Drupal with an existing MediaWiki installation.

Online Chat
Chat Room
This module enables chat rooms to be set up on Drupal sites. No additional software is required. Chat Room uses AJAX to update chats without page reloads. Chat rooms are nodes that provide access to chats and chat archives.
BoWoB
Free Chat for your website, integrated with your Drupal users and design. It is implemented in Flash.

Google Calendar Demo Gadgets

2008-07-17T02:27:00.000-04:00

I have implemented some calendar gadgets based on existing legacy gadgets.

(1) Google calendar viewer gadget
This app does not use Google Calendar Javascript API. Google Calendar generates automatically feeds for public events owned by a user. Address of the feed is http://www.google.com/calendar/feeds/username@gmail.com/public/full. This gadget retrieves public events of a specific calendar user and display them accordingly. Not all events are retrieved and displayed.
Assume today is A, then events that fall into period (A-30days) to (A + 60days) are handled.
Note: this gadget can just get public events.
I deployed it in both Orkut sandbox and Ringside. Following screenshots are taken from Orkut.

Main page:

Description of various parts of the page:

When a user clicks the brief information of an event, detailed information will be displayed in place.

(2) Standalone Google calendar client application.
This app does not rely on Gadget API or Open Social API. It just relies on Google Calendar Javascript library. So it can be run independently without gadget container. I deployed it on Ringside using IFrame.
It can be accessed by visiting http://gf1.ucs.indiana.edu:8080/gcalendar/calendar2.htm or you can use it in Ringside.

If a user has not logged in Google account, he/she will see a "Login" button.

After user clicks the "Login" button, he/she will be redirected to Google login page. After he/she logs in Google, he/she will be redirected to a "Access Request Authorization" page. Then the user can choose to grant or deny the access. It is shown below:

After the user grants access, following initial page will be displayed:

Description of every part in the page:

When a user clicks the time range selection text box, a data picker will be displayed so that the user can easily select time period. It is shown below:

When a user selects his/her time range and clicks "Go" button, information of events within the specific time perios will be retrieved. And when the user clicks the drop-down box with label "Select...", a list of events will be displayed. See below screenshot:

When the user selects an event, detailed information will be displayed in corresponding fields.

The user can modify information of an event and save it back in Google Calendar by clicking button "Save to Google Calendar".

(3) Google calendar gadget which allowes users to log in
This gadget is different from the first gadget mentioned above because this gadget allowes users to log in so that all events(public and private) can be retrieved and displayed. This gadget is based on the app(2) which is described above. The user interface is the same as that of app(2).
I deployed it on Ringside and Orkut sandbox successfully.

Demo Apps

2008-07-09T14:23:00.001-04:00

I have written several demo apps.
(1) A regular gadget with gadgets.* API
This is based on official youtube gadget presented here. Official youtube gadget uses traditional gadget API, which means it can not be run in open social containers.
I modified the source of official youtube gadget to make it compatible with open social API. Source is hosted here.
I run it on Ringside, and it works well. Screenshot is shown below:

Besides, I run it on iGoogle sandbox and Orkut sandbox successfully.

(2) An open social gadget
This gadget is written with Open Social API so that it can be run in all standard compatible containers. This gadget lists all friends you have in a container. If you add it to your Orkut account, it will list your friends in your Orkut account. If you add it to your hi5 account, it will list your friends in your hi5 account.
Source is hosted here.
I run it on Ringside, and screenshot is shown below:

Besides, I run it on iGoogle sandbox and Orkut sandbox successfully.

(3) A Facebook application.
This application gets your friend list on Facebook and displays related information including profile picture, last name, first name and birth date. You can see its information here. Meanwhile you can add it to your Facebook account.
Screenshot is shown below:

(4) A port of application (3)(The facebook application).
Because Ringside implements Facebook API and FBML, application (3) should be able to be run on Ringside with minor modifications. New API key and secret should be used. Besides Ringside RESTful server (instead of Facebook server) address should be used.
Finally, I got it to run successfully on Ringside.

(5) Make Facebook to use Ringside application
I used the method described in this post and this post. It should work. But unfortunately it did not work. Currently I don't know why.

To try it in Ringside
First go to http://gf1.ucs.indiana.edu:8080/ to log in.
Then click tab "Applications":

Following page should be displayed:

Main panel displays a list of all applications you have added to your account. You can click any one to use it.
Or you can click button "Find More Apps" at top right corner to browse more available apps.
Application "youtube gadget" is the application (1) described above.
Application "Friend gadget(Open social)" is the application (2) described above.
Application "Your Friends Native" is the application(4) described above.
To run application (3), you need a Facebook account and add it to your account by accessing here.

Google Gadget and OpenSocial

2008-07-05T20:33:00.000-04:00

Opensocial API is an open set of APIs which can be used to access various social network applications. Currently, social network apps which support OpenSocial API are listed here.

Google Gadget:
Official specification of Gadget server/container is here(http://code.google.com/apis/gadgets/docs/spec.html). It includes abundant information about how gadget container handles gadget requests from clients. The basic idea is gadget container processes gadget .xml definition file and generates corresponding HTML+CSS+Javascript and then transferres the result to end users. Variety of functionalities gadget deveoper can utilize depends on gadget container. Different gadget containers may provide different set of features. However, some APIs are mandatory and must be supported by all compliant implementations. Javascript gadget specification is here(http://code.google.com/apis/gadgets/docs/reference/). Gadget XML reference is here(http://code.google.com/apis/gadgets/docs/reference.html). This site contains many useful hints about how to develop and test Google Gadgets.

There are two set of Gadget APIs:Legacy API and gadgets.* api(used by Open Social spec).

Gadget Host:
If you don't have a web server to host the gadget XML file, Google Gadgets Editor is a tool provided by Google which can be used to edit and host gadgets. You can test your gadgets in GGE withoug logging in Google. But you cannot save your gadgets without logging in. You can add Google Gadget Editor gadget to your iGoogle homepage by accessing this site. It seems that only legacy gadget APIs are supported in preview mode by GGE. GGE just provides very simple functionalities. You can use it just as a hosting service which stores your gadget spec files. If you want more functionalities like version control, you can use Google Code Hosting(http://code.google.com/hosting/) or other full-fledged hosting services. This site contains detailed information about how to use Google hosting services.

Legacy Gadget Container:
Besides spec files, gadget container is necessary to run your gadgets. To test legacy Google Gadgets, you can GGE or iGoogle content directory. Here contains instructions about how to publish your gadgets to iGoogle content directory. You can update your submitted gadgets using this site. NOTE: iGoogle content directory only applies to legacy gadgets.

Open Social Gadget container:
To test new google gadgets(using gadgets.* API), you should use a certain container listed here. Note: two services from Google are in the list. However, to use them, you must sign up for sandbox access besides regular logging in. To sign up sandbox access of iGoogle, visit this site(http://www.google.com/ig/sandbox). To sign up sandbox access of Orkut, visit this site. These two services are still in beta phase and thus provide limited functionalities.

Resources:
iGoogle Dev: http://code.google.com/apis/igoogle/docs/igoogledevguide.html
Orkut Dev: http://code.google.com/apis/orkut/docs/orkutdevguide.html

Interaction between Facebook and Ringside

2008-07-04T15:09:00.001-04:00

(1) Can Ringside use applications of Facebook?
Generally not. To use applications of Facebook, one of the two requirements must be satisfied:
<1> You know the callback URL of the application.
If you know the callback URL, you can create a new application on Ringside serve which points to the same callback URL. However, as far as I know, you have no way to figure out the callback URLs of Facebook applications developed by others.
<2> Facebook exposes a "proxy URL".
In this case, Facebook does not need to expose callback URL of an application. Instead, Facebook acts as a Proxy server and sits between Ringside server and your app. Then you points "Callback URL" of your Ringside app at the Facebook "proxy URL". So the communication style is:
your Ringside app <=> Ringside server <=> Facebook <=> Facebook App.
Facebook is a relay station which connects Ringside server with Facebook applications. However, as far as I know, currently Facebook does not support the "proxy URL".

(2) Can Facebook use applications of Ringside?
Yes. Ringside supports the second method described above.
If you has built an application on a Ringside server and you want to that application to be accessed in Facebook, you should:
[*] add a new application in your Facebook account;
[*] copy configuration information of your Ringside app to the newly created app in Facebook.
[*] set Callback URL(in Facebook setting) to point to your Ringside app. Note: there is a specific format of "proxy URL" of apps in Ringside: http://ringside_server:port/trust.php/facebook/canvas_path. You should replace canvas_path with your real value. It should be the canvas path you set in Ringside server rather than Facebook.
Note that that app is run inside of backend Ringside server instead of Facebook. So the app generates all output based on information stored on backend Ringside server. For example, if the app returns a list of your friends, you will see friend list of the profile you set up on Ringside server. In other words, you will not see your Facebook friend list.
Architecture:

<1> browser sends request to Facebook
<2> Facebook sends request to "Callback URL" (format is: http://ringside_server:port/trust.php/facebook/canvas_path)
<3> Ringside server sends request to your app server
<4> Your app makes calls to Ringside server and integrates response into output
<5> Your app returns final output to Ringside server
<6> Ringside server passes the output to Facebook
<7> Facebook returns response to client.
One issue is: if response in step <5> contains FBML segment, who will render the FBML? Ringside server or Facebook? I guess Facebook renders FBML into HTML. In other words, Ringside server just acts as a pure proxy and it does not render inbound messages. Maybe I am wrong, I am not 100% sure about my guess.

(3) More possibilities
Actually, your app can makes use of all resources that you have privilege to access. No matter where you deploy your app, your app can makes API calls to Facebook, Ringside server and open social containers(e.g. Orkut, hi5...).
Architecture:

In this way, you can integrate different resources and produce better result.

Facebook Application Development and Concepts

2008-07-03T22:35:00.000-04:00

Developing a simple application for Facebook platform is not as easy as developing a "hello,world" application in C/C++ or Java etc.
Here is a step-by-step tutorial about how to build a Facebook app. It is exhaustive and I just supplement two cents.

Resources:
Description about the basic architecture:
http://wiki.developers.facebook.com/index.php/Random_questions
More resources about how to build Facebook Apps:
http://www.merchantos.com/makebeta/facebook/facebook-php-tutorial/

First, it is necessary to introduce the basic concepts. These concepts confused me once upon a time.

Profile Action
When you browse profile of a user, you should see following layout:

The entries within red box are called profile actions. One way to set profile action of your app is through app setting interface provided by Facebook. Following entry "Default Action FBML" should be displayed:

You can make use of fb:profile-action to set profile action.
Another way to set profile action is use API: Profile.setFBML.

Profile Box
On your profile page, there is a small box for every application you have added. This small box is called profile box which is demonstrated below.

You can set content of profile box of your app by accessing app setting page:

Moreover, you can set profile box by using API: Profile.setFBML.
Note: Profile.setFBML can be used to set both profile box and profile action.

Cache in profile action and profile box:
Setting of profile action and profile box is cached in Facebook server. See architecture here.

<1> A user sends requests to Facebook through profile action or profile box.
<2> Facebook immediately returns cached response to the user. In other words, Facebook does not communicate with your app server instantly. The cache is for per user per application.
Update of profile action/box is done asynchronously. Usually, when a user requests your canvas page, profile action/box will be updated if related code is contained in your canvas page.

Canvas Page
When a user click your app in the left navigation bar, your app will be put into canvas page which is shown below:

Facebook retrieves content of the page and renders it in canvas page. By setting Canvas Page URL, you tell Facebook that your app should be accessed through the specific URL. By setting "Callback URL", you tell Facebook where content should be retrieved. Facebook platform is like a proxy. When a user uses your app by accessing "Canvas Page URL", Facebook determines the true URL that should be visited. Then Facebook sends request to the URL and renders response in Canvas Page.

How does Facebook handle your app in canvas page?
See here. I briefly describe the procedure:

<1> browser sends request to facebook platform.
<2> Facebook platform sends requests to your app server
<3> Your app makes API calls to Facebook and integrates response into output
<4> Your app returns output(html segment with or without FBML)
<5> After facebook gets response from your app server, Facebook renders FBML to HTML and displays result in client browser.

Add a new app
After you add developer application into your facebook account, you should be able to create your own applications. There are many options you can set to customize your Facebook app, the interface looks like this:

And this site contains detailed information about almost every field:
http://www.marketing-ninja.com/facebook-app/starting-your-first-facebook-app-demystifying-application-form-field-by-field/

You should have a server which will host your Facebook app. If you user Apache http server, you need to configure the server so that when a directory(not a file) is requested index.php will be returned(if it exists).
In my Apache httpd server, related configuration is like this:

<IfModule dir_module>
    DirectoryIndex index.html index.php
</IfModule>

Ringside application development

2008-07-02T18:03:00.000-04:00

Table of Content

What Ringside server can do?
Client app development
- How to add and set up a new app
- Develop client apps in PHP
Ringside server extension development
- API extension (add more APIs you like)
- Tag extension (customize your own tags)

Ringside Social Application Server seems a promising project. However, because it is still in beta development, online documents are not well organized. A better directory layout may help a lot. After reading the documents and trying Ringside server myself, I got some experience which is shared in this post.

What Ringside server can do?

(1) Open Social container.
It means all open social applications can be run on Ringside server. Of course, the open social applications should not use extensions provided a specific container( e.g. iGoogle, Orkut ) if you want these apps to run on Ringside server.
Note: Gadgets in Google content directory(repository of traditional gadgets) can not be used on Ringside server because traditional Gadget API is different from new Gadget spec used by Open Social API.
(2) Facebook platform mock
It supports FBML and Facebook API and adds some extensions, which means you can use the same function names and markups of Facebook to write your Ringside apps.
(3) Extensibility
Because Ringside server is open source, users can customize and extend Ringside server in the way they wish. For example, users can add their new APIs and extend XML tags to provide customized tags.

How to add and set up a new app?

Add a new app:
First click "Developer" tab, and in the main panel click "Create a New Application", following picture should be displayed:

When you create a new application in Ringside, the procedure is well similar to that of Facebook. The configuration options are almost the same as well. Look at below picture:

One big difference is highlighted in the picture. In Facebook, you can just choose your app type between "Use FBML" and "Use IFrame". However, in Ringside server, one more option("Use Open Social") is supported. The key thing you should configure is "Callback URL" which points to the location of your application. For option "Canvas URL", you just need to specify a unique name and you must not specify the full-path URL. Prefix is implicit which is http://ringside_server_domain:port/canvas.php/. So if you specify "myapp" as your "Canvas URL", the URL to access your app becomes http://ringside_server_domain:port/canvas.php/myapp.
Note!!! When you set "Callback URL", you had better not use loopback domain or ip(e.g. localhost, 127.0.0.1) as host. If type of your app is "Use FBML", it may work well. In this mode, the output content is fully contained in the response. However, if type of your app is "Use IFrame", your app may not work. In this mode, a frame element is embedded in returned html and src attribute is set to point to the "Callback URL". As a result, the browser will send requests to its local host (the host a user is using to try your app) instead of the server which hosts your app. After you deploy and publish the app, using loopback address will definitely cause problems. To make your app work both in FBML mode and IFrame mode, loopback address should not be used!!!

Develop your own Ringside applications in PHP

See instructions here. After you get PHP library, generally you need to modify your php.ini and append path of the library to variable include_path so that PHP interpreter knows where to look for Ringside PHP library.
You applications do not need to be hosted on the same host as Ringside server. Ringside provides a PHP client library which can be used to ease PHP development. Ringside PHP client library is based on Facebook PHP client library. But Ringside wraps the code in original Facebook PHP client library. In original Facebook PHP client library, the rest server which you send requests to is hard-coded in the source code as "http://www.facebook.com". There is a class in Ringside library called "RingsideApiClients" which is a subclass of class "Facebook" and overwrites most of the inherited functions. Besides, Ringside adds some extra APIs which are mainly contained in class "RingsideApiClientsRest"(It is a subclass of class "FacebookRestClient").
So, you should use class "RingsideApiClients" to send requests to backend Ringside server. One thing your must do is configure the Ringside server URL so that your app knows where requests should be sent. There are two ways that I know:
(1) When you create an instance of RingsideApiClients, you can send weburl, serverurl and socialurl as parameters of contructor.
Prototype of the constructor is :
public function __construct($api_key, $secret, $webUrl = null, $serverUrl = null, $socialUrl = null, $client = null ).
(2) Change static variable in class RingsideApiClientsConfig.
Sample usage:

RingsideApiClientsConfig::$webUrl= ringside_web_url;
RingsideApiClientsConfig::$serverUrl = ringside_server_url;
RingsideApiClientsConfig::$socialUrl = ringside_social_url;

Extend API provided by Ringside Server

Good resource (1), (2).

Add your own tags to Ringside server

Good resources (1), (2).

Ringside Networks Social Application Server on Linux

2008-07-01T17:14:00.000-04:00

This post is based on official documents from Ringside, and describes steps to set up Ringside Networks Social Application Server.
(1) Download
First, download the app server from here. In the distributed package, all dependent packages/libraries are included. Main additional components include: Apache2, PHP5 (core package and some extensions), Mysql and some unix/linux tools(e.g. curl, tidy and their libraries). So size of the whole package is sort of big. But it is an all-in-one package so that you can easily make it work without much configuration. Installation directory structure is explained here.
If you don't want the all-in-one package, check instructions here to individually set up your environment.

(2) Install
A GUI installation wizard is provided. You can get instructions here. The process is easy.

(3) Config
You can change various apache setting by modifying following files:
(*)%RNAS_ROOT%/apps/phpmyadmin/conf/phpmyadmin.conf
//This conf file is related to using of phpmyadmin. By default, users can only access phpmyadmin from IP 127.0.0.1. You can modify it
//if you want to use phpmyadmin from remote client.
(*)%RNAS_ROOT%/apps/ringside/conf/ringside.conf
//ringside related conf. It sets where the document root is... Generally, this file should not be modified.
(*)%RNAS_ROOT%/apache2/conf/httpd.conf
//General apache http server configuration is in this file. If you want to change the port, set it here.

(4) Run
Let me assume you installed Ringside Networks Application Server into directory %RNAS_ROOT%. Under directory %RNAS_ROOT%, you should see a script called "ctlscript.sh" which can be used to start/stop/restart server components. Sample commands are shown below:
./ctlscript.sh start                    //start both apache and mysql
./ctlscript.sh stop mysql           //stop mysql
./ctlscript.sh status apache       //get status of apache server.
Official document is here which has more information.

(5) Setup/Initialization
First, you need to access http://your_ip_or_domain:port/setup.php. Default port is 8080. Then the server will carry out some setup work. For example, a database called ringside is created...
You should see following page:

(6) Test/Using
Main page: http://your_ip_or_domain:port
Phpmyadmin access: http://your_ip_or_domain:port/phpmyadmin.
Login: http://your_ip_or_domain:port/login.php. When you run initial setup in step (5), a user is automatically generated for you. See content in the green box in last picture.

(7) upgrades
[*] Upgrade via PEAR:
You can update Ringside source via PEAR. Instructions are described here.
[*] Check out latest code:
You can checkout latest Ringside code and build your own development/deployment environment. You should install the all-in-one package first. By following instructures here, you substitute new version of Ringside for the original one which is included in the all-in-one package.

(8) Remove
Use executable uninstall. Check instructions here.

Client side techniques

2008-06-19T23:30:00.000-04:00

Namespace mimic in Javascript (similar to technique used by YUI):

namespace:   function(   sNameSpace, obj   )   {  
    if   (!sNameSpace   ||   !sNameSpace.length)   {  
        return   null;  
    }  
    var   levels   =   sNameSpace.split(".");  
    var   currentNS   =   CGL;  
    //   CGL   is   implied,   so   it   is   ignored   if   it   is   included  
    for   (var   i=(levels[0]   ==   "CGL")   ?   1   :   0;   i<levels.length;   ++i)   { 
        if( arguments.length == 2 && i == levels.length - 1)
	    currentNS[levels[i]]   =   obj;
	else
	    currentNS[levels[i]]   =   currentNS[levels[i]]   ||   {};  
	currentNS   =   currentNS[levels[i]];  
    }
    return   currentNS;  
}

By default, following namespaces are supported:
CGL.mashup.resourcetype:
This object maintains resource types we support. Currently, text, image and video are supported.
CGL.mashup.addr:
This object maintains base path for our applications.
CGL.mashup.sites:
This object maintains all backend sites we support. Currently, Youtube and Flickr are supported.
CGL.mashup.log:
This object maintains request history so that in the future we can reinvoke a certain old request.
CGL.mashup.init:
All initialization work is done here.
CGL.mashup.urlbuilder:
This object provides functions which can ease construction of request URL to access various resources.
CGL.mashup.service:
This object provides functions which send various requests (e.g. get user information) to server side.
CGL.mashup.helper:
This object includes some utility functions.
GL.mashup.htmlformatter:
This object contains functions that can be used to display response in a specific format .
CGL.mashup.jsonresponse:
This object handls JSON responses from server and parses the content.
CGL.mashup.opmapper:
This object maps operation names to concrete function implementations. Callback can be specified as well.

Web2.0 Integration Summary

2008-06-19T22:21:00.001-04:00

The server-side architecture is described here:
http://zhenhua-guo.blogspot.com/2008/05/mashup-architecture.html.

Authentication: OpenID which is described here (http://zhenhua-guo.blogspot.com/2008/05/openid.html).
Authorization: Although OAuth seems promising, it has not been supported by large web 2.0 applications. So, I make use of various authorization mechanisms of different applications.

RESTful web services are used instead of SOAP-based web services. URL pattern used to access various resources is described here(http://zhenhua-guo.blogspot.com/2008/05/url-pattern-in-mashup.html).

At client side, Ajax is used to enhance user experience. Client side functionality is described here(http://zhenhua-guo.blogspot.com/2008/06/client-side-techniques.html).

OAuth

2008-05-25T23:32:00.001-04:00

In my mashup framework, OpenID is used to do authentication which provides a way to make sure the users are really who they claim to be. The mechanism is that the users prove that they own/control the OpenID they provides. Here is what I wrote about OpenID.

However, OpenID just solves authentication. After users are authenticated, OpenID does not specify how the third-party apps access the protected data on service providers. And the requirement that third-party apps can access protected data stored on service providers on behalf of users is not imaginary, but real. In my mashup framework, I need to manipulate data on behalf of users which needs authorization besides authentication. Youtube and Flickr both provide ways for third-party integration. They provide similar authorization procedures (I am using this mechanism right now) which fundamentally match principles of OAuth specification.

OAuth provides a way for users to grant access to third-party apps. The procedure is pretty much the same as OpenID. What is different are parameters in requests and responses. OAuth is an independent specification which means it is not an extension of OpenID. But, in my opinion, it may be better to combine these two together because they are similar to each other. In this way, user experience can be improved because authentication and authorization can be done in the same pass instead of two. However, some sites may just support authentication service or authorization service (in terms of support of open standards like OpenID, OAuth). So how to combine them together without losing flexibility is an issue.

One problem that OAuth does not address is fine-grained access control. I think this issue must be addressed in the long run. However, nowadays, the goal is too far to reach. Actually, current version of OAuth is not supported widely.

DataPortability is a new organization which aims to promote data portability. Of course, it includes OpenID and OAuth with respect to authentication and authorization data portability. I am not sure which ones will survive and stand out eventually, but this is a good signal that data portability is being considered as a critical issue and I hope it will be addressed in the near future.

Authentication and Authorization in Mashup

2008-05-24T01:27:00.000-04:00

Authentication
Currently, I don't build my own independent user authentication system. I make use of OpenID to utilize existing authentication systems, such as Yahoo Flickr, Google Blogspot, LiveJournal... You can check whether you have already had an OpenID here. I wrote a simple introduction. In essence, we delegate the task to existing authentication systems.
After a user is authenticated successfully, OpenID is used to track the user. Servlet session technique is used to associate session id and user information.

Authorization
Currently, I don't ask end users for their usernames and passwords in existing services (e.g. youtube, flickr) because it requires deep trust of end users. The user id redirected to an authorization web page hosted by the destination service(e.g. youtube). If the user accepts the request, then the user browser will be redirected to my mashup application. Generally, an auth token is appended to the URL which will be used in following requests.
Currently, for every backend service, I request the user to authorize the most powerful privilege level. For example, if the service supports READ and WRITE permission control, my application will request WRITE permission. If the service supports READ, WRITE and DELETE permission control, my application will request DELETE permission. This is not always a good strategy. But because of diversity of permission control systems, I must choose the mechanism which can be used in every possible service.
Maybe in the future, I can come up with a better framework with fine-grained access control.

URL Pattern In Mashup

2008-05-24T00:33:00.001-04:00

URL Pattern

Note: All variables enclosed in ‘{’ and ‘}’ must be substituted with real values.
GET means HTTP GET request type and POST means HTTP POST type.

/cgl/feeds/api/{category}/{service}/users/{userid}
GET:
Get information of a user.

/cgl/feeds/api/{category}/{service}/login
GET:
Login the service.

/cgl/feeds/api/{category}/{service}/users/{userid}/resources
GET:
Get all resources of the user.
POST:
Add a new resource.

/cgl/feeds/api/{category}/{service}/users/{userid}/resources/resourceid
/cgl/feeds/api/{category}/{service}/resources/resourceid
GET:
    Get information of a specific resource.
DELETE:
    Delete the corresponding resource.
PUT:
    Update the corresponding resource.

/cgl/feeds/api/{category}/{service}/resources/resourceid/comments
GET:
Get all comments of the resource.
POST:
Add a new comment.

/cgl/feeds/api/{category}/{service}/resources/resourceid/rating
GET:
Get rating of the resource.

/cgl/feeds/api/{category}/{service}/tag/{tag}?perpage={perpage}&page={page}
GET:
Get records by tag. Parameter perpage indicates how many resources you want to retrieve.Parameter page indicates which page you want to retrieve.

/cgl/feeds/api/{category}/{service}/statistics/{standard}? perpage={perpage}&page={page}
GET:
Get records by standards.
E.g. most viewed, most recent…

Example:

/cgl/feeds/api/videos/youtube/tag/soccer
get resources tagged as soccer.
/cgl/feeds/api/videos/youtube/statistics/most_viewed
get resources that are most viewed.
/cgl/feeds/api/pictures/flickr/users/testuser/resources
get resources belonging to user testuser.

Javascript Templates

2008-05-23T17:01:00.000-04:00

Javascript Templates(http://code.google.com/p/trimpath/wiki/JavaScriptTemplates) is a sub project of project Trimpath. It is a client-side template engine whose functionality is similar to server-side Velocity and Smarty engines. It uses special markup syntax. Besides simple template matching and replacing, it also supports simple control flow, loop and variable declarations... Now I summarize the syntax:
Statements:
(1) Variable Declaration

Syntax	Example
{var variable} {var variable = value}	{var price} {var price = 10}

Note: No ending ';' is needed.

(2) Control Flow

Syntax	Example:
{if expression} {elseif expression} ... {else} ... {/if}	{if price > 10} {elseif price <20}... {else}... {/if}

Note: No surrounding brackets are needed in test expression. No white space between first '{' and 'if'.

(3) Loop

Syntax	Example
{for varariable in list} statements go here {/for}	{for price in pricelist} ${alert(price)} {/for}

(4) Expression

Syntax	Example
${expression}	${car.brand}
${expression\|modifier}	${car.brand\|capitalize}
${expression\|modifier1\|modifer2}	${car.brand\|default:"chevy"\|capitalize}
...	...

Value of the expression will be returned.

Note:
Statement for( ; ; ) is not supported.
These statements are parsed and processed by Javascript Template Library instead of javascript itself.

More:
(5) CDATA Text. Tag is cdata.
CDATA Text will be ignored and will not be processed.
(6) In-line javascript block. Tag is eval.
Multiple line javascript code can be put in a single block to evaluate.
(7) If you have multiple-line js code, you can use minify to tell processor to treat them as a whole.
For example:
<input type="button" onClick="{minify} Javascript event
handling code...
{/minify}">

Note: eval and minify are different althought they are sort of similar.

How to process templates?
API: http://code.google.com/p/trimpath/wiki/JavaScriptTemplateAPI.
Commonly used functions:
(1) TrimPath.parseDOMTemplate ( elementId, contextObject )
This function retrieves content of the element corresponding to the first parameter. It should be a textarea element and its should be hidden.
The second parameter is a context object and it will be "merged" with the template. For example, if a template refers to ${car}, then the processor tries to access contextObject.car.
(2) TrimPath.parseTemplate ( templateContentStr )
This function accepts a string as template instead of a dom element id.
You can parse a string template in this way:
TrimPath.parseTemplate( stringvariable ).process( contextObject ).

Variable Resolvation Precedence:
There are three kinds of scopes:
(1) Processor scope: it consists of all template statements.
(2) Context object scope: All properties of the context object.
(3) Local scope: local variables whic are not defined in template.

Case 1: In template expression and {eval}:
Processor scope is checked first, then context object is checked. At alst, local scope is checked.
For example:
    var data={car : "chevy"};
    var str = '${alert(car)}';
    str.process(data);        //alert "chevy"

var str2 = '{var car = "ford"} ${alert(car)}''
str2.process(data); //alert "ford"

    var car = "toyota";
    var str3 = '${alert(car)}';
    str3.process();      //alert "toyota"

Case 2: All variables defined in template do not have effect in local scope.
For example:
    var str = '{var car="chevy"}${alert(car)}';
    str.process(data);        //alert "chevy"
    alert(car);                   //variable 'car' is not defined.

Template

2008-05-18T18:22:00.001-04:00

Here is link which suverys browser-side templating: http://ajaxpatterns.org/Browser-Side_Templating.
The goal of browser-side templating is to separate presentation and logic. And in my application, I think it will be helpful to use browser-side templating to ease the development. Currently, JSON messages are received from server and then formatted accordingly to display them to end users.

In next step, I would like to try JTL(Javascript Template Library) http://code.google.com/p/trimpath/wiki/JavaScriptTemplates.

OpenID

2008-05-12T18:12:00.002-04:00

Some text is excerpted from Official Specification.

[Resources]

Official web site: http://openid.net
Authentication specification: http://openid.net/specs/openid-authentication-2_0.html
Following is a very good and detailed tutorial about how to develop a web site which makes use of OpenID: http://www.plaxo.com/api/openid_recipe.
Libraries written in various languages: http://wiki.openid.net//Libraries.
This web site lists many web sites which support OpenID: http://openiddirectory.com/.

[Introduction]
       Previously, when we visit some web sites and want to leave comments, generally we need to sign up an account at first. This is kind of annoying because we might just surf the web and we might not visit the website again in the future. Applying for a new account is not a big deal.
       By using OpenID, we can be authenticated in a new web site by signing in an OpenID provider (e.g. Blogger, Flickr, Youtube…) with existing account. In other words, we could use the same account to log in to multiple web sites/applications. Generally, most of users have account in prevailing OpenID providers. These providers (e.g. Flickr, Youtube ...) provide their own core compelling services (e.g. video sharing, picture share…) besides OpenID support. Besides, there are some new sites which mainly provide OpenID support (e.g. myopenid.com,).
       An assumption is that OpenID providers are trusted by various OpenID-aware web applications.
       The advantage of OpenID is that:
       Users can use OpenID to log in to multiple web sites which support OpenID without having to apply for an account. As a result, users don’t need to remember large number of usernames and passwords, which improves user experience greatly.

The mechanisms used to achieve the goal are: redirection and callback.

[Architecture]

1) Background
There are two kinds of communication: direct request and indirect request.
Direct request: one party sends request directly to another party to get response. The message must be encoded as a POST body.
Indirect request: one party sends request to another party by redirecting user agent to the destination party with request data (query string or POST body).
There are two mechanisms which can be used to send indirect request: HTTP indirection or form redirection.
HTTP redirec:
“Data can be transferred by issuing a 302, 303 or 307 HTTP redirect to the end user’s agent. The redirect URL is the URL of the receiver with the OpenID authentication message appended to the query string.”
HTTP Form redirection
A HTML page which contains an HTML form element can be returned to user. And the form contains a mapping of keys to values. Javascript can be used to automate submission of the form to implement redirection.
More info: http://openid.net/specs/openid-authentication-2_0.html#anchor6

2) Procedure
First, the Relying Party presents the user with a form that has a field for entering User-Supplied Identifier. The form field’s name should be “openid_identifier” so that user agents can detect it automatically and provide some extra functionality.

Steps:

(1) The user submits the from to Relying Party
After receiving data submitted by user, Relying Party normalizes the input data and then discovers OP endpoint URL based on user-supplied identifier.

(2) This step is optional. Relying Party can build an association with OP.
An association establishes a shared secret between them which is used to verify subsequent protocol messages and reduce round trips.
More info: http://openid.net/specs/openid-authentication-2_0.html#associations

(3) Relying Party sends authentication request to user agent.
User agent sends authentication request to OP. Then user is authenticated by the OP.
These two sub steps jointly do this: Relying Party send an authentication request to the OP to obtain an assertion. This is an indirect request. So the request actually is sent by Relying Party to user agent and then redirected to OP.
Generally, OP will display a web page which enable user to accept or deny access request from Relying Party.

(4) OP sends authentication response to user agent and then user agent sends response to Relying Party.

(5) After receiving authentication response from OP (indirect communication), Relying Party verifies it and judge whether it is a positive assertion or negative assertion.

More info: http://openid.net/specs/openid-authentication-2_0.html#responding_to_authentication

[More]
Besides authentication, personal information (e.g. gender, location, age …) can be transferred between Relying Party and OpenID provider as well. Then Relying Party can pre-fill some registration fields so that users don’t need to type the same information many times. Of course, Relying Party needs to convert information from OpenID provider to its own format.

[Delegation]
Delegation is also possible: http://simonwillison.net/2006/Dec/19/openid/. It means users may want to delegate their OpenID to another provider behind-the-scenes. An example from http://www.plaxo.com/api/openid_recipe:
       “If I try to sign up with the OpenID josephsmarr.com, I may have actually delegated that URL to a different OpenID like jsmarr.myopenid.com, and when the provider returns to you to complete authentication, you need to remember that I wanted to sign up as josephsmarr.com and not jsmarr.myopenid.com.”
Key thing is that you add following two lines of HTML code to your delegation HTML page:
       <link rel="openid.server" href="http://openid.server">
    <link rel="openid.delegate" href="http://delegation.site">
You should replace the value of href attributes accordingly.

This mechanism actually adds an additional layer which separates Relying party and OpenID provider. Relying Party gets OpenID endpoint URL by accessing your delegation HTML page. After getting value of open.server and openid.delegate, Relying Party can sends authentication request to corresponding OpenID provider. You can easily switch your preferred OpenID provider by modifying value of attribute href in link elements.

[Myth]
       OpenID is a mechanism which can authenticate users by delegating it to corresponding OpenID provider. So it is not appropriate to use it directly in a third-party application which sits on top of an existing service and wants to access the backend service on behalf of the user. For example, if a web site makes use of Youtube API to do some cool things, it necessarily needs users’ information to utilize Youtube on behalf of the user. OpenID does not solve this problem.
       But, in my opinion, this can be done by extending OpenID specification. In authentication response, OpenID provide can include an auth token which can be used as a handle by Relying Party to access service provided by the OpenID provider on behalf of the user. In addition, access level (Read, Write, Delete …) must be negotiated as well.
       However, the extension deviates from gist of OpenID because authorization delegation is involved besides authentication.

Youtube API usage

2008-05-10T16:14:00.002-04:00

Authentication

       http://code.google.com/apis/youtube/developers_guide_protocol.html
       It is recommended that you include the proper authentication headers in all of your requests even if those requests do not explicitly require authentication.
       A request must include the X-GData-Key and Authorization headers.
       The X-GData-Key header specifies your developer key, a value that uniquely identifies your application(s). The Authorization header specifies a token that you obtain for each user using one of two authentication schemes, AuthSub or ClientLogin. Headers are like this:

AuthSub

ClientLogin

Authorization:AuthSub token=<authentication_token>

X-GData-Key: key=<developer_key>

Authorization:GoogleLogin auth=<authentication_token>

X-GData-Key: key=<developer_key>

(1) Functions supported in AuthSub
http://code.google.com/apis/accounts/docs/AuthForWebApps.html

(1.1) AuthSubRequest

A call to this method sends the user to a Google Accounts web page, where the user is given the opportunity to log in and grant Google account access to the web application. If successful, Google provides a single-use authentication token, which the web application can use to access the user's Google service data. This is done by using GET to a specific formatted URL.

Sample URL

https://www.google.com/accounts/AuthSubRequest?scope=http%3A%2F%2Fgdata.youtube.com&next=http%3A%2F%2Fzhguo.blogspot.com

(1.2) AuthSubSessionToken

A call to this method allows the web application to exchange a single-use token for a session token.

Sample request
curl https://www.google.com/accounts/AuthSubSessionToken --header ‘Authorization: AuthSub token="token"’

Response

Token=DQAA...7DCTN

Expiration=20061004T123456Z

(1.3) AuthSubRevokeToken

A call to this method revokes a session token. Once a token is revoked it is no longer valid.

Sample Request
curl https://www.google.com/accounts/AuthSubRevokeToken --header ‘Authorization: AuthSub token="token"’

(1.4) AuthSubTokenInfo.

A call to this method verifies whether a specified session token is valid and returns data associated with the token. This operation applies to both one-time-use and session keys.

Sample request

curl https://www.google.com/accounts/AuthSubTokenInfo --header 'Authorization: AuthSub token="CO7xhO-6GhDT5f2tAwd"'

Sample response

Target=http://www.yourwebapp.com

Scope=http://www.google.com/calendar/feeds/

Secure=true

(2) ClientLogin

http://code.google.com/apis/accounts/docs/AuthForInstalledApps.html

Before using ClientLogin, you must have an existing Google account. The POST request should be structured as a form post with the default encoding application/x-www-form-urlencoded. Parameters should be included in the body of the post.

Action URL parameter: https://www.google.com/accounts/ClientLogin

Sample request format:

POST /accounts/ClientLogin HTTP/1.0

Content-type: application/x-www-form-urlencoded

accountType=HOSTED_OR_GOOGLE&Email=jondoe@gmail.com&Passwd=north23AZ&service=cl&source=Gulp-CalGulp-1.05

CURL command
       curl    --location https://www.google.com/youtube/accounts/ClientLogin
                   --data 'Email=username&Passwd=password&service=youtube&source=Test'
                   --header 'Content-Type:application/x-www-form-urlencoded' -i

Sample Response:

SID=DQAAAGgA...7Zg8CTN

LSID=DQAAAGsA...lk8BBbG

Auth=DQAAAGgA...dk3fA5N

(3) Difference of two mechanisms

To use ClientLogin, a third-party application must own Youtube’s username and password of every user. Then the third-party application can do anything the end user can do. This mechanism is convenient for third-party apps to use because it does not require much intervening from users. However, it requires trust of users who may be afraid that their person information is not stored appropriately.

To use AuthSub, a third-party does not need to own Youtube’s username and password of user. Instead, the user will be redirected to a web site from google to authorize third-party app’s access to user’s account. At first, third-party app is given a one-time access key which can be used to exchange a session key that can be used longer. This method is more secure for users at the cost of manually authorizing access requests.

Some development detail

(1) Some videos are not allowed to be embedded in third-party applications.

In the feed of these videos, mediagroup::content element does not exist!!! I manually construct URL by concatenating http://gdata.youtube.com/feeds/api/videos/ and video id and try to embed it into my app. As I expect, it does not work. However, I figured out a method which can work around this problem. I concatenate http://www.youtube.com/swf/l.swf?video_id= and video id and it works.

(2) Some videos are not allowed to be commented by some users. These videos should be handled carefully.

http://gdata.youtube.com/feeds/api/videos/gmNbl5ZTBZE

(3) Duplicate elimination

If you upload the same video clip more then once using different metadata, Youtube can figure it out and all uploads but the first one will be rejected. I am not sure how they compare two different video clips, maybe using MD5 to calculate a signature. The duplicate uploads are still displayed in your account. And all metadata is preserved except the video clip itself. The only difference between original upload and following duplicate uploads is that you can not play duplicate video clips.

As shown in above picture, metadata (e.g. description, tags, views, published time…) are still there. But the video clip itself is rejected

(4) OpenSearch support

If you search videos on Youtube, in returned feed openSearch extension is supported:

<openSearch:totalResults>274842</openSearch:totalResults>

<openSearch:startIndex>1</openSearch:startIndex>

<openSearch:itemsPerPage>25</openSearch:itemsPerPage>

(5) Navigation of many returned results.

In returned feeds, if not all results are returned, there should be a link element in the feed. It is like this:

Mashup Architecture

2008-05-10T14:50:00.001-04:00

Architecture:

Adapter:

TODO:
Integrate authentication architecture.

Add image support in web2.0 app

2008-04-10T23:36:00.002-04:00

I added image support in the web2.0 app. It means that users can uploads image files besides regular text files. Users can attach tags, description and name when uploading an image. Other users can post comment/rating about the image.

When an image is uploaded, it will be stored somewhere at server side. Moreover, it can be retrieved by accessing a URL. In other words, the server generates a URL which can be used to access that image. When a user tries to retrieve the image, it will be displayed in browser by using img elment.

Currently, I am using suffix of file name to judge type of record (image or text). The advantage is that user intervening is not needed. Drawback is that sometimes suffixes are not accurate. It may be better to use MIME type which is specified when user uploads an object.

Tomcat virtual/physical path mapping

2008-04-10T21:03:00.002-04:00

In my web2.0 application, users can upload files to server. I want to build mapping from virtual path to physical path. It means that the resources accessed are located in a separate directory instead of tomcat sub directory.

Context container can be used to achieve that goal. Official document is here http://tomcat.apache.org/tomcat-6.0-doc/config/context.html.
In my case, I modified configuration file server.xml and insert following text into Host element:

<context crosscontext="true" docbase="E:\\temp\imgService" path="/imgService"></context>

Actually, at first, I tried to modify context.xml instead of service.xml because it is less invasive. However, it didn't work and I don't know why.

A Simple Web 2.0 Framework

2008-04-10T00:41:00.001-04:00

I have completed a simple application which supports common web 2.0 features: rating, commenting, tagging.
Demo address is: http://156.56.104.196:8080/tagService/index_ui.html.

The basic unit in my system is called record. A record is abstraction of an object. Theoretically, it can be of any type: image, text, presentation... However, the key issue is how to present different types of records to end users. Text is easy to present and it can be displayed to users directly. For image, img element can be used but it requires that the image be stored somewhere on the website so that it can be accessed by visiting a URL. Currently, only text is supported. When a user retrieves some record, text content of that record is displayed directly in browser.

Operations:
(1) Add a new record
Users add a new record by uploading a file to server and specifying name, description and tags. Then a unique id is generated for this new record at server side.
(2) List all records
Get all records in the system.
(3) Get a record based on its id
According to record id, get corresponding id.
(4) Get records according to tag
According to user-specified tag, corresponding records are returned. Currently, users can only specify a single tag.
(5) Post comment and rating of a certain record
Users can post comments and ratings about records.
(6) Comment and rating of comment.
Besides comments of records, comments of comments are supported. It means that users can post comments about existing comments besides records.

Architecture:

Client sends request to a server which uses servlet to handles it. Then the servlet accesses a web service which provides web 2.0 functionalities.

Presentation location:
What is stored in backend database is raw data. And it does not contain information about how to present it to end users. E.g. layout, font-size...
There are two strategies to transform raw data into final presentation data: server-side servlet and client-side javascript.
(1) Server-side servlet

When servlet responds to user's request, a HTML document is returned. The HTML document does not only contains data but also contains presentation information. So it can be displayed directly. In this method, servlet formats the raw data received from web service into a corresponding HTML document.

(2) Client-side javascript

In this method, servlet just returns raw data in JSON format. It means returned data does not contain any presentation information. And client side javascript displays the raw data in a specific manner. Actually, to use this method, AJAX should be used. Object XmlHttpRrequest can be used to make asynchronous request without refreshing whole page.

Personally, I prefer the second method.
I remember there are some server-side Java libraries which can be used to make transformation execute with ease.

AJAX
    AJAX is used frequently. However, the format of request is not XML. I manually compose POST query in Javascript and then send it by using XmlHttpRequest. Actually, I use application/x-www-form-urlencoded which is the default data encoding of form submission. I chose this encoding because it is simple compared to the other encoding - multipart/form-data. To make it work well, the data must be escaped.
    Responses from server are in JSON format. So javascript decodes JSON string and dynamically modifies web page accordingly.
    An alternative method is to use XML-RPC.

Possible future work:
(1) User management
Currently, the system is open and everyone can use it anonymously.
(2) RESTful access
(3) Support image type in presentation layer.
(4) Persistence of server data.

Dynamic web page development

2008-04-09T19:05:00.002-04:00

Recently, I am working on creating a simple web2.0 framework which enables commenting, rating and tagging. Of course, there are two parts: client side and server side. Client side programming using Javascript almost kills me because of incompatibility among different browsers.

(1) When content of a div element exceeds size of the div box, different browsers responde in different ways.
In IE, you can use white-space and word-break to force word wrap.
in FireFox, unfortunately there is not an easy way to do it. You can use overflow property to force a scroll bar. However, this is not what I want. I want the text wrapped automatically.

(2) Following is a piece of javascript:

var inputele = document.createElement('input');
inputele.setAttribute('type', 'button');
inputele.setAttribute('name', 'testbutton');
inputele.setAttribute('value', 'click me');
inputele.setAttribute('onClick','javascript:alert(\'hello,world\');return false;');

In FF, it works well. But in IE, it does not work. It seems that setAttribute can not be used set event handlers.
Instead, you should use attachEvent.

inputele.attachEvent('onclick', function(){ ... });

Or:

invokeStr = "calculate(1,2)";
inputele.attachEvent(onclick', function(){ eval(invokeStr); }

Besides, you can not use setAttribute to modify style property in IE.
In IE:

element.style.display = "block";
element.style.position ="relative";
...

In FF:

style = 'display:block; position:relative;';
element.setAttribute('style', style);

Moreover, name property of a newly built input element can not be modified after the element is created. Official description is here http://msdn2.microsoft.com/en-us/library/ms536389(VS.85).aspx. Stupid, right?
It means in IE you must use:

var inputele = document.createElement('<input type="button" name="buttonname" value="click me"></input>');

Many many others......

Currently, I make use of object navigator to judge type of browser. For IE, navigator.appName is "Microsoft Internet Explorer". Actually, navigator.userAgent provides detailed information about browser.

HTTP Form POST Encoding

2008-04-06T16:44:00.001-04:00

Form is a common way to submit requests to server with various parameters. In W3C HTML4(http://www.w3.org/TR/html4/interact/forms.html#h-17.13.4), form element has a special attribute "enctype" to indicate how to encode form data for submission.

(1) application/x-www-form-urlencoded (default type)

Control names and values are escaped. Space characters are replaced by `+', and then reserved characters are escaped as described in [RFC1738], section 2.2: Non-alphanumeric characters are replaced by `%HH', a percent sign and two hexadecimal digits representing the ASCII code of the character. Line breaks are represented as "CR LF" pairs (i.e., `%0D%0A').
The control names/values are listed in the order they appear in the document. The name is separated from the value by `=' and name/value pairs are separated from each other by `&'.

Examples
home=Cosby&favorite+flavor=flies

(2) multipart/form-data
From W3C specification:
"he content type "application/x-www-form-urlencoded" is inefficient for sending large quantities of binary data or text containing non-ASCII characters. The content type "multipart/form-data" should be used for submitting forms that contain files, non-ASCII data, and binary data.

The content "multipart/form-data" follows the rules of all multipart MIME data streams as outlined in [RFC2045]. The definition of "multipart/form-data" is available at the [IANA] registry.

A "multipart/form-data" message contains a series of parts, each representing a successful control. The parts are sent to the processing agent in the same order the corresponding controls appear in the document stream. Part boundaries should not occur in any of the data; how this is done lies outside the scope of this specification.

As with all multipart MIME types, each part has an optional "Content-Type" header that defaults to "text/plain". User agents should supply the "Content-Type" header, accompanied by a "charset" parameter.

Each part is expected to contain:

a "Content-Disposition" header whose value is "form-data".
a name attribute specifying the control name of the corresponding control. Control names originally encoded in non-ASCII character sets may be encoded using the method outlined in [RFC2045]. "

Examples:

    Content-Type: multipart/form-data; boundary=AaB03x
   --AaB03x
   Content-Disposition: form-data; name="submit-name"

   Larry
   --AaB03x
   Content-Disposition: form-data; name="files"; filename="file1.txt"
   Content-Type: text/plain

   ... contents of file1.txt ...
   --AaB03x--

Workaround of bug in integration of db4o and axis2

2008-03-27T00:27:00.001-04:00

I have been trying to solve the problem described in my last post.

I tried different versions of tomcat and Db4o which did not help me out. I posted this problem in forum of Db4o to ask for help. However, no one supplied right solution. I tried countless possible solutions. Finally, I got it to work. Following is the procedure how I discovered my solution.

To enhance speed of development , I tried to find some software support. I have been using Eclipse as my IDE. So naturally WTP(Web Tool Platform) is a great plug-in to support development of web applications in Eclipse. WTP provides support for Tomcat and Axis2 which are being used in my project. Here(http://www.eclipsecon.com/webtools/community/tutorials/BottomUpAxis2WebService/bu_tutorial.html) is a great tutorial about how to deploy and start Axis2 web service in Eclipse.

Then I doubt that the problem maybe results from Axis2. So I decided to write a simple servlet to do similar job. In other words, Axis2 was not used. And test result showed that everything worked correctly with Db4o. So this meaned that it is highly possible that Axis2 is cause of the problem. I did further investigation to uncover what is wrong under the hood. I built the same Axis2 service in Eclipse and deployed the service to Eclipse's temporary publish directory. Surprisingly, it worked!!! However, if I built web service using ADB(Apache Databinding) in Axis2, wrapped it to a .aar archive file and deployed it into specific directory (services), it did not work!!! So, I made sure that the procedure of deployment of Axis2 web service in Eclipse MUST be different from what I did before. Finally, I found that Eclipse does not wrap web service implementation into a .aar archive file. Instead the deployment directory layout is:

axis2
    - WEB-INF
        - lib
        - conf
        - services
            - Sample                        //Sample is the name of this web service
                - META-INF
                    - services.xml       //this file describes the information of this web service.
        - classes
            - package                     //this is path corresponding to package
                - *.class                    //These .class files are implementation of web service.
    - META-INF
    - axis2-web

The original layout is:
axis2
    - WEB-INF
        - lib
        - conf
        - services
            - Sample.aar               //the .aar archive file
        -classes
    - META-INF
    - axis2-web

Difference is highlighted in blue.
The services.xml is also different. The new services.xml is:

<service name="Sample" >
    <description>
        Please Type your service description here
    </description>
    <messageReceivers>
        <messageReceiver mep="http://www.w3.org/2004/08/wsdl/in-only" class="org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver" />
        <messageReceiver  mep="http://www.w3.org/2004/08/wsdl/in-out"  class="org.apache.axis2.rpc.receivers.RPCMessageReceiver"/>
    </messageReceivers>
    <parameter name="ServiceClass" locked="false">package.Sample</parameter>
</service>

Besides, in my previous using of Axis2, I used tool provided by Axis2 to construct automatically a stub class and some other auxiliary classes of client side. Then I added my implementation code to that stub class (this stub class contains Axis2-specific stuff). In this new deployment, I did not rely on any Axis2-specific functionality. I just wrote my implementation and compiled it into .class files. And then copied these .class files to directory ${TOMCAT_ROOT}/axis2/classes/package-path/. In addition, I needed to manually create and edit corresponding services.xml file. In my previous deployment, this file is generated automatically by Axis2 tool.
To sum up, this new deployment method eases development of web service because no Axis2-specific stuff is involved during development. The drawback is it is not compact considering that those files are scattered in different places. With regard to this deployment, I did not find much useful information on web. Maybe this method is not recommended, who knows...

However, I have no choice because only one of them works well.

DB4O corrupts in Tomcat with Axis2

2008-03-24T16:21:00.002-04:00

I integrated Db4o in our project. Db4o is used to persist state information of workflow. After I integrated it, it seemed to work OK. State information can be successfully stored into its object database. And information can be retrieved successfully. However, after I restarted Tomcat, Db4o corrupted during retrieval of data from database. I tried different kinds of query language Db4o supports. Unfortunately, none of these methods work.

For Native Query, the error in Tomcat log is:

java.lang.IllegalArgumentException: argument type mismatch
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at com.db4o.query.Predicate.appliesTo(Unknown Source)
    at com.db4o.inside.query.PredicateEvaluation.evaluate(Unknown Source)
    at com.db4o.Platform4.evaluationEvaluate(Unknown Source)
    at com.db4o.QConEvaluation.visit(Unknown Source)
    at com.db4o.Tree.traverse(Unknown Source)
    at com.db4o.QCandidates.filter(Unknown Source)
    at com.db4o.QConEvaluation.evaluateEvaluationsExec(Unknown Source)
    at com.db4o.QCon.evaluateEvaluations(Unknown Source)
    at com.db4o.QCandidates.evaluate(Unknown Source)
    at com.db4o.QCandidates.execute(Unknown Source)
    at com.db4o.QQueryBase.executeLocal(Unknown Source)
    at com.db4o.QQueryBase.execute1(Unknown Source)
    at com.db4o.QQueryBase.getQueryResult(Unknown Source)
    at com.db4o.QQueryBase.execute(Unknown Source)
    at com.db4o.inside.query.NativeQueryHandler.execute(Unknown Source)
    at com.db4o.YapStreamBase.query(Unknown Source)
    at com.db4o.YapStreamBase.query(Unknown Source)
    at org.cogkit.cyberaide.axis2ws.StatusDB.getStatusByUID(StatusServiceInterfaceSkeleton.java:1204)
    at org.cogkit.cyberaide.axis2ws.StatusServiceInterfaceSkeleton.getJSONStatusByUID(StatusServiceInterfaceSkeleton.java:184)
    at org.cogkit.cyberaide.axis2ws.StatusServiceInterfaceMessageReceiverInOut.invokeBusinessLogic(StatusServiceInterfaceMessageRece
iverInOut.java:80)
    at org.apache.axis2.receivers.AbstractInOutSyncMessageReceiver.invokeBusinessLogic(AbstractInOutSyncMessageReceiver.java:42)
    at org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:96)
    at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:145)
    at org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:275)
    at org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:120)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:263)
    at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
    at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:584)
    at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
    at java.lang.Thread.run(Thread.java:595)

It seems that the error results from incompatible type conversion. I am sure I followed the instructions elaborated in the official document.

Then, I tried SODA. The output was weirder. The program could not function as I expected. State information could not be retrieved successfully. However, when I investigated tomcat log, no error report existed in log!!! So, I had no way to tell what happened under the hood. This was pretty annoying. It seemed that Db4o indeed retrieved something. However, the fields of the object retrieved are invalid.

I read almost all posts in Db4o forum/community. I tried all suggested solutions. But it still doesn't work. I tried different versions of Db4o, none of them solved my problem.

It has taken me lots of time. Currently, I am not sure whether I can solve it finally...

State Information Persistence (Status server)

2008-03-17T16:40:00.002-04:00

Previously, state information of all workflows in status server is kept in memory. As you know, this strategy is not practical if there is too much information so that it cannot fit into memory. In other words, data persistence is necessary. Actually, after careful investigation, I found two solutions.
(1) Cache

    This strategy stores some data in hard disk. And the data recently used is saved in memory to enhance performance. So it is kind of data-centric mechanism making use of locality.
    Ehcache(http://ehcache.sourceforge.net/) is a project based on this strategy.
    In order to put data into cache, you MUST construct a element object which contains key and value. From ehcache 1.2 the key and value can be serializable objects which improve the flexibility. The unique way for data query is to specify a key object. Obviously, much work must be done to compose complex query. Ehcache provides ways for users to control every aspect of cache behaviors.

(2) Database

This strategy makes use of database to store information. There is a corresponding name : object database. The database provides interfaces by which common database operations (insert/query/update/delete) can be invoked. However, this kind of database is different from traditional relation database. The unit of manipulation in object database is an object. In other words, you can insert/query/delete objects instead of tuples. In addition to retrieval of field value in a specific object, member functions of the object can also be invoked.
Db4o(http://www.db4o.com/) is an object database. Here is a simple introduction written by me: http://zhenhua-guo.blogspot.com/2008/03/db4o-introduction.html.

Actually, these two strategies are not competitive. Implementation of object database may make use of cache mechanism. So it is not surprising to hear that some great applications (hibernate, Spring...) use Ehcache. For upper level programmers, I think object database should be easier to work on.

At last, I decided to apply DB4O to our project. One important reason is that this strategy lets programmers write code in high level. I don't need to care about any details of database infrastructure. As a result, modification of my original code is little so that it can be done quickly. Moreover, maintenance of code is easier. We store both data and related operations in database instead of scattered places. What's more, support of query is more powerful in DB4O than in Ehcache. It supports three kinds of query languages: Query By Example, Native Queries and SODA. And complex query can be composed easily.

One lesson: when dealing with data retrieved from object database, users MUST be careful about type conversion so that incompatible conversion won't occur. Because this kind of errors don't appear until run time, debugging becomes more difficult especially for web applications. I was aware of cause of the error by careful reading lengthy Tomcat log.

Result:
State information of workflows is stored in hard disk and cached in memory by using DB4O.

DB4O introduction

2008-03-17T15:20:00.002-04:00

Db4o is a high-performance object database for Java and .NET.

Ø Open db

ObjectContainer db = Db4o.openFile(filename);

Ø Insert

Objects are inserted by using set() method.

ClassName obj = new ClassName(parameters);

db.set(obj);

Ø Retrieve

(1) Query by Example (QBE)

Create a prototypical object for db4o to use as an example of what you wish to retrieve. Db4o will return all of the objects which match all non-default field values. The results will be returned as an ObjectSet instance.

ClassName obj = new ClassName(values…); //prototypical object

ObjectSet result = db.get( obj );

listResult( result );

Db4o supplies a shortcut to retrieve all instances of a class:

ObjectSet result = db.get(ClassName.class);

Following code can be used to iterate over the results:

while( result.hasNext() ){

System.out.println( result.next() );

}

(2) Native Query(NQ) --- main db4o querying interface.

Native Queries provide the ability to run one or more lines of code against all instances of a class. Native query expressions return true to mark specific instances as part of the result set.

List<ClassName> objs = db.query( new Predicate<ClassName>() {

public Boolean match(ClassName obj){

return obj.getProperty() == value;

}

Users must be very careful with side effects --- especially those that might affect persistent objects.

(3) SODA Query API

Ø Update

Updating objects is as easy as storing them. You use the same set() method to update objects: just call set() again modifying any object.

ObjectSet result = db.get(new ClassName(parameters));

ClassName found = (ClassName)result.next();

found.methodName(parameters);

db.set(found);

Note: we query the object first. If the object is not ‘known’ (having been previously stored or retrieved during the current session), db4o will insert a new object instead of updating existing object. In this case, db4o think that you want to insert a new object which has the same field values.

Ø Delete

Objects are removed by using delete() method.

ObjectSet result = db.get( new ClassName(…));

ClassName found = (ClassName)result.next();

db.delete( found );

If you want to tune DB4O to get higher performance, you need to change the default configuration.

2008-03-13T17:18:00.001-04:00

Some workflow related projects

DIET

http://graal.ens-lyon.fr/DIET/

DIET (Distributed Interactive Engineering Toolbox) seems to be similar to Condor. DIET is based on Grid-RPC. Clients submit computation requests to a scheduler whose goal is to find a server available on the grid. The aim of the DIET project is to develop a set of tools to build computational servers. Scheduling is frequently applied to balance the work among the servers and a list of available servers is sent back to the client; the client is then able to send the data and the request to one of the suggested servers to solve their problem. The Distributed Interactive Engineering Toolbox (DIET) project is focused on the development of scalable middleware with initial efforts focused on distributing the scheduling problem across multiple agents. DIET consists of a set of elements that can be used together to build applications using the Grid-RPC paradigm.

So the goal of DIET project is different from that of our project.

Taverna

Recently, I installed and tried Taverna. Then I investigated its functionality in detail.

Its manual is here: http://www.mygrid.org.uk/usermanual1.7.

Taverna is created by myGrid project and is a tool used for designing and executing workflows. “It provides a desktop authoring environment and enactment engine for scientific workflows expressed in SCUFL (Simple Conceptual Unified Flow Language).” SCUFL is proprietary. As a result, I can not find detailed information about SCUFL.

Services are connected with data links (providing data flow) and control links (coordination of services not connected through data flow).

Several features:

Fault Tolerance
- Retries for every processor

If a certain processor fails to execute, it will be retried several times. Users can specify maximum number of retries, delay of retries and backoff. Backoff is a factor determining how much the delay time increases for subsequent retries beyond the first.

Alternative processor

Users can specify an alternative processor or list of processors which perform the same task as the primary processor. And the alternate is used in place of the main processor if the latter has failed. Note: the alternate has its own definable parameters for ‘Retries’, ‘Delay’ and ‘Backoff’.

Iteration

Taverna supports two kinds of iteration. They are dot and cross. Cross iteration is an all-against-all iteration which means it iterates over all combinations of input values. For dot iteration, the first item for one input is related to the first item in the other input and the second item for one input is related to the second item in the other input…

Services

Taverna provides some built-in services. Among them are XML transformation, base64 encoding/decoding, write text file… Besides, beanshell and RShell are supported as well.

It also supports some well-known bioinformatics services including Soaplab, Biomart, Biomoby. I am not familiar with biology related tools. So I am not sure whether these services are based on web services.

Besides, Taverna supports a functionality called WSDL scavenger. Users can specify address of WSDL document and Taverna will automatically fetch the WSDL document and analyze its content to extract supported operations. Then supported operations in the WSDL document are added to list of available processors so that users can make use of them easily in course of workflow composition.

In addition, Taverna can scavenge existing workflow and extract processors.

Summary

Taverna is a tool designed specifically for Bioinformatics. However, some features may be also useful even if they are applied to more generic applications. These features include WSDL scavenger, dot/cross iteration, fault tolerance…

It is different from our project as Taverna is not based on Grid. The enactment engine is located in client-side machine. In our project, enactment engine is located at server side which makes use of Java CoG kit to manage execution of workflows.

Social website myExperiment

This web site supports finding and sharing of workflows and has special support for Scufl workflows. Users can download workflows posted on the site. For every workflow written in Scufl, there is a corresponding .svg image which is easier to understand and verbose xml Scufl document.

It seems that myExperiment supports almost all functionalities most web2.0 web site supports. They include user management, group management, workflow management, blog, forum, tagging, rating, and commenting. Moreover, some statistics (number of reviews, number of comments…) is done.

Moteur

This project is also based on SCUFL.

Karajan Workflow:

Is there a way to invoke some operations described in a WSDL document?

DAG construction

2008-02-21T22:58:00.000-05:00

After communication with Mike, I got the dag.k file. Then I implemented the construction of DAG workflow based on all jobs and their relationship in a workflow. In other words, I put together all this information in a large Karajan workflow by using DAG. Note: this work is done at client side, not at server side. After construction of the huge workflow, it can be submitted to server just as a small job. Then id of the new submitted workflow will be returned. Based on this workflow id, a user can query state of the workflow. In addition, the user can access output files of the workflow by using common HTTP GET request.

More details about DAG construction:
Assume that we have four jobs in a workflow: job1, job2, job3, job4.
And their dependencies are:
job1 -> job2 ( this means job2 depends on job1 )
job1 -> job3
job3 -> job4
job3 -> job5
job1 -> job5
These dependencies are represented in following graph:

Then constructed DAG workflow looks like this:
<project>
<include file="cogkit.k"/>
<include file="dag.k"/>
<discard>
    <dag>
        <node>
            <string>job1</string>            //Here is name of the job.
            <element>
                <quotedlist/>
               content of job1
            </element>
            <edges>
                <string>job2</string>         //Here, it describes that job1 is prerequisite of job2, job3 and job5
                <string>job3</string>
                <string>job5</string>
            </edges>
        </node>
        <node>
            <string>job2</string>
            <element>
                <quotedlist/>
                content of job2
            </element>
        </node>
        <node>
            <string>job3</string>
            <element>
                <quotedlist/>
                content of job3
            </element>
            <edges>
                <string>job4</string>
                <string>job5</string>
            </edges>
        </node>
        <node>
            <string>job4</string>
            <element>
                <quotedlist/>
                content of job4
            </element>
        </node>
        <node>
            <string>job4</string>
            <element>
                <quotedlist/>
                content of job4
            </element>
        </node>
    </dag>
</discard>
</project>

Karajan Workflow Formats: .k and .xml

2008-02-21T22:21:00.001-05:00

In CoGKit, there are two supported formats in Karajan workflow -- .k and .xml.
Personally, I like xml because of its prevalence and openness. There are many handy tools which can process xml documents in various ways. Unfortunately, xml support in Karajan workflow is not comprehensive. In recent programming, I need to use element dag which is used to support Directed Acyclic Graph.
(1) At first, Karajan workflow supports DAG. But it does not give users a dag.xml file to import. Instead, just dag.k is provided. So I need to find a way to convert dag.k to dag.xml. Thank Mike and Gregor to help me get out of the struggle. Following method can be used to do the task:

cog-workflow -intermediate dag.k

Although following error pops up, I still can get the file I need (dag.xml).

Execution failed:
Variable not found: #channel#defs
        kernel:export @ dag.k, line: 44

Note: actually, what is generated is dag.kml not dag.xml. Currently, I assume they are the same because I have no better choice.

(2) In document I can find, sample workflows about DAG are written in .k format.
This is the unique resource I find helpful: http://wiki.cogkit.org/index.php/Java_CoG_Kit_Workflow_Guide#Direct_Acyclic_Graphs.
Once again, I used command cog-workflow to do this task. But the generated .kml(xml) file is lengthy and not appropriate for human readers. So, I decided to convert it manually by myself. I found this useful link(http://wiki.cogkit.org/index.php/Java_CoG_Kit_Karajan_Workflow_Reference_Manual_4.1.5) which describes both formats in detail.

How to access results?

2008-02-21T22:16:00.001-05:00

Now, it is time to consider how to make users easily and conveniently access the results of their workflows. There are several questions here:
(1) How to track output files in Karajan workflows?

The first option is to analyze content of the Karajan workflow to figure out output files. For example, for element execute, attribute stdout indicates the name of output file.
<execute executable="/bin/date" stdout="thedate" host="gf1.ucs.indiana.edu" provider="GT2" redirect="false"/>
However, if we use this method to track all output files, it is difficult and time-consuming because it is possible that many elements generate output files. As a result, we must capture possible output files from all these elements.
Another option I can think of is kind of tricky. The newly submitted workflow is executed in a newly created directory. After execution, the files (except workflow file) in the directory are output files. This is the method I am using in my implementation.

(2) How to organize output files?

For the same workflow, we can categorize it based on different criteria. For example, we can categorize a workflow based on the date on which it is submitted, or the date on which it is completed... I would like to make use of workflow id and user id to categorize the workflows. All workflows submitted by a user belong to the same group which can be accessed by this user. Within these workflows, workflow id is used by the user to access a specified workflow. The id of every workflow belonging to a user is unique.
So, the directory layout may look like this:
users/user1/workflow_122/output_file1

users/user1/workflow_122/output_file2
users/user2/workflow_1/output_file1
...

(3) How can users access output files?

After talking with Marlon, I would like to provide RESTful interface by which users can retrieve output files. In my implementation, URLs to access output files look like this:
http://domain:port/resources/user_name/workflow_id/ This retrieves list of all output files for the corresponding workflow.
http://domain:port/resources/user_name/workflow_id/output_file This will retrieve the specified output file directly.