Scripted vmdk/ova images w/boxgrinder and virtualbox

It wasn't immediately clear to me from boxgrinder or virtualbox documentation how to script the complete creation of a self-contained runnable virtual image. Since it took me some effort to get it all together, I'll post here a complete example. The script creates a Fedora 16 w/graphical UI machine image by first creating a virtualbox vmdk image (pretty straightforward) and then from that, scripted creation of a more self-contained .ova image (less straightforward).

I have the following packages installed:

VirtualBox-OSE

rubygem-boxgrinder-build

Starting from a boxgrinder appliance definition file...

http://boxgrinder.org/tutorials/appliance-definition/

...the script creates an .ova image that can be easily imported and used by 3rd parties like this:

  $ VBoxManage import f16-xfce.ova
  $ VirtualBox --startvm f16-xfce

And now for the script:

#!/bin/sh
NAME=f16-xfce
OS=Fedora_64
IMAGE=/tmp/${NAME}.vmdk
SZMB=1536
INSTDIR=/tmp/boxes/
BUILDDIR=/tmp/builds/

mkdir -p ${BUILDDIR}
cat > ${BUILDDIR}/${NAME}.appl << EOF
name: f16-xfce
summary: Fedora with xfce
os:
  name: fedora
  version: 16
hardware:
  partitions:
    "/":
      size: 5
packages:
  - @base
  - @base-x
  - @fonts
  - @xfce-desktop
  - @critical-path-xfce
post:
  base:
   - "useradd boxgrinder && echo boxgrinder | passwd boxgrinder --stdin"
   - "ln -s --force /lib/systemd/system/graphical.target /etc/systemd/system/default.target"
EOF

cd ${BUILDDIR}
sudo -E boxgrinder-build -p virtualbox ${NAME}.appl
cp ${BUILDDIR}/build/appliances/x86_64/fedora/16/${NAME}/1.0/virtualbox-plugin/${NAME}.vmdk ${IMAGE}
VBoxManage createvm --name ${NAME} --ostype ${OS} --register --basefolder ${INSTDIR}
VBoxManage modifyvm ${NAME} --memory ${SZMB} --vram 32
VBoxManage storagectl ${NAME} --name "SATA Controller" --add sata --controller IntelAHCI
VBoxManage storageattach ${NAME} --storagectl "SATA Controller" --type hdd --port 0 --device 0 --medium ${IMAGE}
VBoxManage export ${NAME} --manifest -o ${BUILDDIR}/${NAME}.ova
# VirtualBox --startvm ${NAME}

ministat example for comparing performance

This is mostly a reminder to myself how to get and use FreeBSD's ministat for performance comparisons. This somewhat useless example shows that -server is apparently the default flag for java these days. Note: you have to strip out some FreeBSD-isms to get it to compile on Linux.

vocal(jason): cat mstatdemo.sh
#!/bin/sh
wget -O ministat.c 'http://www.freebsd.org/cgi/cvsweb.cgi/src/usr.bin/ministat/ministat.c?rev=1.14;content-type=text%2Fplain'
egrep -v '(FBSDID|ttycom)' ministat.c | gcc -x c -o ministat - -lm
wget -O scimark2lib.jar 'http://math.nist.gov/scimark2/scimark2lib.jar'
rm -f results-default.txt results-server.txt
for i in `seq 3`; do
 echo "default $i"
 java -cp scimark2lib.jar jnt.scimark2.commandline | grep Composite | awk '{print $3}' >> results-default.txt
 echo "server $i"
 java -server -cp scimark2lib.jar jnt.scimark2.commandline | grep Composite | awk '{print $3}' >> results-server.txt
done
ministat results-default.txt results-server.txt
vocal(jason) sh mstatdemo.sh
...
x results-default.txt
+ results-server.txt
+------------------------------------------------------------------------------+
|         ++x                                      + x                    x    |
||_________M___|________A_____________________A|_____M________________________||
+------------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   3      518.6688     521.25366     520.35719     520.09322     1.3124921
+   3     518.60099     520.27189     518.64299     519.17196     0.9528036
No difference proven at 95.0% confidence

Cloud abstraction APIs getting interesting

With many people rightly nervous about compute cloud lock-in, we seem to be entering an interesting time in the development of compute cloud abstraction libraries and APIs for trying to work around this. I noticed that last month RedHat's deltacloud has become the apache deltacloud (Ruby) incubator project alongside the already existing apache libcloud (Python) incubator project, which has a pretty impressive list of supported providers. An interesting alternative to these is jclouds (Java) with its own impressive list of supported providers. However, just this month, libcloud has announced that they too are working on a Java version of their API.

According to Ohloh, all of the libcloud, deltacloud, and jclouds projects are showing a lot of recent development activity - very good news for those of us rooting for the possibility of lockin-free compute clouds.

Learning how to teach physics (and probably math)

Eric Mazur, a Harvard professor, talks about his experience discovering that in spite of good in-class exam scores and good end-of-the-semester teacher evaluations, his students shockingly weren't internalizing elementary physics concepts e.g. in the context of everyday situations. He had only learned this because he had seen the result presented at a conference and didn't believe it, so decided to test his own classes to disprove the result. He was shocked that it was true in his classes as well - year after year.

He could have left things as-is - all the standard metrics (exams, evaluations) showed there was no dire need to change anything. But true conceptual internalization wasn't taking place.

Where was the problem? Wasn't he a good teacher? And weren't his students of the highest quality?

Dr Mazur discusses the characteristics of the concepts he wanted his students to internalize, the tests he used to test them, the metrics to measure the results and his multiple attempts to change his teaching style to achieve those results.

The process that in the end produced good outcomes (and has been validated in programs throughout the country) turns out to focus less on lecturing and more on directing peer-instruction. Well worth a watch...

Eric Mazur presents Confessions of a Converted Lecturer

There seems to also be good evidence for peer-assisted instruction for elementary level mathematics as well.

What makes a great teacher?

What makes a great teacher? Probably not what you think. Here is an article in the Atlantic describing data collected and analyzed by Teach for America (the non-profit organization that recruits recent college graduates and professionals to teach for two years in low-income communities throughout the US).

Starting in 2002, Teach for America began using student test-score progress data to put teachers into one of three categories: those who move their students one and a half or more years ahead in one year; those who achieve one to one and a half years of growth; and those who yield less than one year of gains.

According to the article, the characteristics could be summarized as:

set big goals

continuously seek to improve effectiveness

recruit students and families into the process

plan exhaustively and purposefully

work backwards from desired outcome

refuse to surrender to [...] bureaucracy and budgetary shortfalls

At the moment, TfA claims larger gains than can be independently verified (which are also only for math) but their approach still looks promising...

So far, only one independent, random-assignment study of Teach for America’s effectiveness has been conducted. That report, published by Mathematica Policy Research in 2004, looked at the organization’s teachers and found that, in math, their students significantly outperformed those of their more experienced counterparts. (In reading, though, the teachers’ students did the same as other teachers’ students.) Another study is due out in 2012 or 2013.

For the teaching of reading, Siegfried Engelmann's Direct Instruction seems to have years of evidence. And you can purchase an e-tutor that implements the method for home use

web and print with moinmoin, pdflatex, google charts and eastwood

As always, this is mainly a reminder to myself... We wanted to create a nice report/handbook that will be occasionally be printed, but in between printings should be a living (web) document. As typical for reports, it will contain charts and tables. We went with a wiki, which has pretty good tables and google charts looked like a perfect fit, until we saw that output is limited to png with a maximum resolution of 300k pixels. Then we found eastwood, a reimplementation of the google chart api in java with jfreechart as the back end. So here's how we do it:

• each report section is a moinmoin wiki page


• limit yourself to moinmoin's native tables, footnotes, sections, enumerations, descriptions


• use moinmoin's HTML macro to add google charts inline


• for the print version, install Eastwood, the free java re-implementation of the google chart API


• into Eastwood, integrate the Batik library for rendering SVG


• use inkscape -z -E chart.eps chart.svg to convert svg to eps (eps works better than pdf for chart boundaries) and use epstopdf chart.eps to convert eps to pdf


• install the text_latex.py moinmoin formatter from moinmoin FormatterMarket (this adds a renderAsLatex action)


• use wget -O section.tex --user-agent="Mozilla/5.0" --user=user --password=pw 'http://your.moinmoin.site/section?action=format&mimetype=text/latex' (or curl -o section.text -A 'Mozilla/5.0' -u user:pw 'http://your.moinmoin.site/section?action=format&mimetype=text/latex') to download one of your wiki pages as LaTeX


• For professional stuff, hire a LaTeX expert to define/create a professional print layout and its associated macros


• use your script-fu (e.g. awk,sed) to fixup the moinmoin-translated LaTeX markup and e.g. to call the right latex macros for including the pdf charts


• use pdflatex for generating your print-format report


• Automate all of the above with ant, make, shell or even .bat files


• And editmoin allows you to use

  $EDITOR

  to edit content instead of a browser's textarea editor
  

Poor Man's Java memory profiler

Here is an example of do-it-yourself memory profiling e.g. showing used heap memory over the life of a program run. I compared it with results from YourKit to convince myself it's ok.

The concept:

Launch a separate thread that collects used memory stats

add a shutdown hook to output the stats

Here's an example in a Junit test...

public class CacheRebuildReloadTest extends RollbackTestCase {
  private static int  j   = 0;
  private static final long t0 = System.currentTimeMillis();
  private static long t[] = new long[32767];
  private static long m[] = new long[32767];
  private static Runtime r = Runtime.getRuntime();
  private static boolean done = false;
  static {
    Runtime.getRuntime().addShutdownHook(new Thread() {
      public void run() {
        for (int i = 0; i < j; i++) {
          System.out.println(t[i] + "\t" + m[i]);
        }
      }
    });
    new Thread() {
      public void run() {
        while (!done) {
          t[j]   = System.currentTimeMillis() - t0;
          m[j++] = r.totalMemory() - r.freeMemory();
          try {Thread.sleep(500);} catch (Exception e) {}
          if (j > 32767) {done = true;}
        }
      }
    }.start();
  }
  public void testRebuildReload() {
    // stuff here
    done = true;
  }
}

Here is a comparison with a YourKit memory profile (and a plot of the 2 sampling frequencies and distributions - I used .5s and YourKit apparently 1s). Good enough for my purposes.

JUnitPerf timed load JUnit test example

For my own future reference, here is a complete example of a JUnitPerf timed load test of a JUnit test.

First the normal JUnit test case. In this case, just write a non-trivial sized file...

import junit.framework.TestCase;
import java.io.*;
import java.lang.reflect.Array;

public class AppTest extends TestCase {
  public AppTest(String testName) { super(testName); }
  public void testApp() {
    byte[] ary = (byte[]) Array.newInstance(byte.class, 65534);
    try {
      File tmp = File.createTempFile("writeTest", ".txt");
      tmp.deleteOnExit();
      FileOutputStream os = new FileOutputStream(tmp);
      FileDescriptor fd = os.getFD();
      for(int i=0; i<1000; i++) { os.write(ary); }
      os.flush(); fd.sync();
    } catch (IOException e) { }
  }
}

Then a JUnit-compatible junitperf test suite. Run in sizes of powers of two (and 3 times each)...

import com.clarkware.junitperf.*;
import junit.framework.*;

public class AppTimedLoadTest extends TestCase {

 private static final String TEST = "testApp";

 public static Test timedNload(int n, int s, String testName) {
   boolean wait  = true;
   Test test     = new AppTest(testName);
   LoadTest load = new LoadTest(test, n); load.setEnforceTestAtomicity(false);
   Test timed    = new TimedTest(load, s * 1000, wait);
   return timed;
 }

 public static Test suite() {
   TestSuite suite = new TestSuite();
   suite.addTest(timedNload(1, 30, TEST));
   suite.addTest(timedNload(1, 30, TEST));
   suite.addTest(timedNload(1, 30, TEST));
   suite.addTest(timedNload(2, 30, TEST));
   suite.addTest(timedNload(2, 30, TEST));
   suite.addTest(timedNload(2, 30, TEST));
   suite.addTest(timedNload(4, 30, TEST));
   suite.addTest(timedNload(4, 30, TEST));
   suite.addTest(timedNload(4, 30, TEST));
   suite.addTest(timedNload(8, 30, TEST));
   suite.addTest(timedNload(8, 30, TEST));
   suite.addTest(timedNload(8, 30, TEST));
   return suite;
 }
}

The results of a mvn test run (with added spaces to see the grouping)...

-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running AppTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.295 sec
Running AppTimedLoadTest
TimedTest (WAITING): LoadTest (NON-ATOMIC): ThreadedTest: testApp(AppTest): 2245 ms
TimedTest (WAITING): LoadTest (NON-ATOMIC): ThreadedTest: testApp(AppTest): 2397 ms
TimedTest (WAITING): LoadTest (NON-ATOMIC): ThreadedTest: testApp(AppTest): 2241 ms

TimedTest (WAITING): LoadTest (NON-ATOMIC): ThreadedTest: testApp(AppTest): 4274 ms
TimedTest (WAITING): LoadTest (NON-ATOMIC): ThreadedTest: testApp(AppTest): 4425 ms
TimedTest (WAITING): LoadTest (NON-ATOMIC): ThreadedTest: testApp(AppTest): 4374 ms

TimedTest (WAITING): LoadTest (NON-ATOMIC): ThreadedTest: testApp(AppTest): 9067 ms
TimedTest (WAITING): LoadTest (NON-ATOMIC): ThreadedTest: testApp(AppTest): 10204 ms
TimedTest (WAITING): LoadTest (NON-ATOMIC): ThreadedTest: testApp(AppTest): 8802 ms

TimedTest (WAITING): LoadTest (NON-ATOMIC): ThreadedTest: testApp(AppTest): 17439 ms
TimedTest (WAITING): LoadTest (NON-ATOMIC): ThreadedTest: testApp(AppTest): 18558 ms
TimedTest (WAITING): LoadTest (NON-ATOMIC): ThreadedTest: testApp(AppTest): 19724 ms
Tests run: 45, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 103.782 sec
Results :
Tests run: 46, Failures: 0, Errors: 0, Skipped: 0

A perl-wrapped java-based Nagios MS SQL plugin: check_mssql

I didn't find a nagios plugin that I like for monitoring our MS SQL databases. So I hacked up one in java using the jtds jdbc driver. Yes, the 2 minor things I do to improve "security" are horrible. I'll handle those things better later....

First, the java code... MSSQL.java

import java.sql.*;

// constr (jdbc:jtds:sqlserver://hostname:1433/DBNAME), user, pw, query
public class MSSQL {
  public static String MSG[] = {"OK", "WARNING", "CRITICAL", "UNKNOWN"};
  public static String msg   = "";
  public static int    s     = 3; // default is UNKNOWN
  public static long   n     = 0;
  public static long   t0ms  = System.currentTimeMillis();

  public static String rot13(String in) {
    StringBuilder sb = new StringBuilder(); int c;
    for (int b : in.getBytes()) {
      c = b & 32; b &= ~c;
      b = ((b >= 'A') && (b <= 'Z') ? ((b - 'A' + 13) % 26 + 'A') : b) | c;
      sb.append((char) b);
    }
    return sb.toString();
  }
  public static void main(String[] args) {
    Runtime.getRuntime().addShutdownHook(new Thread() {
      public void run() {
        String m = (new Exception().getStackTrace())[0].getClassName();
        System.out.println(String.format("%s %s %s|time=%dms;;; rows=%d;;;",
          m.substring(0, m.length() - 2), MSG[s], msg,
          System.currentTimeMillis() - t0ms, n));
        Runtime.getRuntime().halt(s);
      }
    });
    try {
      if (args.length != 4) {throw new Exception("4 args expected");}
      String c=args[0]; String u=args[1]; String p=args[2]; String q=args[3];
      Class.forName("net.sourceforge.jtds.jdbc.Driver");
      Connection conn = DriverManager.getConnection(c, u, rot13(p));
      Statement stmt = conn.createStatement();
      ResultSet rset = stmt.executeQuery(String.format("select * from %s", q));
      n = 0; while (rset.next()) { n++; } stmt.close(); conn.close();
      msg = String.format("%d rows",n); s = 0;
    } catch (Exception e) {
      s = 2; String m = e.getMessage(); if (m != null) { msg = m; }
    }
  }
}

Now, how to build the perl wrapper... buildmyself.sh

#!/bin/sh
wget http://surfnet.dl.sourceforge.net/sourceforge/jtds/jtds-1.2.2-dist.zip && \
unzip jtds-1.2.2-dist.zip jtds-1.2.2.jar && \
jar xvf jtds-1.2.2.jar > /dev/null && \
rm -rf META-INF && \
javac MSSQL.java && \
printf "Main-Class: MSSQL\n" > .X && \
jar cfm MSSQL.jar .X MS*.class net $0 MSSQL.java && \
rm -rf .X MS*.class jtds-1.2.2-dist.zip jtds-1.2.2.jar check_mssql net
cat > check_mssql << EOF
#!/usr/bin/env perl
use strict;
use File::Temp qw/tempfile/;
my @jar = <DATA>; my \$r; foreach (@jar) { \$r .= unpack('u',\$_); }
my(\$fh, \$fn) = &tempfile('/tmp/MSSQL-XXXXXX', SUFFIX => '.jar');
binmode(\$fh); syswrite(\$fh, \$r) || die('write failed');
close(\$fh) || die('close failed');
my @args = @ARGV; unshift @args, ("java", "-jar", \$fn);
system @args; \$r = \$? >> 8; END { unlink \$fn; \$? = \$r; }
__END__
EOF
uuencode MSSQL.jar MSSQL.jar > .X
N=`wc -l .X | awk '{print $1}'`
tail -`expr $N - 1` .X | head -`expr $N - 2` >> check_mssql && rm .X MSSQL.jar
chmod a+rx check_mssql

Note to self: On fedora uuencode is in the sharutils package...

Now when you run buildmyself.sh, it produces a perl-wrapped check_mssql that looks like this:

#!/usr/bin/env perl
use strict;
use File::Temp qw/tempfile/;
my @jar = <DATA>; my $r; foreach (@jar) { $r .= unpack('u',$_); }
my($fh, $fn) = &tempfile('/tmp/MSSQL-XXXXXX', SUFFIX => '.jar');
binmode($fh); syswrite($fh, $r) || die('write failed');
close($fh) || die('close failed');
my @args = @ARGV; unshift @args, ("java", "-jar", $fn);
system @args; $r = $? >> 8; END { unlink $fn; $? = $r; }
__END__
M4$L#!`H``````$))!3D````````````````)````345402U)3D8O4$L#!`H`
M```(`%Q)!3D$N9B<%````!(````4````345402U)3D8O34%.249%4U0N34;S
...

Usage like this:
vocal(jason): check_mssql jdbc:jtds:sqlserver://hostname:1433/DBNAME user passrot13 "FOO where BAR='baz'"
MSSQL OK 397 rows|time=1534ms;;;
vocal(jason): echo $?
0

check_http as a Nagios Web Service plugin

We typically use nagios to monitor our hardware and hosted services. Now that we are starting to offer more web services, we wanted to replace the ad-hoc curl wrapper we were using with a more standard nagios plugin. After searching for one and not finding any, it occurred to me that if curl can do it, maybe you can trick the standard check_http plugin to also do it. A quick experiment with an https hosted web service (self signed certificate for this test) shows that it works:

./check_http -S -k 'SOAPAction: ping' -r 'Current Status.*OK' -T text/xml -P '<s:envelope s="http://schemas.xmlsoap.org/soap/envelope/"><s:header/><s:body><ping/><s:Body><s:Envelope>' -t 10 -m 512 -p 8083 -H 192.168.1.254 -u /Services/Search/search.svc
HTTP OK HTTP/1.1 200 OK - 0.033 second response time |time=0.032722s;;;0.000000 size=897B;512;0;0

Now time to look at nagios grapher...

Siebel COM programming with Groovy

After my success in using Perl to program the siebel COM interface, I thought I'd try groovy. Yes it works as well.

import org.codehaus.groovy.scriptom.*

Scriptom.inApartment
{
def sa = new ActiveXObject('SiebelDataServer.ApplicationObject');
VariantByref rc = new VariantByref(0 as short);

sa.LoadObjects("C:/sea/client/OBJECTS/siebel.cfg", rc);
(rc.value == 0) || println("a: " + sa.GetLastErrText())

sa.Login("user", "password", rc)
(rc.value == 0) || println("b: " + sa.GetLastErrText())

def bo = sa.GetBusObject("Contact", rc)
(rc.value == 0) || println("c: " + sa.GetLastErrText())

def bc = bo.GetBusComp("Contact", rc);
(rc.value == 0) || println("d: " + sa.GetLastErrText())

bc.ClearToQuery(rc);
(rc.value == 0) || println("e: " + sa.GetLastErrText())

bc.ActivateField("Birth Date", rc);
(rc.value == 0) || println("f: " + sa.GetLastErrText())

bc.ActivateField("First Name", rc);
(rc.value == 0) || println("g: " + sa.GetLastErrText())

bc.ActivateField("Last Name", rc);
(rc.value == 0) || println("h: " + sa.GetLastErrText())

bc.ActivateField("Id", rc);
(rc.value == 0) || println("i: " + sa.GetLastErrText())

bc.SetSearchExpr("[Birth Date] = 01/01/1900 AND [Last Name] LIKE 'R*'", rc)
(rc.value == 0) || println("j: " + sa.GetLastErrText())

bc.ExecuteQuery(true, rc);
(rc.value == 0) || println("k: " + sa.GetLastErrText())

def moreRecords = bc.FirstRecord(rc);
(rc.value == 0) || println("l: " + sa.GetLastErrText())

while(moreRecords) {
 println " Last Name: " + bc.GetFieldValue("Last Name", rc);
 (rc.value == 0) || println("m: " + sa.GetLastErrText())
 println "First Name: " + bc.GetFieldValue("First Name", rc);
 (rc.value == 0) || println("n: " + sa.GetLastErrText())
 println "Birth Date: " + bc.GetFieldValue("Birth Date", rc);
 (rc.value == 0) || println("o: " + sa.GetLastErrText())
 println "        Id: " + bc.GetFieldValue("Id", rc);
 (rc.value == 0) || println("p: " + sa.GetLastErrText())
 moreRecords = bc.NextRecord(rc);
}

println 'Done'
}

Siebel COM programming with Perl

I wanted to play around with the Siebel COM interface using a dynamic language instead of having to re-compile after every change. I happened to have perl installed on my machine so it was the natural choice. Since google didn't show me any usable hits beforehand and it took a bit of investigation, I'm posting my results here.

#!c:/perl/bin/perl.exe -w
use strict;
use Win32::OLE;
use Win32::OLE::Variant;

my $schema = {
'Contact' => [
 'Birth Date', 'Created', 'Created By Name', 'Email Address', 'Fax Phone #',
 'First Name', 'Id', 'Job Title', 'Last Name', 'Updated', 'Updated By Name',
 'Work Phone #']
};

my $cfg = "C:/sea/client/OBJECTS/siebel.cfg";

my $sa = Win32::OLE->new("SiebelDataServer.ApplicationObject") or die "failed";

sub x
{
my ($tag, $rc) = @_;
if ($rc != 0) {warn("[$tag] " . $sa->GetLastErrText());}
}

sub main
{
my ($bo, $bc, $key, $field, $moreResults, $rcref);

$rcref = Variant(VT_I2|VT_BYREF, 0);
$sa->LoadObjects($cfg, $rcref);             &x("a", $rcref);
$sa->Login("user", "password", $rcref);     &x("b", $rcref);

foreach $key (keys(%$schema)) {
 $bo = $sa->GetBusObject($key, $rcref);    &x("c", $rcref);
 $bc = $bo->GetBusComp($key,   $rcref);    &x("d", $rcref);
 foreach $field (@{$schema->{$key}}) {
   $bc->ActivateField($field, $rcref);     &x("e", $rcref);
 }
 $bc->ClearToQuery($rcref);                &x("f", $rcref);
 $bc->SetSearchExpr("[Email Address] IS NOT NULL AND [Last Name] LIKE 'R*'", $rcref);
                                           &x("g", $rcref);

 $bc->ExecuteQuery(1, $rcref);             &x("h", $rcref);
 $moreResults = $bc->FirstRecord($rcref);  &x("i", $rcref);
 while ($moreResults) {
   printf("-----\n");
   foreach $field (@{$schema->{$key}}) {
     printf ("%40s: %s\n",$field, $bc->GetFieldValue($field, $rcref));
                                           &x("j", $rcref);
   }
   $moreResults = $bc->NextRecord($rcref); &x("k", $rcref);
 }
}
}
&main;

Csharp checkstyle partial solution

There doesn't seem to be an equivalent to Java/Eclipse's checkstyle for Csharp.Net

For the purpose of attempting to enforce a common code formatting style, microsoft recommends a per-solution config file together with a VB macro that automatically loads it up when you open the solution

Testing WCF Services using groovyWS (Apache CXF)

I wanted to use groovyWS to test my WSDL-first .Net WCF web services. By default, WCF doesn't publish its WSDL, it doesn't publish a flat WSDL, and the recommended WsHttpBinding binding makes groovyWS unhappy. Once you work around these things, it is possible to very conveniently write independent dynamic tests for your services.

Since it took a while to collect all this information, I thought I would post my result: a WCF ping service that is interoperable with a groovyWS (i.e. Apache CXF) client.

The steps I took:

1. Use GroovyWS to model my service. (groovy mockup\pingserver in one window and groovy pingclient in another)
2. Captured the WSDL it generated (wget -O Ping.wsdl http://localhost:7020/Ping?wsdl)
3. Wrote a .bat program to use the WCF svcutil.exe with XMLSerialization to generate marshaling code from the WSDL, and sed to remove generated ReplyAction="*" per operation so that it will properly expose information when it publishes its WSDL
4. Use the included boiler plate code to do the self hosting (i.e. standalone server) with the BasicHttpBinding, and used the FlatWsdl() class, which along with proper namespace annotations to allow generation of a "flat wsdl".
5. Appended a "ServiceBehavior" class which implements an interface generated from the WSDL, whose operations stubs can be generated from within Visual Studio by selecting the interface name, right click -> "Implement Interface"

All of this is bundled in a VS solution.

Prerequisites:

1. groovy zip distribution
2. groovyWS standalone jar (goes in lib dir)
3. .Net Framework 3.0
4. .Net Framework 3.0 WCF/WPF extensions

References:

Simple WCF Example
Improving WCF Interoperability: Flattening your WSDL
eliminating http://tempuri.org in wsdl
Getting started with groovyWS

Update (6 March): Java/.Net interoperability interviews with various SOAP project spokespersons

Cross Platform Java Analysis with DTrace

Since URLs to Jazoon07 slides and blog postings seem to be changing, I'm posting here a stable link to my look into Cross Platform Java Analysis with DTrace.

Update: I gave an updated talk, Dtrace for Java Developers at Jazoon08

Passive Characterization via Statistical Analysis

One area of my research employs the use of statistical analysis to passively characterize an instrument using artifacts inherent in the output it produces. If asked to make an "elevator pitch" for it, I like to refer to similar but easier to describe applications such as the ability to determine an author's gender solely artifacts inherent in the text, or the ability to identify particular anonymous machines on the internet by watching its packets go by using timing crystal artifacts inherent in is message timestamps. I just read a review in the economist for a book called super crunchers, which sounds like it might be another source for such applications.

Week day

In Europe, week numbers are used pretty extensively: The board is due to meet in week 33. Michael will be gone for military service in weeks 24-25. Please submit the quarterly numbers by week 17. Did you know there are at least 6 different ways that weeks are numbered?

Most of Europe seems to follow ISO 8601 numbering, which seems to be what BSD's "ncal -w" computes. But I need to fix my quarterly agenda generator in postscript (that I hacked up starting with pscal code), since I apparently coded up the American numbering by mistake.

If you're always online, there is of course a web-based alternative for looking it up. Its not clear which week number standard yahoo's calendar uses, since I only remember being able to specify which day the week begins on. Maybe it is inferred from your configured time zone?

Programming like it is still 1975?

There is an interesting new project called Varnish, which implements a reverse proxy. It is apparently up to 10 times faster than squid. But more interesting is why.

The software architect, who is usually a kernel programmer, decided to do a "userland program" for a change, as a fun project. Apparently he was disappointed that most programs haven't even taken advantage of things that have already been available in FreeBSD and Linux for at
least 10 years.

Linux threading scalability problems

After a couple of years of work on trying to make the FreeBSD kernel
more concurrent, a couple of kernel hackers started to enable the
concurrency (which is off by default) on an 8 CPU system and started
to see much improved results using a mysql benchmark
to measure it.

To make sure they weren't too far behind linux (which they have been
over the past couple of years) they compared the same benchmark on
the same hardware with the most recent linux and found that linux had
problems scaling when the number of threads was greater than the
number of CPUs.

Some linux hackers began to investigate this and found out that it is
not just mysql. The most likely cause seems to be 2 problems with
glibc - meaning that any program that does threading and uses glibc
(this includes java) would have the problem.

I guess once they identify the problem, it will probably be fixed within
the next few months.

Java Conference: Soundly choosing language features

After a philosophical discussion with my brothers on programming languages, I posted my thoughts on how it is about time to add a little more science into the art of designing programming languages. Instead of relying on the individual tastes of Wirth, Gosling, Wall, or these days von Rossum or Matsumoto, we need languages that help less IT projects to fail. And we need objective metrics of real programmers for evaluating and quantifying our hypotheses on what works.