yeah

I bought a pair of pants. Six days later I notice the stitching has come apart. No need for vexation. I go to the store, I show the pants and the receipt to the nice guy behind the counter. He calls the manager just to be sure. The manager decides to come over and look at the pants.

The manager then says the pants are fine, the hole is probably of my own doing, but she's not saying it is, she's merely examining the threading. In the end she tells the cashiers to refund me and she leaves. I get my refund. When the cashier ends with "I'm sorry about your pants, they really should last more than six days," I realize that I'm not mad at the cashiers, I'm mad at the manager.

Flip the tables. You're the cashier. A calm customer comes in for a routine refund of a defective good. Suddenly, your boss butts in, she tells the customer to fuck off, she orders you to finish the transaction, and she retreats to the office.

-

clarksfield turbo

though clarksfield chips are reported to be clocked at measly 1.6, 1.8, and 2.0ghz, nehalem's turbo boosts them up to 3,2ghz for a lone running thread.

the current top of the line mobile core2 chips x9100 and qx9300 are dualcore @ 3,06ghz and quad core @ 2,53ghz, both with 1066mhz fsb. clarksfield uses dual channel 1333mhz memory, and has DMI instead of the desktop variant's QPI.

conclusion: a 2ghz clarksfield will beat an x9100 in single-thread performance, and a qx9300 in parallel performance. clarksfieldy macbook pros should be out right around the time my summer vacation ends.

how bad is DES?

DES was kicked out of srs bsns over a decade ago, yet there are no bruteforce des crackers available for home PCs. DES encryption/decryption is not well suited for the instruction set of a general purpose computer. in 1999 distributed.net harnessed the idle cpu time of around 100.000 computers together with EFF's deep crack to break des within 24 hours. the peak of 170 billion decryptions every second on 100.000 computers averages out to 1,7 million decryptions per second per computer.

for computers 10 times as fast now as those of 1999, a large amount of botnet time would be consumed in cracking just a single key. the current GPGPUs aren't particularly well suited for DES cracking either. a single decryption takes a lot of cycles and a lot of power.

FPGAs are optimal for DES. the described implementation is a 37 stage pipeline composed of single cycle steps that changes keys, plaintext, ciphertext and operation mode on every cycle without stalls.

in 2006, german researchers implemented a 9000 euro device with a total power consumption of 600 watts consisting of 120 FPGAs, each capable of 400 million encryptions per second. they eventually raised the clock speed so that an exhaustive search of the DES keyspace takes 12.8 days. DES is futher susceptible to time/memory tradeoff attacks which copacabana performs.

-

how wicket protects you from CSRF

the beauty of wicket finally became apparent to me when I wrote my first delete operation.

the traditional delete works like this:

miau.biz/delete.php?id=5

instead, wicket sets up a bounded number of continuations that the user can invoke. each continuation corresponds to one of the allowed deletions. if an item isn't allowed to be deleted, that continuation does not exist, and neither does there exist a way to invoke it.

an attacker can however induce a victim to invoke any existing continuation against their will.

say the url to delete id 5 is:

miau.biz/::wicket::invokeContinuation:2

putting that url in an img tag will make the browser load it automatically. oops.

the suggested solution is to include a hard to guess token with each link, and requiring separate login for dangerous operations.

wicket uses null as the name of the default PageMap, and sequential numbering for page id's, page versions and markup id's.

wicket has a builtin fix against csrf. you aren't forced to use their solution however. in fact nextPageId() in Session is protected and not final; I can override that with

private static SecureRandom random = new SecureRandom();

synchronized protected int nextPageId() {
return random.nextInt(Integer.MAX_VALUE);
}

not every request gets a unique id, but many requests do. and the whole thing is fairly transparent.

Wicket's CryptedUrlWebRequestCodingStrategy has a fallback mechanism to allow you to also use urls that aren't gibberish:

   public RequestParameters decode(final Request request)
  {
      String url = request.decodeURL(request.getURL());
      String decodedQueryParams = decodeURL(url);
      if (decodedQueryParams != null)
      {
          // The difficulty now is that this.defaultStrategy.decode(request)
          // doesn't know the just decoded url which is why must create
          // a fake Request for.
          Request fakeRequest = new DecodedUrlRequest(request, url, decodedQueryParams);
          return defaultStrategy.decode(fakeRequest);
      }

      return defaultStrategy.decode(request);
  }

*cough*

eclipse containers for fun and profit

given a multi module project in eclipse, where project X depends on project Y,
when I close project Y,
I want project X to depend on a jar of project Y.

eclipse classpath containers are a mechanism for dynamic late binding of classpath elements.

ibm has a tutorial that requires you to register on their site. m2eclipse does the whole thing for maven projects, but also 99 other things, most of which involve running random maven tasks when your trying to write code.

despite all the static methods and integers enumerations, eclipse makes the whole thing easy; including creating your own plugin and publishing an update-site for it.

the heart of the class path container is the is the IClasspathContainer. The getClasspathEntries() method returns whatever is relevant for that situation. in my case it's either a JavaCore.newProjectEntry() or a JavaCore.newLibraryEntry() depending on whether the referenced project is open or closed at the time. For some reason JavaCore.newVariableEntry() comes out as K_SOURCE which doesn't work for a jar. so I use wrapped the resolved path from newVariableEntry() into a newLibraryEntry to make it a K_BINARY. your type is K_APPLICATION, your path is your unique identifier, and stuff after the slash in the path is parameters for your container. in my case the parameters might be the other projects that this project depends on.

then you need a ClasspathContainerInitializer, which will call JavaCore.setClasspathContainer with a new instance of YourContainer on the dependent projects everytime you open or close one the dependee projects. this then sends an F_CLASSPATH_CHANGED signal to those concerned.

you'll know that a project was opened or closed, because you added an IElementChangedListener with JavaCore.addElementChangedListener().

Your initializer is registered by adding it at the extension point org.eclipse.jdt.core.classpathContainerInitializer. never mind the xml, eclipse will generate it for you.

to allow the addition of containers from the build path configuration menu, eclipse requires a classpathContainerPage. either create a dummy one and register it, or add your container to the classpath file by hand.

Finally, I exported my plugin, and then binary imported the same. This is a nonrecoverable operation that replaces the source code of the plugin with the class files.
*cough*

throwing money at it

sometime in the future apple will get past the awful 1.6.0_07. meanwhile joel on software, did a wonderful piece on using money/ssd drives to speed up their build. I wasn't able to test this myself so I am glad joel did it for me, and confirmed my expectations that compiling/testing is cpu/memory bandwidth bound rather than io bound. our whole build takes up around 200 megabytes of physical space with all its dependencies and the jdk takes another 200 megs. further, a good part of both chunks is probably never loaded. so with enough memory there would be virtually no need for disk io.

what would be interesting beyond this would be to figure out if ssd could help in the overall workloads we use, that is (again): reading email, im'ing, running virtualbox, using an IDE, building, using the browser, profiling, running jetty. all at once. and that could hard to measure unless the difference was so great it would be obvious in actual use.

then I had the chance to work on a brand new windows box with a t5500 (a 1,66ghz core2duo), 2gigs of ram, and corporate antivirus software set to scan all class files, and read michael feather's excellent book working effectively with legacy code.

by the time I highlighted a block of code in eclipse, held down alt and hit the up arrow and nothing happened. 5 seconds elapsed, there was screech from the harddrive for a few seconds and then my method moved up a few lines. I was just smiling from the absurdity.

michael feathers points out the importance of having a fast feedback cycle and his book also alleviated my fears that, while i am waiting for the computer to do something it's natural for my mind to wander off and get frustrated, rather than spending that time subconciously processing the customer's business logic. I've actually found myself trying to optimize my day so that when I have to go to the bathroom, I'll hold out a bit so I can start a build, or get an update from version control before I go. then I realized I was way off putting any mental effort into this.

we now have some good steam going into getting our sprawling build back under control. the experience so far has reminded me of the importance of communication. I just can't guess what others are thinking from just looking at them and watching them act. I really need to make the effort to conciously communicate with people in the same room, the next room, and farther away to find out. and it is often very enlighting to do so.

as an example, it turns out part of the frustration of the slow build was of my own doing. the whole issue could be circumvented not by cleaning up the build, getting faster hardware, optimizing our continuous integration environment, switching to a more branch friendly version control system, but by the simplest solution of all. it all goes away if you just stop running the build.

I also started using a remote git repository on my own server to move scripts around between home and work. I'm expecting it to be even better than posting stuff on my blog and copypasting it at work.

for those on jaunty, who do run the build:

#!/bin/bash
function notify {
 test -x "`which notify-send`" && `which notify-send` -u $1 -c $2 -i $3 "$4" "$5"
}
/usr/bin/time -f "%E real" -o ./time.txt buildr $@
if [ $? -eq 0 ]
then
 notify normal transfer.complete sunny "successful build" "`cat time.txt`"
else
 notify critical transfer.error error  "failed build" "`tail -1 time.txt`"
fi
rm ./time.txt

and this too, because it uses the totally sweet erlang icon in the notifications!

       case regexp:match(ServerName, "compute.amazonaws.com") of
               nomatch -> ets:delete(instances, Nick),
                       os:cmd("/usr/bin/notify-send -i erlang -c device.error -u normal 'joe experienced an error'"),
                       bot:irc_privmsg(Socket, Nick, "some kind of error occurred.");

enjoy.

static analysis

module FindBugs
  include Extension

  first_time do
    desc 'run findbugs on your project'
    Project.local_task('findbugs')
  end

  before_define do |project|
    project.recursive_task 'findbugs'
    Rake::Task.define_task 'findbugs' do |task|
      fail "you must set FINDBUGS_HOME" unless ENV['FINDBUGS_HOME']
      findbugs_home=ENV['FINDBUGS_HOME']
      Buildr.ant 'findbugs' do |ant|
        ant.taskdef :name=>'findbugs', :classname=>'edu.umd.cs.findbugs.anttask.FindBugsTask', :classpath=>File.join(findbugs_home, "findbugs-ant.jar")
        ant.findbugs :home=>findbugs_home, :outputFile=>project.path_to(:target, "findbugs.xml") do
          ant._class(:location => project.path_to(:target))
          ant.sourcePath(:location => project.path_to(:source, :main, :java))
          ant.sourcePath(:location => project.path_to(:source, :test, :java))
          project.test.dependencies.each do |dependency|
            ant.auxClasspath(:location => dependency)
          end
        end
      end
    end
  end

  after_define do |project|
    findbugs = task('findbugs'=> project.test.compile)
  end

end

class Buildr::Project
  include FindBugs
end

bugs

so I really kicked myself in the nuts with that TMPDIR stuff.

since I have my m2 repository on the EBS volume, I should really have $TMPDIR on the volume too, instead of in my home directory. and the AMI I was using is geared towards IPv6. and my version of tsocks doesn't play too well with that so. add -Djava.net.preferIPv4Stack=true to java options to get back within the realm of transparent proxying.

I sure am glad I can delete all those broken ebs volumes now.

the missing pieces of the puzzle

here's the rest. btw, controlling the ebs volumes as shown in the previous post has the added benefit of making changes simple when problems/fixes are discovered.

here's stop.sh which maps an amazon host to an instanceid.

#!/bin/bash
set -e
echo "stopping $1" >> log.txt
ec2-describe-instances | grep $1 | cut -f 2 | ec2-terminate-instances - >> log.txt
echo "completed"

here's the user end script:

#!/bin/sh
if [ ! -e "./buildfile" ]
then
echo "no buildfile in this directory"
exit 1
fi

if [ -z "$EC2_HOST" ]
then
echo "set the EC2_HOST environment variable lol"
exit 2
fi

chmod 600 keys/id_dsa_ec2_buildr
export EC2_HOST_AND_USER="buildr@$EC2_HOST"
export SSH_COMMAND="ssh -i keys/id_dsa_ec2_buildr"
echo "syncing"
rsync -rt -e "$SSH_COMMAND" --delete-excluded --exclude target --exclude .svn --exclude bin --exclude "/*/reports" . $EC2_HOST_AND_USER:teh-build
echo "'n sync.  building"
$SSH_COMMAND -R 10022:a_host_on_your_network:22 $EC2_HOST_AND_USER ./build.sh $@ && echo "build successful"
rsync -rt -e "$SSH_COMMAND" $EC2_HOST_AND_USER:teh-build/reports .
echo "the end"

this is setup.sh on the ebs volume:

#!/bin/bash
useradd -m buildr
dpkg -i /mnt/super/debs/* > /dev/null
rsync -rt /mnt/super/buildr_home/ /home/buildr
chown -R buildr /home/buildr
chmod 700 /home/buildr/.ssh
chmod 600 /home/buildr/.ssh/authorized_keys

this is the directory structure of the ebs:

buildr_home  debs  jdk  jruby  m2

debs contains tsocks and vim-nox .deb files, jdk is openjdk7 b54, m2 is a populated repository, jruby is jruby patched with a different version of jodatime. and finally buildr_home is the prototype home directory for the build user.

here's the directory:

build.sh  scratch  tasks  teh-build  tsocks.conf

scratch is empty, tasks contains some additional tasks that are not needed in the regular build, more specifically stripedjunit and the monkey patch to ant.rb. teh-build is a prepopulated copy of the build directory, so the first rsync doesn't take as long.

here's build.sh:

#!/bin/bash
echo "start"
export EBS_MOUNT=/mnt/super
export LD_PRELOAD=/usr/lib/libtsocks.so
export JAVA_HOME=$EBS_MOUNT/jdk
export M2_REPO=$EBS_MOUNT/m2/repository
export PATH=$JAVA_HOME/bin:$EBS_MOUNT/jruby/bin:/usr/local/bin:/usr/bin:/bin
export BUILDR_PARALLEL=true
export TSOCKS_CONF_FILE=$HOME/tsocks.conf
export TMPDIR=$HOME/scratch
export JAVA_OPTS="-XX:+UseCompressedOops -noverify"
export JAVA_OPTIONS=$JAVA_OPTS
mkdir -p $TMPDIR
cp -r $HOME/tasks teh-build/super
cd teh-build
echo "in directory"
ssh -fN localhost
echo "ssh established."
sed -i "s/svn_revision/99999/" super/buildfile.rb
sed -i "s/\(define \"toplevel\" do\)/\1\ntest.using :stripedjunit/" super/buildfile.rb
echo "start build."
time tsocks buildr $@
export BUILDR_EXIT=$?
echo "that's it"
ssh -O exit localhost
echo "killed ssh"
exit $BUILDR_EXIT

I use a separate temp dir, because jruby is can't move files across devices. I don't copy the .svn files along so I use 99999 for the revision instead.

tsocks.conf, and server_type can be 5 for openssh DynamicForward contrary to what I claimed earlier:

server = 127.0.0.1
# Server type defaults to 4 so we need to specify it as 5 for this one
server_type = 5
# The port defaults to 1080 but I've stated it here for clarity
server_port = 50505

and .ssh config:

Host localhost
User commonuser
DynamicForward 50505
Port 10022
ExitOnForwardFailure yes
ControlMaster auto
ControlPath ~/.ssh/master-%r@%h:%p
IdentityFile ~/teh-build/keys/id_dsa_commonuser

here you need to have a host on the local network that has a user that allows logins with a shared key, from version control in our case.

but after all this, the user shouldn't need to do more than, get a hostname from the ircbot, do

export EC2_HOST=the.host.compute.amazonaws.com

and they can run, in the toplevel directory:

super/scripts/ec2-build.sh build_and_report

or any other build command. which reminds of build_and_report, and explains the rsync backwards after the build is finished.

 task :resources do
 #great
 test.using :fail_on_failure=>false
end

def report_failures
failed_tests = projects.map { |project| project.test.failed_tests }.compact.flatten 
report_files = []
projects.each { |project| project.test.failed_tests.each do |failure|
 report_file = File.join(project.test.report_to.to_s, "TEST-#{failure}.txt")
 report_files << report_file
end unless project.test.failed_tests.nil?
}

projects_with_tests = projects.reject { |project| project.test.tests.nil? }.join(", ")

puts "ran tests from #{projects_with_tests}"
unless failed_tests.empty?
 failed_tests.each { |failure| puts "fail: #{failure}" }
 report_files.each do |report|
   FileUtils.cp(report, project('toplevel')._("reports/#{File.basename(report)}"))
 end
 fail "you have #{failed_tests.length} test failures"
end
end

task :build_and_report do
task(:build).invoke
report_failures
end

that is, 1) run all the tests even if there is a failure, but 2) fail the build at the end in case there were failures, 3) report on the failures, 4) put all the failure reports in toplevel/reports/

enjoy!

clouditude

there's no easy way to share ebs volumes between accounts, and there are no corporate ec2 accounts. and I don't want to share my personal login. and people don't want to do mystifying stuff. so a shell script to set the thing up:

#!/bin/bash
set -e
if [ -z $EC2_PRIVATE_KEY ]
then
 echo "you must set EC2_PRIVATE_KEY" >> log.txt
 exit 4
fi
if [ -z $EC2_CERT ]
then
 echo "you must set EC2_CERT" >> log.txt
 exit 2
fi

export EC2_URL=https://eu-west-1.ec2.amazonaws.com
export EC2_ZONE=eu-west-1b
echo "getting volume" >> log.txt
export EC2_VOLUME=`ec2-describe-volumes |grep $EC2_ZONE | grep available | head -1 | cut -f 2`
if [ -z $EC2_VOLUME ]
then
 echo "no volume available in $EC2_ZONE" >> log.txt
 exit 1
fi
echo "volume selected: $EC2_VOLUME"  >> log.txt
echo "starting instance in $EC2_ZONE" >> log.txt
export EC2_INSTANCE=`ec2-run-instances ami-4acfe73e -t c1.xlarge -g default -g "ssh allowed" -k super-key -z $EC2_ZONE | grep INSTANCE| cut -f 2`
echo "instance started $EC2_INSTANCE" >> log.txt
while [ -z $EC2_HOST_NAME ]
do
 sleep 3
 export EC2_HOST_NAME=`ec2-describe-instances $EC2_INSTANCE | grep INSTANCE | cut -f 4`
 echo "host name is $EC2_HOST_NAME" >> log.txt
done
echo "attaching volume" >> log.txt
ec2-attach-volume $EC2_VOLUME -i $EC2_INSTANCE -d /dev/sdf >> log.txt
sleep 1
echo "running setup script" >> log.txt
ssh -o StrictHostKeyChecking=no -i ~/ec2/super-key.pem root@$EC2_HOST_NAME "mkdir -p /mnt/super; mount /dev/sdf /mnt/super; /mnt/super/setup.sh" 2>> log.txt
echo "great success" >> log.txt
echo $EC2_HOST_NAME

but, I don't want to be running shellscripts all day. so an ircbot

connect(Host, Port) ->
  ets:new(instances, [named_table, public]),
  {ok, Sock} = gen_tcp:connect(Host, Port, [{packet, line}]),
  Ec2Controller = spawn_link(ec2, loop, [Sock]),
  register(ec2_controller, Ec2Controller),

bla bla bla

respond(Nick, Channel, Sock, ["!"|TheRest]) ->
  case TheRest of
          ["hello"] -> irc_privmsg(Sock, Channel, Nick ++ " ! hello");
          ["palkka"] -> irc_privmsg(Sock, Channel, "9,5k nettona");
          ["halp"] -> irc_privmsg(Sock, Channel, "use 'joe ! start' and 'joe ! stop' to control your ec2 server"),
                  irc_privmsg(Sock, Channel, "use 'joe ! check' to see your current status");
          ["check"] -> check_ec2_status(Nick, Sock);
          ["start"] -> io:format("start received~n"), start_ec2_server(Sock, Nick);
          ["stop"] -> stop_ec2_server(Sock, Nick);
          _ -> 0
  end;

and then...

start_ec2_server(Sock, Nick) ->
  Status = ets:lookup(instances, Nick),
  case Status of
          [] -> ets:insert(instances, {Nick, starting}), Ec2Controller = whereis(ec2_controller), Ec2Controller ! {start, Sock, Nick};
          [{Nick,Value}] -> case Value of
                          starting -> irc_privmsg(Sock, Nick, "plz hold I am already processing a request from you.");
                          {running, ServerName} -> irc_privmsg(Sock, Nick, "you already have instance " ++ ServerName)
                  end
  end.

stop_ec2_server(Sock, Nick) ->
  Status = ets:lookup(instances, Nick),
  case Status of
          [] -> irc_privmsg(Sock, Nick, "you dont have a running server.");
          [{Nick,Value}] -> case Value of
                          starting -> irc_privmsg(Sock, Nick, "your server isn't even started yet.");
                          {running, ServerName} -> Ec2Controller = whereis(ec2_controller), Ec2Controller ! {stop, Sock, Nick, ServerName}
                  end
  end.

check_ec2_status(Nick, Sock) ->
  Status = ets:lookup(instances, Nick),
  case Status of
          [] -> irc_privmsg(Sock, Nick, "you dont have a running server.  use 'joe ! start' to start one.");
          [{Nick,Value}] -> case Value of
                          starting -> irc_privmsg(Sock, Nick, "your server is starting, I will notify you when it is available.");
                          {running, ServerName} -> irc_privmsg(Sock, Nick, "you have a running server at: " ++ ServerName), irc_privmsg(Sock, Nick, "you can use 'joe ! stop' when you don't need it anymore.")
          end
end.

yeah. finally:

loop(Sock) ->
      receive
              {start, Sock, Nick} -> io:format("received start~n"),
                      start(Sock, Nick),
                      loop(Sock);
              {stop, Sock, Nick, ServerName} -> io:format("received stop~n"),
                      stop(Sock, Nick, ServerName),
                      loop(Sock)
      end.

stop(Sock, Nick, ServerName) ->
      os:cmd("./stop.sh " ++ ServerName),
      ets:delete(instances, Nick),
      bot:irc_privmsg(Sock, Nick, "your server has been stopped.").

start(Sock, Nick) ->
      io:format("launching instance for ~s~n", [Nick]),
      bot:irc_privmsg(Sock, Nick, "started a server for you.  I will send you the hostname when it is up and running"),
      register_server(Sock, Nick, os:cmd("./launch.sh")).

register_server(Sock, Nick, ServerName) ->
      io:format("server is ready ~s - ~s~n", [Nick, ServerName]),
      case regexp:match(ServerName, "compute.amazonaws.com") of
              nomatch -> ets:delete(instances, Nick),
                      bot:irc_privmsg(Sock, Nick, "some kind of error occurred.");
              _ -> ets:insert(instances, {Nick, {running, ServerName}}),
                      bot:irc_privmsg(Sock, Nick, "your server is " ++ ServerName)
      end.