The Adventures of Greg

Climbing Tips and Tricks

2021-05-02T00:00:00-07:00

Over the course of a year my wife and I went from our first day at the climbing gym to leading multipitch trad routes in Yosemite. Over the years we've dived into several technical/physical sports and doing this over and over gives a good background on how to get up to speed in a new sport as quickly as possible.

Now that we've been at this for two years I thought it is a good time to do a write-up of some of the things we learned that were not in books. These are little tips and tricks which either make climbing more efficient or lowers risk.

Here they are:

A persons level of confidence is not tied to their level of competence. Particularly important in climbing when an (over) confident climbing partner can result in an injury or epic if they are not also competent. Evaluating this is particularly challenging when starting out and being the less experienced of the two – so ease into these relationships; climb with someone in the gym on toprope first, then lead in the gym, then a day at the crag, before setting off up a remote multipitch with them.
When loading for rappeling, both climbers can rig their rappels before either starts rappelling, this allows you to safety check each other.
When tossing a rope-half for a rappel, hold the stopper knot, toss the rope, and then toss the stopper knot. This 2-phase toss reduces the chance of the rope ending up in a pile halfway down the rappel route.
When you have a line bag or tarp - if you tie both ends in immediately after climbing you don't have to re-flake the rope on the next climb. If both ends are tied in it cannot form a knot.
Girls: if your harness has clips on the leg loops – this means you can simply unclip these to use the bathroom, no need to remove your harness.
A quick and dense source of energy while climbing is Gu energy packets. These fit in pockets and can be eaten one-handed while belaying. Squeezy foods also meet this need.
When rigging to belay a follower, when you pull up all the rope, throw back down 8 feet. Don't take this 8 feet in until you have put them on belay. The follower will know not to climb until the pile of rope at their feet becomes taut, even if you can't hear each other.
A good mnemonic check when setting out on a (multi) pitch to ensure you don't leave any required equipment behind: ABCD - Anchor, Belay device, Cams, Draws.
Order 'fishing lanyard's from Amazon. You can clip this to your phone's case and then never worry when you pull it out to check a topo or take a photo. These are also good for retaining nut tools.
Put your phone into airplane mode when doing a multipitch or at a non-urban crag. Low signal will drain your batteries quickly, and doing this will preserve your batteries for more photos / topo checks / having some battery if you have to call in a rescue.
Small multipitch survival kit: space blanket, lighter, whistle (and carry a knife and headlamp too!)
The bag your harness came in is handy for climbs with walk-offs - put your approach shoes in these and clip to the back of your harness. keeps them neatly stowed and out of the way
Learning about trad placements while at home – you can put on firstpersonbeta's videos while doing other work on your computer; its a never-ending stream of trad placements so you can get familiar with what placements look like and how to spot good ones. (BTW, not a substitute for professional instruction. I think you can learn mostly on your own, but spend a day with a guide when you feel like you are ready, to get your placements dialed in and get a professional evaluation of if your placements are good)
If you are trad climbing halfway up a route and getting shut down on a section – you can always aid climb through with a couple single and double length slings for your feet. May not be the style you want, but it may save you from leaving gear.
Take photos of topos and route descriptions with your phone, then they'll always be handy to reference while you are on your climb.
If you have a lock-screen on your phone, switch it to something you can unlock ham-fisted one-handed; for my android I use a pattern unlock.
Our gym has 'garage sales', the unclaimed 'lost and found' items, its good to pick up a cheap pair of passable shoes here, so that when your main shoes need resoling you have something to climb in for the two weeks they are out for resole.

Things in books, but bear repeating

I've read through some of the AAC accidents archives. 90% of self-inflicted accidents can be avoided by the following maxims:

Everyone who's lowered off the end of their rope thought it was long enough that they didn't need stopper knots. Always Tie Stopper Knots.
Always weight test a new system before committing to it. If setting up for a rappel or a lower-after-cleaning put your full weight on the rope and confirm its the rope not your tether taking your weight before untethering.

Nacra Infusion Mk2 for sale

2019-12-28T00:00:00-08:00

Nacra Infusion

We are selling a 2012 Nacra Infusion Mk2 F18. The boat was originally owned by an America's Cup team and used to familiarize the sailors with SF Bay in 2012. It was bought by me just prior to the 2017 North-Americans, and just about everything besides hulls spars and foils replaced. Boat is kept mast-up at Mission Beach YC San Diego.

This boat is ready to race - and perfectly located for the upcoming F18 nationals and US multihull championship taking place in San Diego in the fall.

Includes:

Cat Trax wheels
Galvanized Trailer with generously-sized storage
sumbrella canvas trampoline cover
forward sailing canvas spinnaker cover
all standing & running rigging replaced summer of 2017
main and jib, in good condition
second main jib and battens
polyester spinnaker, with a few professional repairs
Rudder covers (sun damaged)
Spare boom (repaired)
dagger board covers
spare repaired boom
some spare standing rigging

Upgraded systems include * integrated under-trampoline righting system * port board uphaul system * retracting chicken lines * staymaster shroud adjusters * in-beam semi-continuous downhaul system * mast rotator quick release

Nacra Infusion Nacra Infusion Nacra Infusion Nacra Infusion Nacra Infusion Nacra Infusion Nacra Infusion Nacra Infusion Nacra Infusion

Using Spacy to build Conversational Interfaces for Alexa

2018-03-20T00:00:00-07:00

Over the last year I've been working on an Alexa Skill to automate some of the devices in my home. Working with the Amazon API's and dashboard for the provider-side parts of the setup has been good. However there's an assumptive built into the tools for Alexa and Google Home that the developer will use tools like Dialogflow or the skills console to create the conversational interfaces. The reason for this assumptive is that traditionally creating conversational interfaces is non-trivial.

Take the simple case of an app which provides the time to the user. The user may say what time is it? . If you were to write a regex or other matcher to match this string, you'd have many other variations like what's the time which would have the same intent but would not match. New techniques which are not normally part of the developers toolbox - like machine learning - have to be employed to determine the user's intent across many variations of an utterance.

In this day and age, can a developer use off-the shelf tools to handle these variations in utterances directly, rather than depend on a service like Dialogflow? The answer is yes. Fantastic NLP libraries such as Spacy can be used to perform this function. A developer can use Spacy to handle the Natural Language tasks and have better control over processing of language then they would if they used a third party.

We'll install Spacy below, and demonstrate how to use it to handle the Natural Language part of the Conversational Interface - using the raw user utterances coming directly from the Alexa service. We will do this with a simple skill which can tell the user the current time in any major city around the world.

The environment

The blog below assumes you've worked with python and flask-ask before. The example code is at https://github.com/gregretkowski/alexa-spacy My dev environment is An Ubuntu Xenial host and I use ngrok to proxy for Alexa Skills development.

Installing Spacy

Spacy is easily installed, and you'll need both the Spacy package and a pretrained model for English. We also install numpy to do some maths when we are categorizing utterances.

pip install flask-ask
pip install spacy
wget https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.0.0/en_core_web_sm-2.0.0.tar.gz

pip install file://`pwd`/en_core_web_sm-2.0.0.tar.gz#en_core_web_sm
pip install numpy

Skill setup in the console (for passing through raw text)

Amazon has a data type called AMAZON.LITERAL which will give you the text of the user's utterance. We can use this type to pass the full raw string to our application, and do all of the utterance processing ourselves. To do this, in the Alexa Skills console under the new Interaction Model interface we need set up our intent:

"intents": [
    ...
    {
    "name": "Spacy",
    "samples": [
      "{how are you|rawtext}",
      "{help|rawtext}",
      "{what time is it|rawtext}"
    ],
    "slots": [
      {
        "name": "rawtext",
        "type": "AMAZON.LITERAL"
      }
    ]
  }

Classifying the utterance

Our application will have three main routes that a user can take. The user can first ask for help, and get back some help. Secondly the user can ask 'how do you feel'. And finally, the user can ask what time it is, and optionally provide a city.

We will use Spacy to determine if the utterance is a member of one of these three CLASSES (or of none of them) - and use that information to decide which route the user is taking through the application.

First we set up a dictionary of CLASSES with example utterances:

CLASSES={
    "help": [
        "help",
        "i am lost",
        "help me"
    ],
    "time": [
        "what time is it",
        "what time is it in london",
        "when is it now",
        "what is the time"
    ],
    "mood": [
        "how are you doing",
        "what mood are you in",
        "what do you think"
    ]
}
# If Spacy is less confident then 75% then its an unknown utterance.
THRESHOLD=0.75

Next we write the classification function. This function will take the user's utterance, and have Spacy calculate its semantic similarity to all of the example utterances in our CLASSES. If it matches one of these CLASSES at high confidence we return the class name - if it doesn't match any of them we return unknown.

import en_core_web_sm
nlp = en_core_web_sm.load()

def nlp_classify(voice_string):

    # Convert to Spacy 'doc'
    utter = nlp(unicode(voice_string))
    scores = []
    cats = CLASSES
    cats_keys = cats.keys()
    # Iterate through each of the sample utterances...
    for idx,key in enumerate(cats_keys):
        v = cats[key]
        for i in v:
            # Spacy calculates the semantic similarity between
            # the user's utterance and the example utterance -
            # and gives a similarity score
            sample = nlp(unicode(i))
            sim_score = utter.similarity(sample)
            scores.append([idx,sim_score])
    # We now find the example utterance with the highest
    # similarity score, and determine its CLASS
    a = np.array(scores)
    my_cat = cats_keys[int(a[np.argmax([e for e in a[:, 1]])][0])]
    my_val = np.max([e for e in a[:, 1]])
    # if the similarity score was below THRESHOLD, then the
    # user uttered something that was not close to any of our
    # example utterances, and we treat it as an 'unknown' utterance.
    if my_val < THRESHOLD:
        my_cat = 'unknown'
    return my_cat, my_val

And then in our main routing code - we get the label/class, and take the appropriate action based on that class:

@ask.launch
@ask.intent('Spacy', default={'rawtext': ''})

def mainroute(rawtext=None):
    nlp_label, nlp_score = nlp_classify(rawtext)
    if nlp_label == "mood":
        r_st = "Good afternoon. Everything is going extremely well."
        return statement(r_st).simple_card('My Mood', r_st)
    elif nlp_label == 'time':
        return get_time(rawtext)
    elif nlp_label == 'unknown':
        return r_help()

Using Spacy to identify Entities

Spacy can also detect and identify entities (generally, proper nouns). We can use this feature to pull information from the user's utterance. For example, we can give the user the current time in a particular city instead of just the local time. To do that we have Spacy identify any Entities . If an entity such as a city is present, we geolocate the city, get its timezone, and then return the local time in that city.


#  GETTING A TIME AT A CITY
from datetime import datetime
from geopy import geocoders
from tzwhere import tzwhere
from pytz import timezone

tz = tzwhere.tzwhere()
TIME_FORMAT='%-I %M %p'

def get_time(voice_string):
    # We use 'title' to capitalize all the words, entities being proper
    # nouns are capitalized. 
    doc = nlp(voice_string.title())
    print "DOC ENTS"
    print len(doc.ents)
    print "STRING"
    print voice_string
    # If an entity is detected, assume it's a specific city!
    if len(doc.ents) > 0:
        # get the city name, geocode it with Google, find the
        # timezone from the lat/lon and then get the time in that zone
        city = doc.ents[0]
        g = geocoders.GoogleV3()
        place, (lat, lng) = g.geocode(city)
        tz_str = tz.tzNameAt(lat,lng)
        c_tz = timezone(tz_str)
        now_time = datetime.now(c_tz)
        my_loc = "in %s" % city

    # Otherwise, it's just the 'local' time
    else:
        now_time = datetime.now()
        my_loc = ""

    nice_time = datetime.strftime(now_time,TIME_FORMAT)
    response = "The time %s is %s" % (my_loc, nice_time)
    return statement(response).simple_card('The Time', response)

Summary

This just touches the tip of the iceberg on the capabilites of Spacy for processing utterances for digital assistants such as Alexa and OkGoogle. You no longer have to depend on third parties to create the conversational interfaces for your Alexa skills!

Deploying ML workloads with Azure and Kubernetes

2018-03-05T00:00:00-08:00

Running Task Screenshot

Today we'll be exploring deploying Machine Learning workloads to a Kubernetes cluster deployed on the Microsoft Azure cloud. The main focus of this is not a deep dive on the ML methods used, but to focus instead on the infrastructure considerations when deploying these types of workloads.

I'm currently looking for my next DevOps / Cloud infrastructure automation gig - this time focusing on Machine Learning. So it was a good time to try out these technologies used by many ML shops. For a video on how ML companies are using Kubernetes for their workloads check out Building the Infrastructure that Powers the Future of AI .

This post will cover the basics of pushing ML jobs onto a Kubernetes cluster. But we won't cover more advanced topics like using GPU instances or autoscaling based on workload. I apologize for the terseness of this posting - its a long and complex process and so I've kept it terse to prevent it from becoming much too long!

My environment is a an Ubuntu Xenial VM with docker, Xwindows, and a web browser. If you do not have one of these you can spin up a Ubuntu/Xenial VM in Azure. The steps are the same for other flavors of Linux - but the command lines will differ.

All of my Dockerfile and Kubernetes configurations are available here: https://github.com/gregretkowski/kube-es-starter .

Intro to Evolution Strategies and Roboschool

The ML task we'll be tackling today is to use Evolution Strategies to teach a bipedal robot to walk. A bit about Evolution Strategies from OpenAI:

We've discovered that evolution strategies (ES), an optimization technique that's been known for decades, rivals the performance of standard reinforcement learning (RL) techniques on modern RL benchmarks (e.g. Atari/MuJoCo), while overcoming many of RL's inconveniences.

-- OpenAI Blog: Evolution Strategies

My main interest in it is that it is a learning strategy that is well suited to being deployed as a distributed system, demonstrating some of the features of deploying to a Kubernetes cluster. It was straightforward to take the Evolution Strategies Starter Code and adapt it with a few modifications to be the experiment running on our cluster.

The environment our ES will train on is the Roboschool Humanoid task. Roboschool is a set of environments from OpenAI which replicate the Mujoco environments but use an open source physics engine, Bullet3. Because the cost for Mujoco licenses - the cheapest license available for anyone to run within a container is $2000 - the availability of Roboschool opens up experimentation with physics environments to a much wider audience.

What our implementation will look like

We will set up a Kubernetes cluster running in Azure. Our Kubernetes job will have a single master and many workers. Master and workers will run from the same Docker container image, differentiated by running a different startup script. The master will provide a Redis service for communicating to the workers. Result snapshots will be written out by the master to an Azure file share.

Working with a local container

The first step is to get things working locally. Creating and debugging our docker container locally allows us to quickly iterate. Once we have it working locally we can move onto deploying it at scale in a cloud service.

Creating the container

The workflow I've found most effective is to simply start with a base container in an interactive shell, and then run each command to set up your desired environment. I copy/paste the working command lines into my notes and use those notes to create the Dockerfile.

docker run -i -t ubuntu:xenial /bin/bash
> root@5fd5a12008ab:/#
apt-get update
apt-get dist-upgrade
apt-get install python3 python3-pip git build-essential
...

I ended up with a lengthy Dockerfile. You can read along with the Dockerfile on GitHub. The Dockerfile contains the following:

Start with an Ubuntu Xenial container
Update the OS and install build tools
Install the Roboschool dependencies
Install the Bullet3 physics engine
Finish Roboschool installation
Install an X11 virtual framebuffer
Install evolution-strategies-starter & dependencies
Install and configure redis, needed for the ES implementation
Add our task definition JSON file humanoid.json
Add startup scripts for master and worker nodes

Here's a small part of the Dockerfile, setting up the ES starter.

# Install prereqs & evolution-strategies-starter
RUN apt-get -y install tmux screen vim
RUN apt-get -y install python3-click python3-redis python3-h5py
RUN pip3 install tensorflow==0.12.0
RUN git clone https://github.com/openai/evolution-strategies-starter.git
# Make it Roboschool compatible.
RUN sed -i "s|^    import gym$|    import gym, roboschool|g" /evolution-strategies-starter/es_distributed/es.py
RUN sed -i "s|^    import gym$|    import gym, roboschool|g" /evolution-strategies-starter/scripts/viz.py

Once we've finished editing our Dockerfile, we'll create our docker image ala:

docker build -t gretkowski/kubeevo:v1 -f Dockerfile .

Testing and debugging the docker image locally

To test locally we'll use docker to set up a test network, and then launch one master and one worker container. Chris Sainty has a good rundown of how to connect docker containers. We'll use those instructions to create a docker network and connect our two containers over that network.

# start the master.
mkdir -p logdir
docker network create --driver=bridge testnet
docker run --net=testnet --name redis -i -t \
    -v `pwd`/logdir:/logdir gretkowski/kubeevo:v1  ./master-start.sh
# In a separate terminal window
docker run --net=testnet --name worker -i -t \
    gretkowski/kubeevo:v1 ./worker-start.sh

With the containers running we should be able to see the worker connect to the redis server on the master. We should see the worker pick up tasks from the master and push results up to the master.

Showing results

We can even use the container we created to show the results of our experiments. The docker container can display visualizations based on the latest checkpoint file.. As long as we allow the container access to open up a window on our X11 display.

# NOTE - this opens up your X display to the world!
xhost +
docker run --net=testnet -i -t -v `pwd`/logdir:/logdir --rm \
    -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix \
    gretkowski/kubeevo:v1 /bin/bash
# Should now be in the container
ls /logdir/es_master*
# Note the name of the newest snapshot file for the next command...
python3 -m scripts.viz RoboschoolHumanoid-v1 \
    /logdir/es_master_201803051820/snapshot_iter00020_rew100.h5 \
    --stochastic

Once we are satisfied that everything is working as you expect, we'll clean up our docker environment.

docker rm redis
docker rm worker
docker network rm testnet

Setting up the Azure Kubernetes cluster

The first step is to get an Azure account. Its easy to do, but you will need to have a credit card.

Get the trial account, right now you get $200 in free credit.
Upgrade it to the pay-as-you-go account unless you only want to use just a few very basic VM's. You need to upgrade to get access to better VM's.

Once you have your account established its a good time to familiarize yourself with the Microsoft Azure Portal.

Getting your Kubernetes cluster set up

The next step is to set up our Kubernetes cluster within Azure. For more verbose instructions check out the Azure walk-through for setting up Kubernetes.

First we'll install the Azure CLI tool az and then install the Kubernetes CLI tool kubectl. If you are running something other than Xenial check out Installing the Azure CLI for instructions for your platform.

AZ_REPO=$(lsb_release -cs)
echo "deb [arch=amd64] https://packages.microsoft.com/repos/azure-cli/ $AZ_REPO main" | \
    tee /etc/apt/sources.list.d/azure-cli.list
apt-key adv --keyserver packages.microsoft.com --recv-keys \
    52E16F86FEE04B979B07E28DB02C46DF417A0893
apt-get install apt-transport-https
apt-get update && apt-get install azure-cli

We'll verify the tools are working, by first logging in to Azure (requires a web browser) and then doing an acr list command

az login
az acr list

We need to create a resource group to contain all of our Kubernetes resources, this also makes it easier later to delete all our resources in one go, rather than having orphaned resources racking up your bill.

az group create --name myResourceGroup --location west-us-2
az acs create --orchestrator-type kubernetes \
    --resource-group myResourceGroup --name demoKubeCluster \
    --agent-count 1 --generate-ssh-keys

Next we'll get kubectl installed and configured to talk to your cluster:

az acs kubernetes install-cli
az acs kubernetes get-credentials --resource-group demoKubeGroup \
    --name demoKubeCluster --file `pwd`/kubeconfig
export KUBECONFIG=`pwd`/kubeconfig
kubectl top node

Setting up a private container registry

We need to have a place to put our docker images that Kubernetes can retrieve them when deploying. To do this we'll deploy a private container registry within Azure.

az acr create --resource-group demoKubeGroup \
    --name demoKubeRegistry --sku Basic
az acr login --name demoKubeRegistry

We'll try to push an image into the registry, just to verify everything is working.

docker pull microsoft/aci-helloworld
docker tag microsoft/aci-helloworld demokuberegistry.azurecr.io/aci-helloworld:v1
docker push demokuberegistry.azurecr.io/aci-helloworld:v1
az acr repository list --name demoKubeRegistry --output table

Pushing your container image into the registry

Now is a good time to push our container to the registry. To do that we will tag the container with the remote registry hostname, and then push the container.

docker tag gretkowski/kubeevo:v1 demokuberegistry.azurecr.io/kubeevo:v1
docker push  demokuberegistry.azurecr.io/kubeevo:v1
az acr repository list --name demoKubeRegistry --output table

Azure file service setup

To get results out of our experiment we need our master node to write out checkpoints to a place which is easy for us to access. To accomplish this we will have our master node attach a volume which maps to an Azure File Share. We can then download checkpoint files by browsing through the Azure console.

Azure Files Configuration

The implementation is a mix of the one provided in Francisco Beltrao's post and Karim Vaes' post

az group list --output table
# Note the long one associated with your Kube cluster

az storage account create \
    --resource-group demoKubeGroup_demoKubeCluster_westus2 \
    --name kubeevologdir --location westus2 --sku Standard_LRS
az storage account show-connection-string \
--resource-group demoKubeGroup_demoKubeCluster_westus2 \
--name kubeevologdir

az storage share create --name kubeevologdir \
    --connection-string "THE_CONNECTION_STRING_FROM_THE_LAST_CMD"

az storage account keys list \
    --resource-group demoKubeGroup_demoKubeCluster_westus2 \
    --account-name kubeevologdir --query "[0].value" -o tsv
# Note the 'storage key' for the next commands.

# These values go into secrets/secrets/kubeevologdir.yaml
echo -n kubeevologdir | base64
echo -n {enter the storage key} | base64

# Now create our Kubernetes resources
kubectl create -f secrets/kubeevologdir.yaml
kubectl create -f pv/kubeevologdir.yaml
kubectl create -f pvc/kubeevologdir.yaml

Now kubectl get persistentvolumes and kubectl get persistentvolumeclaims should indicate the volume is ready to use.

Starting up our 'master' container

Read through the definition of our master master.yaml. Some of the highlights here:

It uses the container image we uploaded to our private registry
It defines a port 6379, opened up for Redis
It mounts the kubeevologdir volume on /logdir
The startup command is the ./master-start.sh script

We'll start up our master, and monitor its progress:

kubectl -f deployments/master.yaml
kubectl get pods
kubectl describe pods master

Hopefully you end up with a pod in the Running state. If not the describe argument will often have some indication on why the pod failed to start.

You can check out the logs from the running container using the logs argument and the pod name, i.e. kubectl logs master-1285880246-1x227. To login to the pod to debug something locally use kubectl exec -it master-1285880246-1x227 -- /bin/bash .

Starting our workers

The master is running Redis, but to make that available to workers we need to tell Kubernetes that its a 'service' - which creates a DNS name the workers can reference.

 kubectl create -f services/redis.yaml

Next we can start up a couple of workers

kubectl create -f deployments/workers-2.yaml

Using kubectl logs should show the workers connecting to the master and pulling down tasks:

kubectl logs worker-3207115605-v0lwh
...
[2018-03-06 23:06:43,344 pid=34] [relay] Received task b'0'

Scaling up the cluster & adding more workers

Next lets scale up the number of nodes in our Kubernetes cluster, and then add several more workers. When we created the cluster we only added a single agent. We then only launched 2 worker containers. Here we'll scale up to have a total of 4 agents, and 8 workers.

az acs show --name demoKubeCluster --resource-group demoKubeGroup
az acs scale --name demoKubeCluster --resource-group demoKubeGroup --new-agent-count 4
kubectl replace -f deployments/workers-8.yaml
kubectl get pods

Checking out progress, and grabbing results

If you run kubectl logs for your master you should see the experiment chugging along...

********** Iteration 2 **********
[2018-03-06 23:12:18,352 pid=29] Skipped 25 out of date results (1.15%)
----------------------------------
| EpRewMean           | 32.7     |
| EpRewStd            | 13.1     |
| EpLenMean           | 32.7     |
| EvalEpRewMean       | 44.2     |
| EvalEpRewStd        | 14.6     |
| EvalEpLenMean       | 39.3     |
| EvalPopRank         | 0.746    |
| EvalEpCount         | 6        |
| Norm                | 501      |
| GradNorm            | 0.701    |
| UpdateRatio         | 0.0737   |
| EpisodesThisIter    | 1e+04    |
| EpisodesSoFar       | 3.04e+04 |
| TimestepsThisIter   | 3.27e+05 |
| TimestepsSoFar      | 8.39e+05 |
| UniqueWorkers       | 8        |
| UniqueWorkersFrac   | 0.00368  |
| ResultsSkippedFrac  | 0.0115   |
| ObCount             | 3.69e+03 |
| TimeElapsedThisIter | 83.7     |
| TimeElapsed         | 765      |
----------------------------------
********** Iteration 3 **********

The experiment will write out checkpoint files every 20 iterations. To access the files the most expedient way is to browse through the Azure Portal. Go to 'Storage Accounts' then drill into kubeevologdir, 'files', kubeevologdir. You can then take those h5 checkpoint files and use the scripts.viz command line we used previously to look at the progress of the experiment in learning a solution.

And finally, saving our bank balance

After we have finished using your Kubernetes cluster, we'll be sure to clean it up - making sure it doesn't continue to rack up big charges for VM's and other Azure resources. A big plus is that Azure kept all our resources in a ResourceGroup so to clean everything up we can simply delete that group:

az group delete --name demoKubeGroup

Additional Resources

Kubernetes Cheatsheet

Actor Critic with OpenAI Gym

2016-07-05T00:00:00-07:00

This tutorial was inspired by Outlace's excelent blog entry on Q-Learning and this is the starting point for my Actor Critic implementation. I highly recommend you read his three tutorials on Reinforcement Learning first.

I got interested in Actor-Critic reinforcement learning after skimming DeepMind's follow-up to their original Atari paper. The new paper Asynchronous Methods for Deep Reinforcement Learning uses an Actor/Critic learning implementation to surpass the performance of their original Deep Q-Network. I couldn't find a good straightforward implementation/example of it.

For a slightly more in-depth explination check out Actor-Critic Methods from the book 'Reinforcement Learning: An Introduction' by Sutton and Barto

This will be a basic example of using Actor Critic methods to solve a simple reinforcement learning problem from OpenAI Gym

Couple thing's we have here:

Actor Critic
Neural Networks as Function Approximators for both Actor and Critic
Experience Replay to avoid catastrophic forgetting

If you are running the ipython notebook this tutorial expects you to have OpenAI Gym and Keras installed. The Notebook is available from GitHub.

Lets get to coding...

We are going to start off with some bookkeeping - importing the components we need.

%pylab inline
# Just some initial setup and imports
import logging
logger = logging.getLogger()
logger.setLevel(logging.ERROR)
from IPython.display import clear_output
import time
import numpy as np
import math
import gym

Populating the interactive namespace from numpy and matplotlib

For the environment our agent is going to interact with we'll use the OpenAI Gym, and use a variation of an existing environment 'Frozen Lake' - however we're going to make a version which does not include slippery ice. This simplification will make it much easier to visualize what's happening within our Actor/Critic implementation.

# Create a non-skid version of Frozen Lake
from gym.envs.registration import register, spec

MY_ENV_NAME='FrozenLakeNonskid8x8-v0'
try:
    spec(MY_ENV_NAME)
except:
    register(
        id=MY_ENV_NAME,
        entry_point='gym.envs.toy_text:FrozenLakeEnv',
        kwargs={'map_name': '8x8', 'is_slippery': False},
        timestep_limit=100,
        reward_threshold=0.78, # optimum = .8196
    )
env = gym.make(MY_ENV_NAME)

[2016-07-05 20:13:07,755] Making new env: FrozenLakeNonskid8x8-v0

I'll define some constants, to make the code clearer and easier later. The constants OBSERVATION_SPACE and ACTION_SPACE come from OpenAI gym. For this problem each of these is just a single integer. I then define a constant OBS_SQR and STATEGRID which will help when we create visualizations showing the square grid of our gridworld. These constants are not needed for the learner.

Finally I create a helper which will allow us to convert the observation from OpenAI Gym, from an integer, like 64, to a one-hot encoding - which is better for a neural network to consume.

# This helper is for later. 
def to_onehot(size,value):
  my_onehot = np.zeros((size))
  my_onehot[value] = 1.0
  return my_onehot

OBSERVATION_SPACE = env.observation_space.n
ACTION_SPACE = env.action_space.n

# Assume gridworld is always square
OBS_SQR= int(math.sqrt(OBSERVATION_SPACE))
STATEGRID = np.zeros((OBS_SQR,OBS_SQR))

The Actor

In Actor/Critic there are two networks. The Policy network (the Actor) and the Value network (the Critic). You will recognize the policy network as being essentially the same as the network from the Q-Learning example referenced above.

from keras.models import Sequential
from keras.layers.core import Dense, Dropout, Activation
from keras.optimizers import RMSprop, SGD

actor_model = Sequential()
actor_model.add(Dense(164, init='lecun_uniform', input_shape=(OBSERVATION_SPACE,)))
actor_model.add(Activation('relu'))

actor_model.add(Dense(150, init='lecun_uniform'))
actor_model.add(Activation('relu'))

actor_model.add(Dense(ACTION_SPACE, init='lecun_uniform'))
actor_model.add(Activation('linear'))

a_optimizer = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
actor_model.compile(loss='mse', optimizer=a_optimizer)

Using Theano backend.

The Critic

Next, we set up the Critic network. This network looks very similar to the Actor model, but only outputs a single value - it outputs a value (the score) for the input state.

critic_model = Sequential()

critic_model = Sequential()
critic_model.add(Dense(164, init='lecun_uniform', input_shape=(OBSERVATION_SPACE,)))
critic_model.add(Activation('relu'))
critic_model.add(Dense(150, init='lecun_uniform'))
critic_model.add(Activation('relu'))
critic_model.add(Dense(1, init='lecun_uniform'))
critic_model.add(Activation('linear'))

c_optimizer = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
critic_model.compile(loss='mse', optimizer=c_optimizer)

Now we'll add a nice helper function to show what value the Critic gives to being in certain states. The critic starts out uninitialized and will just give random values for any given state.

# Plot out the values the critic gives for the agent being in
# a specific state, i.e. in a specific location in the env.
def plot_value(initial_state):
    # Assume gridworld is always a square
    obs_sqr = math.sqrt(OBSERVATION_SPACE)
    np_w_cri_r = np.zeros((OBS_SQR,OBS_SQR))
    # make a working copy.
    working_state = initial_state.copy()
    for x in range(0,OBS_SQR):
        for y in range(0,OBS_SQR):
            my_state = working_state.copy()
            # Place the player at a given X/Y location.
            my_state[x,y] = 1
            # And now have the critic model predict the state value
            # with the player in that location.
            value = critic_model.predict(my_state.reshape(1, OBSERVATION_SPACE))
            np_w_cri_r[x,y] = value
    np_w_cri_r.shape
    pylab.pcolor(np_w_cri_r)
    pylab.title("Value Network")
    pylab.colorbar()
    pylab.xlabel("X")
    pylab.ylabel("Y")
    pylab.gca().invert_yaxis()
    pylab.draw()


env.reset()
env.render()
plot_value(STATEGRID)

_FFFFFFF
FFFFFFFF
FFFHFFFF
FFFFFHFF
FFFHFFFF
FHHFFFHF
FHFFHFHF
FFFHFFFG

This may not be necessary, or even possible in all cases - but in this case we can zero-out the Critic network, so that all states start with a value of Zero. This should help it converge later when it sees real training data.

def zero_critic(epochs=100):
    for i in range(epochs):
        for j in range(OBSERVATION_SPACE):
            X_train = []
            y_train = []

            y = np.empty([1])
            y[0]=0.0
            x = to_onehot(OBSERVATION_SPACE,j)
            X_train.append(x.reshape((OBSERVATION_SPACE,)))
            y_train.append(y.reshape((1,)))
            X_train = np.array(X_train)
            y_train = np.array(y_train)
            critic_model.fit(X_train, y_train, batch_size=1, nb_epoch=1, verbose=0)

print("Zeroing out critic network...")
sys.stdout.flush()
zero_critic()
print("Done!")
plot_value(STATEGRID)

Zeroing out critic network...
Done!

The Trainer

The trainer is implemented below.

The big main new things here are the split of the Critic network, and state values, from the Actor, and action selection.

To train the critic network, we use a simlar process to training value into Q networks. We look at the initial state, make a move, and then look at the new state. For the value network, if we are in a terminal state, that's the value we tell the value network to place on that state. If we are in a non-terminal state, we tell the value network to place a value on the original state which is the reward in the original state, plus the discounted value from the new state. Note that the value network should return the maximum possible value for a given state. If the player's next move could be either jumping into the pit or arriving at the goal, we should set the value as if the best-possible action will be selected.

After the critic network has assigned a value to the original and the new state, we adjust the policy. This is simply by looking at our value in our old state, and the value in the new state. If the value improves we encourage that action. If it decreases we discourage the action.

When we start training, both the actor and the critic networks are spitting out nonsensical values. Which means initially the actor network is training on values from the critic network which are garbage. However as the critic network improves, those improvements naturally correct and improve the performance of the actor network.

The Experience Replay is essentially the same as the example in the Q-learning tutorial. However we replay both the Actor and the Critic's experiences.

from IPython.display import clear_output
import random
import time

def trainer(epochs=1000, batchSize=40, 
            gamma=0.975, epsilon=1, min_epsilon=0.1,
            buffer=80):

    wins = 0
    losses = 0
    # Replay buffers
    actor_replay = []
    critic_replay = []

    for i in range(epochs):

        observation = env.reset()
        done = False
        reward = 0
        info = None
        move_counter = 0

        while(not done):
            # Get original state, original reward, and critic's value for this state.
            orig_state = to_onehot(OBSERVATION_SPACE,observation)
            orig_reward = reward
            orig_val = critic_model.predict(orig_state.reshape(1,OBSERVATION_SPACE))

            if (random.random() < epsilon): #choose random action
                action = np.random.randint(0,ACTION_SPACE)
            else: #choose best action from Q(s,a) values
                qval = actor_model.predict( orig_state.reshape(1,OBSERVATION_SPACE) )
                action = (np.argmax(qval))

            #Take action, observe new state S'
            new_observation, new_reward, done, info = env.step(action)
            new_state = to_onehot(OBSERVATION_SPACE,new_observation)
            # Critic's value for this new state.
            new_val = critic_model.predict(new_state.reshape(1,OBSERVATION_SPACE))

            if not done: # Non-terminal state.
                target = orig_reward + ( gamma * new_val)
            else:
                # In terminal states, the environment tells us
                # the value directly.
                target = orig_reward + ( gamma * new_reward )

            # For our critic, we select the best/highest value.. The
            # value for this state is based on if the agent selected
            # the best possible moves from this state forward.
            # 
            # BTW, we discount an original value provided by the
            # value network, to handle cases where its spitting
            # out unreasonably high values.. naturally decaying
            # these values to something reasonable.
            best_val = max((orig_val*gamma), target)

            # Now append this to our critic replay buffer.
            critic_replay.append([orig_state,best_val])
            # If we are in a terminal state, append a replay for it also.
            if done:
                critic_replay.append( [new_state, float(new_reward)] )

            # Build the update for the Actor. The actor is updated
            # by using the difference of the value the critic
            # placed on the old state vs. the value the critic
            # places on the new state.. encouraging the actor
            # to move into more valuable states.
            actor_delta = new_val - orig_val                
            actor_replay.append([orig_state, action, actor_delta])

            # Critic Replays...
            while(len(critic_replay) > buffer): # Trim replay buffer
                critic_replay.pop(0)
            # Start training when we have enough samples.
            if(len(critic_replay) >= buffer):
                minibatch = random.sample(critic_replay, batchSize)
                X_train = []
                y_train = []
                for memory in minibatch:
                    m_state, m_value = memory
                    y = np.empty([1])
                    y[0] = m_value
                    X_train.append(m_state.reshape((OBSERVATION_SPACE,)))
                    y_train.append(y.reshape((1,)))
                X_train = np.array(X_train)
                y_train = np.array(y_train)
                critic_model.fit(X_train, y_train, batch_size=batchSize, nb_epoch=1, verbose=0)

            # Actor Replays...
            while(len(actor_replay) > buffer):
                actor_replay.pop(0)                
            if(len(actor_replay) >= buffer):
                X_train = []
                y_train = []
                minibatch = random.sample(actor_replay, batchSize)
                for memory in minibatch:
                    m_orig_state, m_action, m_value = memory
                    old_qval = actor_model.predict( m_orig_state.reshape(1,OBSERVATION_SPACE,) )
                    y = np.zeros(( 1, ACTION_SPACE ))
                    y[:] = old_qval[:]
                    y[0][m_action] = m_value
                    X_train.append(m_orig_state.reshape((OBSERVATION_SPACE,)))
                    y_train.append(y.reshape((ACTION_SPACE,)))
                X_train = np.array(X_train)
                y_train = np.array(y_train)
                actor_model.fit(X_train, y_train, batch_size=batchSize, nb_epoch=1, verbose=0)

            # Bookkeeping at the end of the turn.
            observation = new_observation
            reward = new_reward
            move_counter+=1
            if done:
                if new_reward > 0 : # Win
                    wins += 1
                else: # Loss
                    losses += 1
        # Finised Epoch
        clear_output(wait=True)
        print("Game #: %s" % (i,))
        print("Moves this round %s" % move_counter)
        print("Final Position:")
        env.render()
        print("Wins/Losses %s/%s" % (wins, losses))
        if epsilon > min_epsilon:
            epsilon -= (1/epochs)

We'll now train the Actor/Critic for a number of epoch's which seems to give decent results.

%%time
trainer()

Game #: 999
Moves this round 19
Final Position:
SFFFFFFF
FFFFFFFF
FFFHFFFF
FFFFFHFF
FFFHFFFF
FHHFFFHF
FHFFHFHF
FFFHFFF_
  (Down)
Wins/Losses 312/688
Wall time: 5min 31s

Now we can examine what the value network has learned - and the values it has placed on any given location on the board.

env.reset()
env.render()
plot_value(STATEGRID)

_FFFFFFF
FFFFFFFF
FFFHFFFF
FFFFFHFF
FFFHFFFF
FHHFFFHF
FHFFHFHF
FFFHFFFG
  (Down)

We can see that the value network has placed a high value on the winning final position. That value equaling the reward gained when in that position. It also places a very low value on the 'hole' positions. Makes sense. Then we can see that the network places ever growing value on positions which move us closer to the winning position. Thus the value network can express to the policy network that a move which moves us closer to the winning position is more valuable. Lets take a look at what the policy network has learned.

# Maps actions to arrows to indicate move direction.
A2A=['<','v','>','^']
def show_policy(initial_state):
    grid = np.zeros((OBS_SQR,OBS_SQR), dtype='<U2')
    #working_state = initial_state.copy()
    #p = findLoc(working_state, np.array([0,0,0,1]))
   #working_state[p[0],p[1]] = np.array([0,0,0,0])
    for x in range(0,OBS_SQR):
        for y in range(0,OBS_SQR):
            #for a in range(0, 4):
            my_state = initial_state.copy()
            my_state[x,y] = 1
            #
            obs_predict = my_state.reshape(1,OBSERVATION_SPACE,)
            qval = actor_model.predict(obs_predict)
            #print(obs_predict)

            action = (np.argmax(qval))
            grid[x,y] = A2A[action]
    grid
    return grid

env.reset()
env.render()
print(show_policy(STATEGRID))

_FFFFFFF
FFFFFFFF
FFFHFFFF
FFFFFHFF
FFFHFFFF
FHHFFFHF
FHFFHFHF
FFFHFFFG
  (Down)
[['>' 'v' 'v' '>' '^' '>' 'v' 'v']
 ['>' '>' '>' '>' 'v' 'v' 'v' 'v']
 ['^' '^' '^' '<' '>' '>' 'v' 'v']
 ['>' '^' '>' '>' 'v' '^' '>' 'v']
 ['>' '^' '^' 'v' '>' '>' '>' 'v']
 ['^' '^' '<' '>' '^' '^' '>' 'v']
 ['<' '^' '^' '^' 'v' '^' '^' 'v']
 ['^' '>' '>' '>' '>' '<' '^' '^']]

Now we'll have our learner show us what it has learned. It is important to note, that when testing, we only need the actor/policy network. The critic network is not involved. The Actor has learned the correct policy/moves as it trains on the values supplied by the critic network.

def play(render_every_step=False):
    observation = env.reset()
    done = False
    reward = 0.0
    max_moves = 40
    move_counter = 0
    while not done and move_counter < max_moves:
        state = to_onehot(OBSERVATION_SPACE,observation)
        qval = actor_model.predict( state.reshape(1,OBSERVATION_SPACE) )
        action = (np.argmax(qval))
        observation, reward, done, info = env.step(action)
        print(A2A[action])
        if render_every_step:
            env.render()
        move_counter += 1
    env.render()

play(render_every_step=True)

>
S_FFFFFF
FFFFFFFF
FFFHFFFF
FFFFFHFF
FFFHFFFF
FHHFFFHF
FHFFHFHF
FFFHFFFG
  (Right)
v
SFFFFFFF
F_FFFFFF
FFFHFFFF
FFFFFHFF
FFFHFFFF
FHHFFFHF
FHFFHFHF
FFFHFFFG
  (Down)
>
SFFFFFFF
FF_FFFFF
FFFHFFFF
FFFFFHFF
FFFHFFFF
FHHFFFHF
FHFFHFHF
FFFHFFFG
  (Right)
>
SFFFFFFF
FFF_FFFF
FFFHFFFF
FFFFFHFF
FFFHFFFF
FHHFFFHF
FHFFHFHF
FFFHFFFG
  (Right)
>
SFFFFFFF
FFFF_FFF
FFFHFFFF
FFFFFHFF
FFFHFFFF
FHHFFFHF
FHFFHFHF
FFFHFFFG
  (Right)
v
SFFFFFFF
FFFFFFFF
FFFH_FFF
FFFFFHFF
FFFHFFFF
FHHFFFHF
FHFFHFHF
FFFHFFFG
  (Down)
>
SFFFFFFF
FFFFFFFF
FFFHF_FF
FFFFFHFF
FFFHFFFF
FHHFFFHF
FHFFHFHF
FFFHFFFG
  (Right)
>
SFFFFFFF
FFFFFFFF
FFFHFF_F
FFFFFHFF
FFFHFFFF
FHHFFFHF
FHFFHFHF
FFFHFFFG
  (Right)
v
SFFFFFFF
FFFFFFFF
FFFHFFFF
FFFFFH_F
FFFHFFFF
FHHFFFHF
FHFFHFHF
FFFHFFFG
  (Down)
>
SFFFFFFF
FFFFFFFF
FFFHFFFF
FFFFFHF_
FFFHFFFF
FHHFFFHF
FHFFHFHF
FFFHFFFG
  (Right)
v
SFFFFFFF
FFFFFFFF
FFFHFFFF
FFFFFHFF
FFFHFFF_
FHHFFFHF
FHFFHFHF
FFFHFFFG
  (Down)
v
SFFFFFFF
FFFFFFFF
FFFHFFFF
FFFFFHFF
FFFHFFFF
FHHFFFH_
FHFFHFHF
FFFHFFFG
  (Down)
v
SFFFFFFF
FFFFFFFF
FFFHFFFF
FFFFFHFF
FFFHFFFF
FHHFFFHF
FHFFHFH_
FFFHFFFG
  (Down)
v
SFFFFFFF
FFFFFFFF
FFFHFFFF
FFFFFHFF
FFFHFFFF
FHHFFFHF
FHFFHFHF
FFFHFFF_
  (Down)
SFFFFFFF
FFFFFFFF
FFFHFFFF
FFFFFHFF
FFFHFFFF
FHHFFFHF
FHFFHFHF
FFFHFFF_
  (Down)

Conclusion

We've demonstrated using an actor-critic learner to solve a toy gridworld problem. I hope this serves as a good jumping off point for folks trying to understand Actor/Critic, or other RL methods which separate the Policy network from the Value network. If you have questions or comments please leave them below!

"Eject Alarm Clock Icon"

2015-04-23T00:00:00-07:00

Attention morning zombies: prepare to give up your snooze addiction. I created an awesome new feature on my latest android app: Eject Alarm Clock. It's different from every other alarm out there because it actually forces you to get out of bed and go to another room to turn it off. The magic happens with this new indoor positioning code I wrote.

Android friends please download Eject Alarm Clock and see if it isn't the most stubborn alarm app you've ever tried. I'm committed to getting you out of bed in the morning. Bonus: for a limited time, we're giving our friends all the premium features for free. Download now: Eject Alarm Clock, Google Play Store

"Installing Docker on Centos 6"

2013-08-04T00:00:00-07:00

If you haven't checked out docker yet you definitely should! It hype says it's going to do to application deployment what intermodal containers did to the cargo transportation industry. That's a tall order, and we'll see if it lives up to such a lofty goal. Regardless, it is a technology that could solve a lot of problems for Development and Operations teams and it is worth checking out.

Being such a new project it's built to run on the latest-and-greatest version of a single distribution, Ubuntu. However many of us are working at CentOS/RHEL shops, and to use this technology you'll need to jump through a few extra hoops to get it deployed.

Docker does its magic via LXC, cgroups, and a layered filesystem called AUFS. LXC is currently included in RHEL kernels, but AUFS is not. You'll need to install components of both these systems to get docker working on your system.

Upgrading your Kernel
First you'll need to update your kernel to one that supports AUFS. The one I use is one built by dotcloud. It conflicts with the kernel-firmware package, so remove that first, install the kernel the update your initrd via dracut.
rpm -e kernel-firmware rpm -i http://get.docker.io/kernels/kernel-3.2.40_grsec_dotcloud-4.x86_64.rpm /sbin/dracut --add-drivers dm-mod --add-drivers linear "" 3.2.40-grsec- dotcloud

edit /boot/grub/grub.conf to add a new entry for the 3.2.40 kernel, and append 'selinux=0' to the end of your command line. The dotcloud kernel isn't compiled with selinux support. Then use grub-install /dev/(your boot disk) to install the updated bootloader configuration.

There's a few other things to do because of the differences in this kernel's version and configuration vs. a standard RHEL kernel.
`
echo "blacklist evbug" >>/etc/modprobe.d/blacklist.conf

plymouthd doesn't behave properly w/ chroot_caps

echo "kernel.grsecurity.chroot_caps = 0" >>/etc/sysctl.conf

If you want to enable this after the system comes up:

echo "sysctl kernel.grsecurity.chroot_caps=1">>/etc/rc.local
Be sure ip forwarding is turned on. You can accomplish this via
echo "net.ipv4.ip_forward = 1" >>/etc/sysctl.conf`

You also need to prevent iptables from starting at boot, or modify your iptables rules for docker networking to work.

Be sure your system mounts the /cgroups filesystem at boot. If not, add it to /etc/fstab:
echo "none /cgroup cgroup defaults 0 0" >>/etc/fstab

You should reboot into the new kernel at this point.

Installing the required tools

Next we're going to build an aufs-utils RPM and install it.. You could just compile it from source, but if you are like me, it's likely you are doing this for a ton of systems, so much cleaner to build an RPM, and install it on many systems keeping as much as possible under the management of the package management system..

Be sure you have development tools installed.. If not yum groupinstall "Development tools" should do the trick. Next lets build the aufs-utils package and install it.. Here's how I did it under CentOS6:
wget "ftp://ftp.pbone.net/mirror/ftp5.gwdg.de/pub/opensuse/repositories/home%3 A/awk2007%3A/fixes/Fedora_17/src/aufs-util-9999-13.1.src.rpm" sudo yum install glibc-static rpmbuild --rebuild aufs-util-9999-13.1.src.rpm rpm -U (path-to)/aufs-util-9999-13.1.x86_64.rpm

You'll also need to install lxc and lxc-libs - if you have the dag repo set up you can just 'yum install' it.. Otherwise, download and install them directly:
wget ftp://ftp.univie.ac.at/systems/linux/dag/redhat/el6/en/x86_64/dag/RPMS/lx c-0.8.0-1.el6.rf.x86_64.rpm wget http://apt.sw.be/redhat/el6/en/x86_64/dag/RPMS/lxc- libs-0.8.0-1.el6.rf.x86_64.rpm rpm -U lxc-0.8.0-1.el6.rf.x86_64.rpm lxc-libs-0.8.0-1.el6.rf.x86_64.rpm

Installing the docker binaries
Finally, we'll download, install, and test the docker binaries. I tried to get compiling to work, by rebuilding the golang package from fedora on my CentOS box but didn't get it working - the binaries work just fine:

wget http://get.docker.io/builds/Linux/x86_64/docker-latest.tgz tar xzf docker-latest.tgz cd docker-latest ./docker -d & ./docker run -i -t busybox /bin/sh
The final command should give you a shell prompt from inside a busybox docker container. Hopefully it is all working for you at this point. If you had problems, or have changes to the directions, post them in the comments below!

"Monitoring and Configuration Management to restart services"

2011-11-20T00:00:00-08:00

This is the text of the talk I gave at the LSPE meetup in November 2011.

Using Monitoring & Configuration Management to restart services.

View more presentations from gregretkowski

Introduction

Good Evening. Tonight I'm going to talk about something that I hope will free up your time from firefighting during the day and help you sleep better at night instead of catching up on your sleep during thesetalks. I'm going to talk about setting up your servers so that they can recover from faults automatically, without your intervention. And once you've followed this self- healing recipe you'll be freed from firefighting your most common system failures.

My name is Greg Retkowski, and I'm an Operations Engineer at OnLive a cloud gaming startup. I've sysadmin'ed at a dozen or so internet startups around the bay area since moving out here in 1997. I'm going to talk today about the self-healing setup I used at another startup I worked at; a company called Avvenu. This setup ties together your network monitoring and your configuration management system so that common faults that your monitoring system detects can be quickly fixed by your configuration management system.

This talk is based on an article I wrote a few years ago for Oreilly. Originally it used NAGIOS and Cfengine. As most people are more familiar with puppet I've updated this talk to use it instead. With just a few small changes to tools you're already using you'll be able to tie these two systems together to resolve faults as they occur.

What's in it for me?

So, how does this setup help you?

First It'll free you up from firefighting interruptions. I didn't like getting paged late at night and having to fix common problems that a configuration management system could rectify. For example, we had some custom apache modules that'd sometimes crash the apache daemon. I'd have to VPN in and restart apache which would be easy to automate.

Second It can react faster than a human can. Once I implemented it, a pleasant side effect was that the setup would resolve common issues even faster than a human could. When I was paged I'd login and find that the system had already self-corrected. Failures were shortened by removing a human from the loop.

Third It's a hedge against technical debt. We don't want to get software from engineering that crashes every millionth request, but sometimes we do, and something like this can get us through till the next release.

Required Tools

Nagios

The first tool is NAGIOS. Most people in this room are already familiar with NAGIOS. Its the most popular open-source monitoring
package. It runs service checks against services and notifies sysadmins when things fail. It has the capability of running an external script when a failure occurs, and we'll leverage that in our setup.

Puppet

The next tool is Puppet. When I mention puppet some people think of a scarey doll with strings but in this case I'm talking about
the configuration management system developed by Luke Kaines. It will check the configuration of a host against policies you create, and will update the host to match your policies. Puppet has many capabilities that help in our system, it can correct corrupted config files, can fix directory permissions, and can ensure processes are running. In most installations puppet runs only, say, twice an hour. However it can be run on demand, and run remotely, and we'll use this capability in our setup.

As an aside, if there's one skill that'll be crucial to have in the next five years it's going to be a familiarity with configuration management systems be that puppet or chef. It's nearly impossible to manage large server farms without them. If you haven't investigated either of them yet I recommend you do.

High Level diagram

This is a high level diagram of how it is all tied together. Nagios monitors services, and when a falt occurs it triggers an RPC mechanism to tell puppet to run. Puppet is configured to ensure apache is running and if it isn't it restarts it via its init script.

Setting up puppet to start downed services

This is our puppet policy for our apache server. This may look like giberish if you aren't familiar with puppet. I'll quickly walk through it..

This class 'httpd' tells puppet how to configure apache on our system. It says we should have the apache package installed, there's a bunch of config file definitions we skip in this example, and then at the bottom we tell puppet that the service 'apache' requires the 'apache' package and that puppet should ensure it is running.

When puppet runs, if it finds the apache process is missing, it'll restart it via the apache init script. This will be how puppet will restart apache if nagios notifies it that it isn't running.

Setting up nagios with a postfail script

Now we'll talk about the changes to Nagios. To make this work you'll need to configure nagios to call an event handler script whenever a service goes into a different state. You'll need to make changes in two places. First the services config file:
The important lines in this file are the 'event_handler_enabled' and the 'event_handler' lines. The first tells nagios to turn on an event handler for state changes for this service. 'event_handler' tells nagios what event handler to use.

The next file to update is the commands config file - we add a command handle_puppetrun, which invokes the handle_puppetrun shell script with several arguments. This will tell the script what host is affected and what the service state is. I've wrapped the lines here, but the command_line line must be all on the same line.

Setting up the glu between nagios and CM

Now we are going to set up the RPC glue between Nagios and Puppet. All these examples are with Puppet 2.6

Setting up the puppet daemon on the host

First you'll need to edit puppet's auth config, and add a stanza that will allow it to accept remote requests to kick off puppet runs.

You'll also need to create an empty namespaceauth config this is a known issue with 2.6.

Next you'll need to tell puppet to listen for incoming requests. You can do this by adding a stanza like this to your main puppet config.

You could also use the 'listen' flag on the command line when invoking the puppet agent.

For testing this you should use this command line. This will cause the puppet agent to stay in the foreground and print debugging information to your console. I recommend you run it like this at first while debugging your configuration. It'll print all the logging information to the console which makes troubleshooting much easier.

Testing puppetrun from your monitoring host

You should now try invoking puppetrun from the command line, as the nagios user. This will ensure the end-to-end communication is working and that your nagios server will be able to fire off puppet when it needs to. If you are running the puppet agent in debug mode on your apache server you should see it running through its configuration.

When this isn't working it's often because the puppet certs aren't issued to both hosts, or that the users running the commands don't haveaccess to the certs. Check both of these if you have trouble. In my installation I added an entry to the sudoers file so that nagios can invoke puppetrun as root to have access to the certs.

The puppetrun invocation script

Once you are satisifed that puppet is communicating propery; You'll need the last piece the handle_puppetrun shell script. It goes into the Nagios plugins directory. Once it's in place, make sure it is executable via the NAGIOS user.

Nagios calls the script on all state changes. So the script looks for critical errors (here) and either HARD failures or three SOFT failures (here). In either case it calls puppetrun with the remote hostname this causes puppet to run on that remote host.

Demonstration

And this is how it works once it is deployed.

Here's our nagios instance happily monitoring our network.

And here I segfault the apache process

Nagios notices that apache is down and calls our handle_puppetrun script

Puppet gets invoked on our webserver, and restarts apache for us

And here Nagios has noticed that our webserver has recovered

And our network is happy again.

Other monitoring packages and CM tools

There's no reason you couldn't apply this to other network monitoring systems or configuration mangement tools. I originally had this running under cfengine, and you could use chef as well. Other monitoring systems also support event handlers, same as nagios

Conclusion

To find the examples and the original Oreilly article visit my site for this talk at this URL.

To wrap up, I've shown a setup where your network can self correct for its most common failures. We've used tools that many of you are already using -- tied together in a novel way. And I am hopeful I've freed you up from some firefighting so you'll sleep better at night and more productive during the day..

To Alaska: The US Northwest

2011-06-15T00:00:00-07:00

Mount Shasta

We headed out on our trip on the first of May. Well, to be real, we spent most of the first swapping things between our storage unit and our RV... The trip begins before the first mile ticks off the odometer.. We spent several weeks planning, and the first of May, first day of our trip, we spent packing at the storage unit, provisioning up at Costco, and rounding out or 'wine cellar'. Once we headed north we had a chance to visit with several friends as we drove north... Paul and Christine -- Cherie's first trip to Alaska with with Paul after college on their 50-states tour; Bill and Karen who we we got to know in Puerto Vallarta; and a brief return to San Jose to see Kevin and Brooke's new baby Luna -- who luckily for us arrived early so we had a chance to see her before heading to Alaska. We had worried about our encounters with bears in Alaska Alaska has both Black and Brown (Grizzly) bears. But Paul set us straight, the most dangerous animal we were likely to encounter was the Honey Badger.. The honey badger will be stung by bees or bitten by a cobra and it doesn't even care. It eats all kinds of things and the other animals just eat the scraps.

Our trip north took us past mount Shasta, which I always thought of as the most picturesque mountain in California, as it stands alone in the middle of a great plain, and we got a great view of it as we drove through on I5. As we headed through Oregon we had breakfast with Jimmy David. While I was stationed in the army in Hawaii we were roommates, and I fondly remember our guitar jams and our trips to bar-hop in Waikiki. It was great to catch up with him, hear about his family and life in Salem.

Just an hour up the road from Salem was Portland, home of Cherie's favorite doughnut shop, 'Voodoo Doughnuts'. They feature doughnuts that look like voodoo dolls, or are made of all sorts of crazy ingredients, like smashed snickers bars, or have names like 'the dirty old man'.

The next day we arrived in Everett, Washington for a tour of the Boeing factory. At this location they build 747's, the largest aircraft Boeing builds, and 787's, their newest aircraft which is the first commercial airliner to be built mostly of composite materials (fiberglass, carbon fiber, etc..). The tour takes you into the largest building on earth as measured by interior volume. It is a giant space, needed to be able to house assembly lines for 747's and 787's. The entirety of Disney Land can fit inside this structure. The building has its own weather patterns, and it took their engineers a while to figure out how to keep rainclouds from forming and producing showers inside the space. The 747 is assembled in this building, much in the same manner as a traditional assembly line for cars, just super- sized. However the 787 assembly line is a whole other animal. For the 787 Boeing is trying out 'just in time' supply chain management and outsources most of the assembly to sub-contractors. A converted 747 ferries 787 components from subcontractors across the world, and then the whole plane snaps together from about seven pieces. Two fuselage sections, two wings, an empennage, and two engines.

When we did our trip planning we wrote down the long list of places we wanted to see, and then made a route between them.. The route had us driving over half the days of our vacation! So I took a different tack, I decided we'd just put down the ten places we wanted to see the most, and then plan our trip around those so we'd spend more time seeing and less time traveling.. I looked over Cherie's list and asked Where's Polar Bear and Orca? I've never heard of those places . They were of course not places but animals on Cherie's 'Top-10' list.. Cherie did the research and the best place to see orcas was Friday Harbor in the San Juan islands. She called ahead and talked to a tour company Yeah, we see orcas 90% of the time! so we set off for our orca tour.. As we left the dock Cherie smelled a rat when the tour guide mentioned the humpback whale that we'd likely see that day.. Well perhaps we'll see the humpback and then the orcas, we hoped. After the humpback we motored a bit and saw a river otter swimming alongside a cliff. The guide was enthusiastic Wow we never see a river otter out here! This is amazing! On the trip back in I pointed out some mountain sheep on the side of one of the islands Look Cherie, you like mountain sheep! , she dryly replied I wanna see mountain sheep on the mountain sheep tour, not on my orca tour. As we were approaching the harbor the guide finally confessed that the resident orca pod had not yet returned for the summer, and that they only sporadically see roaming orcas until the pods return.. We'd have to wait till we were in Alaska to see the orcas.

Mount Shasta Cherie enjoys a VooDoo doughnut The 'dreamlifter' shuttles 787 components all over the world A humpback whale (not an orca) Bighorn sheep (not orcas)

"Managing your information diet"

2010-08-31T00:00:00-07:00

How many emails do you have waiting for you each morning? 10? 50? 500?

Managing your information diet is going to give you amazing gains in getting productivity back, AND improve the response time to the really important messages you get.

I believe too much information is worse then not enough information. If your pager didn't go off, you suffer an outage and you get an irate phone call from your boss. If your pager is going off every 5 minutes because your monitoring sucks you are going to be extremely frazzled AND you are going to miss that one alert that REALLY means your site is down and you STILL get that irate call.

So what can you do to manage the emails you are getting?

First thing, EVERY monitoring alert should be followed up by an action. Either you are fixing a broken server, or you are tuning the false alert. You have to be vicious about chasing down and fixing false alerts. Don't let yourself become numb to them.

Next, SPLIT your HIGH traffic messages from your LOW traffic messages. While you may want to follow every Subversion check-in in engineering or know every time your ISP replaces a router line-card in Azerbaijani, be sure these aren't also going to your group's discussion email list. It may be nice to be able to refer back to them, but they should not be polluting your INBOX.

Get adept at the mail filtering tools you have at your disposal I'm partial to procmail, just because it is what I'd grown up with. Do your best to only have ACTIONABLE email land in your INBOX, everything else should get filtered into a folders that you can reference at your leisure.

Finally, do something SMART with your system mail. If you have hundreds of nodes and they all send you a note every time they rotate logs or burp on a cron job that is bad.

I built a NNTP server for all system mail. All of that mail gets thrown into an NNTP news-group. So I don't have to look at it, but when I need to investigate a problem we can always refer back to several days of history.

Managing your communications, particularly your email, is one of the keys to going from being a good sysadmin to a great one. A little effort on the front end will reap great gains. The better you manage your incoming email, the better your piece of mind, AND the faster you'll be able to respond when a serious incident occurs.

"Lessons Learned Building OnLive"

2010-08-27T00:00:00-07:00

I just presented at the Large Scale Production Engineering group at our August 26th meeting, giving a talk on some of the things I learned while building out OnLive. I've included the text of the talk below as well as the slide-deck. The area that generated the most interest was our use of Kanban to manage workflow in our group. I was inspired to try out kanban based on this blog entry on Jordan Sissel's blog.. Jordan's blog is one of my favorites, you could spend hours going through his entries on operations and engineering.

I'll be adding blog entries on things I didn't cover in the presentation in the interest of time. But I will be blogging about soon..

The importance of a low-information diet, and not being overwhelmed by your email
Unicast for server software releases is dead at scale.

Onlive lessons learned

View more presentations from gregretkowski.

Introduction

I am looking forward to sharing with you several things I learned over the last two years building the OnLive game service. My name is Greg Retkowski, and I'm team lead for Operations Engineering at OnLive. I'm going to share with you what I learned about when to automate, why you can solve scaling problems by breaking it into chunks, and how to balance between interrupt- driven work and long-term project work. But first a little about us..

About OnLive

OnLive is a cloud gaming service that we launched in June it provides instant- satisfaction gaming. You click 'buy' on a title, and you are playing that title within seconds no media to bring home from the store, and nothing to install on your machine. You can play the latest games on your underpowered and old PC's or Macs.

We do this by running everything in the datacenter When you play a game, that game is running on our server you send up controller inputs, we send down video.

Through a bunch of smart video and networking guys, and a few puppycorns, we provide an experience that's the same as playing on your game console at home.

About Me

I've been working in Operations since 95', I started as the first employee of an Internet Service Provider in Ft Lauderdale, Florida. Two years later I moved here to the Bay Area and I've been doing operations for internet startups ever since.

I joined onlive in 2008 at the time we had around 100 nodes. Since then we've grown by more than a couple of magnitudes... and we're continuing to grow today.

I currently lead a team called Operations Engineering, and we're responsible for...

Automating the deployment and configuration
Creating the tools used to manage the service
Designing the operational fabric for the service
And we're the FINAL escalation path for the most difficult
sysadmin issues.

Basically, were the greybeard sysadmins in the organization..

There were a lot of growing pains going from almost nothing to the size we needed to build to support the service. We had challenges, we had failures, and even a dumpster
fire; BUT we learned a lot, and ultimately succeeded.

I'm going to cover three areas where I learned a bit about managing complexity in my last two years.

I'm sharing what I learned hoping that I learned the hard way so that you don't have to.

When to automate.

You probably have worked in shops where every server is hand-built, some poor sysadmin sitting in a loud cold datacenter, swapping redhat CD's and clicking NEXT prompts.. And then repeating it for a row of servers that has to be installed...

That's probably not the best approach if you need to scale.. But, then, is the answer to spend a ton of time in automating everything?

Our machine database is a rails app and I needed to decide if it would run under mongrel or mod_passenger. We ultimately decided on mod_passenger. I could have spent weeks building our puppet rules around mongrel, automating the install, rotating the logfiles,
excetra.. And those weeks would have been wasted as we chose something different... So what is the right balance?

You want anything going into production to be automated and repeatable. However you don't know what's going into production the first time you build it. The first time you are building this you are building it for engineering to evaluate.. You could be building
couple different NOSQL boxes for engineering, each to be evaluated, and one, or none, will be used in the final product. So when do you automate something?

The FIRST time you build something JUST LEARN about what you are building. At this stage you may be doing throw-away work, you are learning what the software can do. And, more likely than not, you are going to implement it wrong out of inexperience. At this stage, there is no value in automating something you are as likely as not to throw
away.

The SECOND time, DOCUMENT IT. Now you are building a second one you are probably going to build a few more. You've learned your lessons on how to do it better the second time. This is the time to DOCUMENT how you did it and make a CHECKLIST. That'll let you hand off the next build to another team- member, and validate the procedure.

The THIRD time, AUTOMATE IT. By now you know the software inside-out, and you know you will continue using it. Now is the best time to codify it using puppet, chef, or your automation du jour.

The FOURTH time - bonus points for pulling this off is to TEST IT. There are many ways to approach testing your automation. You could consider your monitoring service checks as your 'tests', or you could use a VM test harness to test your automation code.

We're using a VM test harness to test all of service installations, but that's a story for another presentation.

Break things into smaller chunks

The next big lesson we learned as we were building was to spit our environment into chunks whenever possible. That takes some explanation, so I'll relate what we experienced building out our service.

As a side effect of our architecture, we've split things into chunks we call slices. This side effect turned out to be very important on our efforts to scale later.

As we grew our service we ran into performance issues in different parts of our mamangement software. One example - as we grew, generating our DNS data got longer and longer, and it eventually took an hour to generate.

But - Because we cut our environment into chunks, we were able to re-write our DNS generator to generate slice-by-slice. It was able to cache slices that didn't change, and that cut the generation down to a couple of minutes.

It is much easier to scale your service out when you've broken it into smaller and well understood chunks. Where we faced the issue of understanding how to generate DNS for a thousand hosts, we didn't know how that would behave at two thousand hosts.

But we did understand how our system behaved at the smaller slice size, and going from ten slices, say, to twenty slices, was a much easier to manage as we had good knowledge of how everything behaved within the slice.

So, whenever you can, break your environment into manageable chunks, and put as much of your scaling dependencies down into those chunks and when you have to do things at a whole, cache whenever you can.

Balancing interrupt & Project Work

Our next big lesson was about how to balance interrupt and project work. It took a while to try and find a good balance between the two.

How many of you think that if they reduced interruptions they'd get a lot more work done? If you are like most in operations you know it is a huge problem.

I remember one case A release deadline was approaching, and my group was on the hook for delivering a new web proxy system and some new admin tools to manage the site. At the same time our interrupt-driven requests from engineering went up because they had the same deadline and needed our support to get their projects done. It was very challenging to balance the two, to be able to unblock engineering, while delivering our projects by the deadline. We worked long hours and still some things got dropped because we didn't have a good way to balance it out.

Our teams are pulled in many directions. A pure engineering team specs something out and then has several weeks to deliver. A 'helpdesk' team goes from interrupt to interrupt to solve problems in a short time window, but no long-term deliverables. In operations, you have to be both engineering and helpdesk. So how did we balance the two?

I looked at development methods like Agile that would help my team deliver complex projects BUT weeks between identification of a problem and a resolution isn't acceptable.

I knew the solution of going from interrupt to interrupt wasn't the right way to go. That may make people happy in the short term, but then we'd deliver our long-term projects late and poorly implemented, because we didn't give them the attention they deserved.

I dug a bit deeper and found a solution that was ideally suited for my group.

We borrowed from the lean manufacturing world to solve this problem. We use a system called KANBAN to manage our tasks. KANBAN is easy to implement you only need a whiteboard and some post-it notes. The central premis of KANBAN is that projects go through different states INCOMING WORKING INTEGRATION PUSH TO PRODUCTION and that only a limited number of projects are allowed to be in each state at any time.

We have an INCOMING queue where all projects arrive each team member can have two projects going at one time, an as they finish one they pull the next-most- important project into their WORKING queue.

It is bad kanban form to either overload the number of projects in a given state or to pull a project and put it in a previous state. We only do that when something crucial comes up.

Using kanban makes sure everyone can work on their projects with as little interruption as possible and makes sure the next project that gets worked on is always the next most important project.

Point Of Contact Rotation

I believe interruptions are a HUGE productivity killer in operations groups, so I've done what I can to minimize them.

It is very hard for you to focus on projects when you are constantly interrupted. After an interruption you remain unproductive for 20 minutes before you can be fully engaged in what you were doing. In ops this means you could go all day and get nothing done
due to interruptions.

So there's another strategy we use to minimize interruptions We rotate everyone on the team through an Point-of-Contact shift each week. Any interrupt-type work goes through to PoC first. You know your interruptions will be minimized most times, and that the week
that you are PoC you know most of your time will be interrupt-driven.

It also gives my newer team members an opportunity to learn about all the parts of our service. They have to field all questions for our group regardless of their comfort level.

During this week the PoC guy is responsible for the bug queue and to triage any new issues that arrive. This helps the rest of the team as everyone knows if they aren't on-call they'll be able to work on their current project uninterrupted.

Conclusion

I covered three areas where I learned important lessons, and I hope you find these lessons useful in your environments.

I covered...

When to automate Automation is necessary, but is more
effective when you do it at the right time.
If you break your environment into small chunks, and scale at
those chunks, you'll have less scaling headaches later.
And finally, if you can find a bring order from the chaos of
interruptions, while still delivering on your projects, you'll be
happier and the engineers you support will be happier too.

Thanks to Yahoo for organizing and hosting this meetup thanks to all of you for your time, and if you have any questions I'm happy to answer them now.

"A week of work on Zortit"

2008-10-01T00:00:00-07:00

It has been a hectic week working on our webapp. I planned to put up regular updates but have been so busy coding that I haven't been able to keep up with it. Here's a big update of what we've had going on for the last week or so. We're coming up on the home stretch with the contest deadline looming on friday..

Marketing
Cherie has been hard at work. She's been working on creating the product description, writing a rough draft of a marketing plan, doing competitive research, putting together a board of advisors, and coordinating the efforts of our graphics team.

Branding
Our graphics artist, Chris Barela, got us a great logo last week and we've gotten some great website mockups. HTML/CSS guru Jean Leitner has been hard at work converting the mockups into code. Here's the new logo:

Infrastructure

Our application runs in the AWS cloud. We're using EC2 instances. We're using S3 to cache some web API results. There are other AWS services we use which I won't dive into here.

Right now our app is running as a single EC2 instance, however I've partitioned the components out on this instance so that they can be spread across machines. On the front end we're using HAProxy, with apache/mod_passenger (aka mod_rails) running rails instances, with MySQL as the database. We're using memcache for a performance speedup, as well as S3 as a cache. I'm doing deployments via Capistrano which works pretty well.

'I have a dream' of having instances come up and self configure. Sometime in the future (probably when things are burning down) I'll set up iclassify and puppet, and perhaps even configure user auth via LDAP. And then systems will spin up, register with iclassify and I'll be able to provision them mostly automatically. I'd hoped to use pool party, but it's in re-write right now -- perhaps when it is finished.

I also got nagios set up to monitor from an existing machine, and during the process found one of my nameservers was broken - funny things you find out when you start monitoring things!

Collaboration
We've been using trac to collaborate - and we've managed to proxy tickets from email into trac's bug tracking. So testers can click a mailto link when something breaks on the site. Very neat! Trac also has a subversion browser I've used on occasion and I've been posting links to system management pages and whatnot there.

So that's where we're at. More news as it happens!

"Some random links"

2008-09-24T00:00:00-07:00

I've been cleaning out my email today, and finding several gems among the cruft..

A friend sent me a link to a pretty rocking collaborative whiteboarding application: Dabbleboard.

I found this great slide-deck from a presentation at the Velocity conference done by Adam Jacob. It's a great introduction to the latest tools that you can leverage for 'deploying to the cloud'. Check it out:
Building an Automated Infrastructure (Powerpoint Slides)

Right now I'm busy hacking on an AWS Startup Challenge entry. Our entry is using the theme 'redefining search'. The product is called Zortit, and will be leveraging the AWS cloud services and be built around Ruby on Rails. Keep tuned in for more updates!

"The bailout - why you should say no."

2008-09-21T00:00:00-07:00

In case you've been in a cave for the last year, lots of large corporate banks are on shaky ground after years of making exremely risky loans/investments and reaping huge profits in the process. Now the people who helped create this huge mess tell us if we just give them one trillion dollars (that's $10,000 per US household) they'll make it all go away, oh and they won't be answering to congress or the courts on how they decide to spend it. To read more about why you should be very angry about this bailout and calling your senators to demand they block and vote against the bill see these links..

http://www.nakedcapitalism.com/2008/09/why-you-should-hate-treasury- bailout.html

http://globaleconomicanalysis.blogspot.com/2008/09/bush-administration-seeks- dictatorial.html

This is the letter I sent to both of my senators.

I am writing you today about the legislation facing you this week to bail out the banking industry. This rushed legislation provides no accountability, and leaves the American people with an astronomically large debt. It is signing over one trillion dollars of taxpayer money to the people most directly responsible for causing this crisis in the first place!

There is existing market for these securities at a low - but fair - market value. What this bailout proposes is buying these securities at above market values, leaving the taxpayer on the hook for the difference.

To be honest this 'you must act now' (an the implicit statement that you don't have time to think about other options) is something I expect from late night infomercials, not the people entrusted to secure our financial institutions.

I cannot, in good conscience, continue to vote for any member of congress who supports this piece of legislation.

Warmest Regards,

"Editing Video from SD Video Cameras"

2008-09-14T00:00:00-07:00

A month or so ago I got a Panasonic SDR-SW20 Video camera that records onto high-density SD. It is a handy small camera that's waterproof (and dustproof!), and fits into a pocket. It records in a format that's difficult to use in most editing programs (.MOV) so after shooting a bunch of video of Burning man I had to write my own converter with FFMPEG to get the video into Windows Movie Maker. If you have the same problem get a copy of ruby and ffmpeg for your windows box and use my script to import it.

NOTE: I just updated the script with /working/ encoder options. The old options would cause output to stop halfway through the video!

http://pastie.org/272288

"Sizing your infrastructure before launch"

2008-03-12T00:00:00-07:00

So you got a webapp - How do you decide how many servers to deploy??? Even if you are still in development and don't have a single outside user you can make an informed decision on how big to build and what your future network infrastructure will look like.

By gathering some data and doing a little load testing you can launch a new application confident in the fact that you know how many users your application will support.

I will outline the process you can use to size your infrastructure. I'll be discussing it in the context of a web-based application but these methods can be applied to other types of applications. At my last client, Avvenu, half the network communication was not HTTP based and I used these methods to scale it regardless.

At the end of this process you'll have a spreadsheet where you'll be able to plug in arbitrary numbers and get out the scaling information you need. If bizdev asks "what happens if we close this deal and double our user base?" or if engineering finds a way to increase server performance by 100% you'll be able to quickly answer what the impact on your network would be.

Understanding your usage

The first step in building our scaling model is to understand how your users use the system. There are a big series of questions that you'll need to answer to get an idea of what that usage looks like.

First you'll need to know how many active users to expect in the future. This data often comes from your marketing department.

The data is usually presented something like - in one month we'll have X active users, in two months we'll have Y, in three months we'll have Z. You'll need all these for your scaling spreadsheet.

Next you'll need to find out how the typical user either uses the site (for existing sites) or is expected to use the site (for new sites). You'll want this data in a given time period, such as per week. Some examples of what you'll want to know are:

How many times a week does he visit?
When he visits what does he do?
Downloads a large file?
Looks at pages that require a large amount of processing
How many times and which ones?
Looks at images that are dynamically created?
Looks at static pages?
Uploads Data?

How much data do you have to maintain per users? This includes files, database rows, or in some applications constant open connections. This will also have to be accounted for in your scaling model.

For an existing application you'll be able to mine your access logs. Always keep and archive these logs when at all possible. They come in handy to mine for useage pattern data. Throw together some scripts to extract the answers from your access logs.

For new sites put together a detailed but not overly technical questionnaire for your product manager. The answers from the questionnaire can be used to model typical visitor usage patterns.

One final note on usage patterns. You'll find that you'll have some users that look at a few pages every couple of months, and then some users who integrate your site into their daily routine. You'll need to find the /average/ across all your active users.
&amp;nbsp;
Distilling the estimated traffic

Now you have how many users you have, vs. the activity of each user. You can now determine how many requests your service will have to handle. You can figure this out just by multiplying the number of users against the number of operations and then divide that by the number of seconds in your time period (i.e. a week) to find the average number of operations you'll have to perform per second.

Important to note, when sizing your bandwidth that file sizes are measured in BYTES and bandwidth in BITS. multiply all file sizes by 8 to find the number of bits they would be when crossing Ethernet.

Load Testing

Once you've determined what your average user will do you'll need to automate that behavior for load testing. Typically you'll set up a load testing cluster - or just test against your pre-production or development environment on off hours. You'll need to ensure your load-generating machines that run your load testing scripts do not become your bottleneck. In this phase it is very useful to be running server monitoring and graphing software like NAGIOS and CACTI. Make sure your server graphing captures CPU, Disk, Memory, Network, and process utilization so that you can identify which machines bottleneck and what parts of the machines have to be scaled. Sometimes you'll think an application should bottleneck on CPU and find it bottlenecks on Memory. This helps you make informed purchasing decisions when you buy new machines for your production environment.

You can set up scripts and use tools such as AB (apache benchmark) to throw traffic at your servers and determine the number of operations per second your servers can handle. You'll have to try to isolate each class of machine (i.e. DB or HTTP, etc) and determine it's maximum load. With unlimited resources you could load test a single webserver to determine it's limits, then throw 100 load-testers against 100 web-servers to find your DB's load limits. But for most of us this is impractical. So you may have to be clever and try and profile the database traffic generated by the webserver load testing and then create a script to drive simulated load at your DB server directly.

It is important in this step to discover any horizontal scaling issues. If you find adding new servers does NOT increase your capacity as you expect then you'll need to work with your software engineering team and fix the scaling problems or warn management that their is a likely hard limit of X number of users the system will support.

Peak vs. Average usage

You will need to determine the peak usage hour(s) of your service and how these relate to your average usage.

I have found that your peak usage will typically be double your average usage. If you have no other data then go ahead and size for that.

If you are sizing an existing application you already know your ratio of peak vs. average by looking at your log data.
&amp;nbsp;&amp;nbsp;&amp;nbsp;
Building the Spreadsheet

TOTAL          (users * usage / time-period-to-seconds ) * peak/avg


REQUIRED  =  --------------------------------------------


SERVERS       benchmarked-requests-per-second-per-server

Do this for each class of server, web servers, app servers, DB servers, etc. Then make a column for each month of growth. Make your formula round-up the number of servers. you can't deploy 2.3333333 servers can you?

Often I'll break this down into the number of active users each server can support. I can then divide the number of projected users and have the number of required servers.

USERS       benchmarked-requests-per-second-per-server


PER       = ---------------------------------------


SERVER     (per-user-usage / time-period-in-seconds ) * peak/avg





TOTAL                USERS


REQUIRED = ---------------------


SERVERS      USERS-PER-SERVER

Your total servers numbers can drive other parts of the spreadsheet as well. Every so many servers you'll need a new Ethernet switch, another rack at the colo, and perhaps increased headcount (try and reduce this by automating as much as possible!)

Make sure your spreadsheet also accouts for the amount of static data you have to maintain per user. For example how many file servers will you need for the files your users upload? How many users will the disks on your DB server support?

Your model should also determine the maximum network traffic at peak times so that you'll understand when you'll need to order more bandwidth from your connectivity provider or will need bigger routers and load balancers.

In Conclusion

Using this process has allowed me to help size networks for many internet startups and kept my network operations groups from being caught with their pants down. Determining your scalability and using this data to anticipate required infrastructure growth will help you and the rest of your organization have confidence going forward with a growing userbase.

"Maintaining documentation -- It's in the wiki!"

2008-02-19T00:00:00-08:00

One of the important things of maintaining a big network environment - with a small staff - is to keep up to date documentation on configurations, customizations, and instructions for frequently executed tasks. Commonly when I walk into a new company the documentation is terrible? Why? Because there is either no thought to maintaining documentation or the documentation system/procedure in place is too time consuming to use.

If a documentation system us hard to use it wont be used at all. It should take less effort to update a piece of documentation than to send an email. Locating a document should be as easy and should support freeform text searching. Thats why the best documentation setup I've worked with is a wiki. It's easy to create, locate, and change documentation which encourages people to actually document things! You will have current verbose documentation when you need it.

If you do use a wiki to maintain your documentation produce an offline copy of periodically and burn it on cd. Put this CD along with one copy of every vendor supplied CD into a CD wallet and keep it at the datacenter. it will prove invaluable when you have outages.

Heres the wiki engine I've used - and liked - in the past. It runs on top of your vanilla LAMP stack.

tikiwiki.org -- TikiWiki CMS/Groupware

"Small Business: How not to behave on the internet"

2008-02-17T00:00:00-08:00

This is an example of how not to behave if you are a small business on the internet. A friend of mine simply posted a question on a forum, the entirety of his question was: I'm curious if anybody knows anything about Lucas Environmental Stormwater Services, Inc.? This simple question has led to the owner threatening legal action in email and via rambling voice- mails. It is never a good idea to threaten someone unless they are blatantly in the wrong and doing something clearly illegal. Otherwise you just rile people up and turn what should have been nothing into a huge negative- publicity exercise for your company. For more information see: mhalligan: Greatest voicemail transcript EVER.

Technorati Tags: lucas environmental services

"RoR: Testing with simple_captcha & HTTP-Auth"

2008-02-09T00:00:00-08:00

While developing a small Ruby on Rails application for The Pilot's Camping Directory website I ran into a few problems that weren't solved by a simple google search - so I'm documenting them here for future posterity and googling. I had problems with testing when using some security features to keep out riff-raff. It was not obvious how to handle simple_captcha or simple_http_auth while doing testing so I scratched around the net and pieced together a solution for each of the problems. These work with Rails 1.2. With Rails 2.0 YMMV - but then 2.0 breaks every rails tutorial ever written so I don't feel bad if this blows up in 2.0.

Using Mocks for testing with simple_captcha

Tests will fail when trying to save something protected by a captcha - obviously - as stoping automated lever-pulling is exactly what a captcha is designed to do. In my application I use capcha at the model level, so I simply override the save_with_captcha method with a simple save.

Here's what my mocks/test/recipient.rb looks like:

`# Can't fake captcha for testing - so we mock it out.
require_dependency 'models/recipient'
class Recipient < ActiveRecord::Base
def self.save_with_captcha
self.save
end
end

`
**Functional Testing HTTP-Auth

**To test HTTP Authorization / Authentication you must set up your request environment to pass the http authorization into the application. This is known to work with the simple_http_auth plugin, the plugin that I used for my application. Specify this in the setup section of your functional test.
``
def setup
@controller = SupersecretController.new
@request = ActionController::TestRequest.new
@request.env['HTTP_AUTHORIZATION'] = "Basic " + Base64.encode64(ADMIN_USER +':' + ADMIN_PASSWORD )
end

`
**Integration Testing HTTP-Auth

**Integration testing simulates making requests directly to the webserver. To work with http authorization here you must pass in the appropriate authentication headers when making each get/post request. An example is below:
@htauth = "Basic " + Base64.encode64(ADMIN_USER+':' + ADMIN_PASSWORD ) get("/supersecret/index", nil , {:authorization => @htauth})

"Sharpening the saw, html and graphics."

2008-01-16T00:00:00-08:00

In my off-season (winter) I am usually traveling internationally - mostly places that are sunnier and warmer than the San Francisco bay area. It's often the perfect time for me to sharpen my various skills , being unconstrained by the usual grand infrastructure projects I do in the summer.

It's often these times that I bring back up my html/coding/graphics skills. Wifi Bandwidth here in Puerto Vallarta has gotten much more ubiquitous and reliable and so I've got connectivity almost as good as back in SF. I've been diving back into apps like
Gimp, Aptana, & Inkscape.

I also enjoy catching up on the avant guard of web artistry and seeing what people are creating with html and css. I appreciate simplistic designs and so I really enjoyed the sites on display at the link below:

25 Beautiful, Minimalistic Website Designs - Part 2 | Vandelay Website Design

Baja Haha 2007

2007-12-10T00:00:00-08:00

Snowgoose sailing with spinnaker on the 2007 Baja Haha

Much like the sailor tails of yore, I found myself shanghied on a sailboat in a foreign land. This is my story.

More Photos

It all started at the Baja Haha party in San Diego, which takes place the day before the start of the Baja Haha sailing rally to Cabo San Lucas. I had done the Haha twice already, once on my own boat 'Scirocco' in 2001, where I met my girlfriend Cherie and several other great friends like Rennie and Anne. I had done it a second time in 2003 aboard Rennie and Anne's 'Cassiopeia'. Because we all had several friends who were doing the Haha this year the four of us decided to drop into the party and say our goodbyes to those leaving for Mexico. Rennie, a retired United captain, ran into a friend of his, another united captain named Mike. All but one of Mike's crew canceled at the last minute and so he was going to do the Haha very short handed. There was free beer there, and after several of them I decided to volenteer to crew with Mike so he'd have enough crew to have a comfortable trip south.

It's tough with two to do multi-day trips, especially when 'racing'. It can be exhausting because for 12 hours you're on watch, another 8 hours you are up anyway cause it's the middle of the day, and then even in the middle of the night you are often woken up to deal with things like spinnaker wraps or vessels crossing very close to your path. So with just two aboard you get almost no rest. I think three is a pretty ideal number, assuming a reliable autopilot to aleviate hand-steering. Everyone gets a reasonable amount of rest only having to have 8 hours of watch a day, and off-watch crisisis are easier to deal with when there's two people off watch to choose from. My first Haha we had six aboard, and we had a great time. It works well when sailing and a third of the crew is sailing the boat, a third sleeping, and a third doing whatever else. But it can be cramped if you have conflicting personalities, and you are all sharing the space of a small boat during idle times like when anchored in port.

So, in very short order (18 hours) I dropped everything else I had planned for the next month, packed my bags, and prepared for a new adventure.

Leg 1, San Diego to Turtle Bay, 360 Nautical Miles.

Cherie delivered me to the dingy dock where Mike picked me up and ferried me out to Snow Goose, anchored off Coronado. I was impressed with the boat, the preparations were spectacular. The key systems such as the autopilot have redundant backups, everything is in good working order, and Mike was meticulous with the preparations for the trip. Snow Goose is a 50 foot mapleleaf, although I've come to call it a 50 foot unicorn; because much like unicorns, sailboats where everything works are equally rare. Dave, our other crew member was already aboard and we rapidly got everything ship-shape to get out of harbor. Raising anchor we found our first treasure of the trip, an abandoned dingy anchor, which wrapped itself around our anchor chain.

A bit about the crew. Mike is the Captain of Snow Goose, and is a retired United Airlines pilot. He's done some offshore racing such at the Carribean 1500. He's heading down to the South Pacific after spending a little time in Mexico. Mike is detail oriented and it shows in how well everything has been done on the boat. That's one of the wonderful things about sailing with commercial pilots. Everything it thought-out and done properly. There's none of the half-measures, hacks, and otherwise unseamanlike thown-together systems on this boat that are found on many other sailboats. It's a big ease on one's mind to not have to constantly worry about what critical system is going to break next. Dave, the other crew member, is a pilot for United Airlines. He is based out of Baltimore where he has a power boat he cruises around the Chesapeake bay. Dave has limited sailing experience, but is enthusiastic. It's been great spending time with both of them.

Although just prior to the race start there was hardly any wind and everyone was preparing for a motoring start, just five minutes before the gun the wind came up and provided enough strength for a sailing start. We ended up very well positioned on the line and got our chute set and crossed the line well ahead of most of the fleet. We were in very good company crossing with the J-world boat and a Santa Cruz 52. The wind was lighter than we liked, around 12 knots, but it kept our sails full enough that we were able to keep pace with the J-boat all day long. The Santa Cruz 52 daftly pulled ahead of us, although we can't fault ourselves as we've got a rail-mounted-BBQ handycap. We passed the islands off Tijuana, and as Mike and I were having a discussion about how he'd never had a cruising cat pass him in an ocean race, the familiar shape of Profligate appeared behind us trucking along fast. Most crusing cats are loaded down which kills their performance, but in Profligate's case the boat is kept very light for it's size and so it moves along snappily compared to most other vessels.

We had a decent sunset, away from the clouds that dominated our departure from San Diego, and as the dark aproached we had our dinner of hot dogs and settled into watches of one-person each. On dave's watch we had a curious fishing trawler come alongside to observe these crazy sailors with a spinnaker up at night, and just as Dave called me up on deck they cut in front of us causing some evasive manuvering and banging of spinnaker hardware against rigging - luckily no damage was done and we were able to get back on track. That's one way to wake yourself up at the beginning of a watch. We had a 900-ft cargo ship also on an intercept course - and were able to manuver to avoid it by half a mile. Mike's chart-plotter can plot AIS information, which is basically transponder data from each of the ships near our position. It gives the vessel name, destination, course, speed, and position, as well as additional data like the type of ship and if they are motoring or anchored. The chart plotter can then take our GPS position and compare it to the AIS data to derive each ship's nearest approach as we pass each other.

On my second night watch the wind freshened up to 16ish aparent (boat speed 8-9, so 25kts true) and the spinnaker was at it's limit starting to overpower the autopilot. So at 5AM, in the dark we set to dousing the spinnaker in heavy wind conditions, and my first time dousing the chute on this boat. Luckily the chute has a sock and Mike has figured that we can just blow the tack and it'll be trivial to pull down the sock. Well that works flawlessly. We continued on for another hour with just the main, then at daybreak we set the jib winged out on the spinnaker pole.

We weren't trying to catch any fish, although one may have hit the water- powered generator, as we discovered it had broken over the night. was broken. According to the morning check-in we were doing superbly well for the first day of sailing.

First Leg, Day two.

The second day held conditions much like the first. The wind continued to blow in the low 20kt range (true wind) and we were able to sail the rumbline at hull-speed with the main-sail out on one side of the boat, and the jib poled out the other side. It's kinda funny, the day in the middle of the passage, especially that conditions stayed much the same and no sail changes had to take place. Thinking back, there's nothing I can really note about the day. It's as if my memory of it is all but a blank - the day was consumed with the same activities as the 100 other day's I've been on passages doing watches. Check for other boats/ships, check that nothing is breaking, check that the sails are set for the weather conditions we have, repeat this for two hours every six or so.

The sea conditions did build a bit as the strong wind drove up the size of the waves. Imagine lying in bed and having someone shove you at full cow-tipping speed once every ten seconds and you've got some idea of what it is to try to settle into sleep with waves going as you are sailing downwind. With time either your body adjusts or the sleep deprevation knocks you out -- probably a combination of both. So the latter half of the second night I was able to get decent sleep at the times when I was off-watch and felt more coherent by the morning of day three than most of day two.

When I came up for one of my night watches I noticed something wrong with the jib. It was still poled out and the top looked fine, but the bottom looked floppy. Looking a little harder I noticed the tack was just dangling, not connected to the base of the jib furler. During the night the pin securing the bottom of the jib dissapeared. I went forward and rigged up the jib with a bit of line. Later in the day Mike discovered bolts had backed almost completely out of two lower shrouds (important parts for keeping the mast from crashing down) and we were able to get them tightened back up. The fact that the maintenance/repair list for the passage is pretty short is a testiment to Mike's preparations before this trip.

We crossed the finish line at 07:08 the third morning out from San Diego; We made very good time. We arrived in Turtle bay proper at around noon. Out of about 170 boats I'd say we were in the first 10 to arrive to the anchorage - many boats had tougher times with the 20 knot winds. We cleaned up the boat, launched the dingy and just before sunset went into town.

Turtle Bay, Baja California

Turtle bay is a sleepy little town on a protected bay frequented only by transient yachts and fishing boats. The town used to thrive around a fish cannery, but modern fishing methods rendered the cannery obsolete and so what's left of the waterfront is the rusted out skeleton of the cannery and a run-down pier where cruisers tie their dingies to the rusty steps leading to the water. Youngsters wait for the arriving sailors and offer to watch the dingys for a few pesos or some candy.. Going into town one passes a cinderblock shack that serves as a tienda, resturant, and bar. It's the first place to get a wonderfully cold pacifico or sol beer with a wedge of lime, sit back in one of the plastic chairs with your toes in the sand and watch the flags flutter on the sailboats in the anchorage, securely embraced by the rough red arms of the bay entrance.

Further into town, on the single dirt 'main street' are a few tiendas, about the size of an circle-K which sell vegitables, eggs, and soap trucked in from California. Most everything on the Baja Peninsula gets shipped in from California with a corresponding mark-up. Further up is the largest resturant in town, the Vera Cruz. Due to good fortunes from the two days the HaHa fleet is in town they've expanded the facility.. The main dining room hosts about eight tables, and a new disco/bar is in a back room with a dirt floor and Mexican oompa music played way too loud. There's a cement patio in front of the resturant with another dozen tables. All the furnature is the standard mexican fare. Plastic lawn furnature, the kind that's the cheapest stuff you can find in walmart. All of it is oxidized and chalky from exposure to the blaring mexican sun.

Vera Cruz was practially deserted, with just two other crews in the place. Because of the drastically low traffic we actually got our beers and meals in a very timely manner. Despite the Haha arriving year after year the owners of Vera Cruz have not discovered that they should own more than 8 plates to serve the large crouds. The anchorage continued to fill throughout the day but many of the crews just arriving were too beat to head into town. Turtle bay was much colder than usual. Instead of the t-shirts and shorts I was accustomed to we were equipped with sweaters and forced to sit inside for warmth. I enjoyed the ambience of Turtle Bay, but I think Mike and Dave were taken aback by the rustic decor compared to many landfalls in the Carribean. After exploring town I and the rest of the crew slept like logs, catching up on the sleep missed over the three night passage.

First full day in turtle bay

The morning we spent working off the ToDo list of problems we discovered on the passage. It all went quickly, the largest parts were checking out all the halyards for chafe and double-checking the spinnaker pack. I offered to repair a computer problem on one of the fleet's boats but got no answer when hailing them at the appointed time. I guess it wasn't that much of a problem after all.

The morning VHF nets on the HaHa are interesting, although a bit long, as so many boaters who've rarely ever used their VHF except to call sea-tow now were getting the feel for using the VHF for social reasons. The "cruiser's telephone" requires a certain degree of ediqite that is more learned by observation, practice, and repremands from the radio police. The progression from radio anarchy at the begining of the haha to smooth refined communication by the end is something to behold.

Mid-morning Dave and I trekked into town in search of beer. The mercato we found had a very limited selection, not what we wanted, and only had cold beer. Normally cold beer is fine, except that we had to hike around with it till it was warm before cooling it again on the boat guaranteeing it to become skunky. Such is life in a small mexican town, no selection like Safeway - we got a 12-pack to keep the boat stocked. I then hiked out to turtle bay airport, to scout it out for if I'd ever want to fly down to turtle bay to meet up with future HaHa fleets. I found the airfield as well maintained as anything in town, which is to say not maintained since the cannery went out of business. It is suitable for cessnas or cubs but not suitable to my Long-EZ which would either shear a landing gear in a pothole or have a rock take out a prop-blade.

After returning from the airport I bummed a dingy ride from the crew of Tranquilo. I was dissapointed to learn that the 'listerine guy' was not in attendence this year. In 2003 he made this stuff by mixing a half dozen liquors that tasted like listerine but would knock you on your butt like nothing else could. When I arrived Mike cooked up an amazing steak dinner, and then we headed back to shore for the Vera Cruz party. We showed up at 8PM, and suprisingly most of the place had already emptied out. There were just like 5 crews left. I caught up with Richard and Donna from profligate, talked to a few other folks, we had a few beers, and when it looked like the final crews were leaving I headed down to the dingy dock. I bummed a ride from Bill of Moonshadow and got to meet his friend/crewmemeber Mark, also from the Newport Beach area.

Second day at Turtle Bay - the beach party.

The next day was the turtle bay beach party. The crew of Snow Goose loaded up in the dingy with a backpack of beer and some raw chicken to grill up for the potluck and practiced our beach landing skills. We got a little fast, got ahead of a wave, and got a little sea-water in the dink, but got ashore without any real danger except to our pride.

The potluck was great, as I got to catch up with a lot of friends who I hadn't seen since the start of the race.

I ran into Al Holmes of Creme Brulee. I had met Al on the 2003 haha when he and his wife Diane and cat Damnit were taking their trawler down to cruise Mexico. Al is another United Airlines Captain who I met through Rennie. Al was happy to be back down in Mexico and more enthusiastic since it wasn't as hot and the heat is what gets to him most about being south of the border. I left Al, Dave, and Mike to chat about how unethical the airline executives are and went to find a place to cook the chicken we brought ashore.

This is when I met Chris and Vicky of Wingstar. They had brought in a BBQ grill and were cooking some fish they had caught on the first leg. I asked if I could use a corner of their BBQ for my chicken and struck up a conversation. They are cruising on their boat, with four childeren and only three adults. On their boat I think I'd rather be on watch than off; it's much easier to face the cruel sea than changing diapers.

After serving up the chicken at the potluck I ran into Christian and Mary of Capricorn Cat. Christian has done six of the seven last HaHas. I met Christian on the 2001 rally when my boat Scirocco ran out of fresh water and I was able to borrow some from Christian who was at the time sailing on a beautiful three-masted 70-foot boat named 'Millenium Falcon'. Christian was up at the top of the mesa overlooking the beach party taking the 'HaHa 07 group photo'.

Shortly thereafter I ran into the full crew of MoonTide. Bill and Mark introduced me to the rest of their crew, Tessa, Jennifer, and Angelina. True to reputation Bill had filled his boat up with pretty women.

Hallie, who I met through Cherie's colledge friend Anita, was sailing in the haha, and the beach party was the first time I ran into Her. I got to meet the rest of the crew from the boat she's sailing on named 'Blind Luck'. The blind captain had the good sense to fill his boat up with mostly women, having 8 aboard along with himself and only one other man. I and the crew decided to head from the Beach Party to 'Mexican Wrestling', which just happened to be happening the last night the fleet was in town. We stopped at 'Blind Luck' on the way and I got to meet the rest of the crew and check out this large cruising (piver style) trimaran.

At the wrestling match the favorite wrestler of the night was a guy who had a skeleton costume and would dance to 'Thriller' every time he got into the ring. The mexican kids ate the whole thing up, cheering for the 'good guys' and booing the 'bad guys'. In mexico it's not just the boys who like wrestling, the girls love it too = perhaps because of the intricate costumes.

Second Leg - Turtle Bay to Bahia Santa Maria - 240 Miles

We got out to the start line, and were playing a bit of catch-up getting into position because the commitee boat set up in a different spot then we expected. We rigged our chute on the wrong side for the wind and didn't have time to fix it, so we popped up the spinnaker and headed the opposite direction than everyone else we flopped over to the other direction after a couple of minutes and were outside the fleet. This actually seemed to be fortuitous for us, as we had lots of clean air and were further out where the wind was stronger. We worked our way forward of most of the fleet and as the hours passed got further and further away from the shore and the other boats.

On this leg we decided to give fishing a chance. Mike put a reel in on the side of the boat, we tried one lure for a few hours, then another. We heard a dozen other boats on the radio hooping it up about their catches yet we were having no luck. We asked what lures were working for folks and got lots of good suggestions. So we changed the lure on the pole and Mike set up a hand- reel on the other side of the boat. The changes worked out as just before sunset we landed an 8lb tuna which provided a nice course of ceviche. This was the first fish I'd ever bludgened and I may have been a bit zelous to ensure I killed the fish quickly - the back deck looked like a murder scene with blood from the fish.

Just a few hours before sunset there was a metalic pop noise on the boat. This is usually an indicator of something that'll need to be replaced - and in this case we had the block for the spinnaker halyard let go. The halyard is inside the mast and this is where the spinnaker was now flying from. So we were able to get it down without further incident and sailed through the sunset with the jib poled out opposite from the mainsail.

Leg two, day two - Turtle Bay to Bahia Santa Maria

By early morning the wind had lightened and rather than wallow aimlessly we started motoring. The slightest breeze came up in late morning and we set the spinnkaer and tried to slowly bring up boat speed. We found we could slowly work our speed up, and point more into the wind - our forward speed adding to the real wind speed, creating an effective wind speed the sum of the two components to drive more air into the sails. Using this technique we were able to do 5-6 knots in 8 knots of wind. That wind held for the day and we kept Profligate in sight throughout the day, a pretty good feat considering it's a much faster boat. By nightfall the wind fell again and we motored the remaining way to the anchorage.

It was a moonless dark night. This was Mike's first time entering an anchorage at night, but with the help of the radar and keeping a sharp lookout for boats and terrain we were able to find a cozy spot on the outside to spend the night.

The next day I went over to Capricorn Cat to take a look at their ailing generator. It was put in just as they were leaving and it turned out the wiring was not quite right. Captain Wayne and I re-wired the generator while the rest of the crew fed us drinks. Capricorn Cat was also hosting a jam session for the musicians in the fleet. By the time we were done with the generator the party up on deck was in full swing. I had a great time hanging out with Captain Wayne, his girlfriend Carol, Christian and Mary. I also met John and Katie. John is a managing editor for Latitude 38 and was just down for the HaHa. Everyone crewing on the boat was great to hang out with. Banjo Andy brought his banjo over and another fleet musician, Tom brought over an electric bass. Several other musicians from the fleet joined in and the music went through the orange and red burning sunset into the moonless sharply contrasting starry evening. Rather than brave the dingy trip back after so much celebration I borrowed a sleeping bag and spent the night under the stars.

The next morning I made my way back home, hung over. I worked off the hang- over jigging for halibut off the side of the boat ( didn't catch any ), took a nap, then made for the beach party. There was a strong southerly swell running right up onto the beach, which neccessitated most everyone to take pangas into the beach. There was a band, and they served mararitas, beers, and trays of fish. As sunset came, the line for the departing pangas grew. A couple of guys who came in their dingy tried to motor out through the surf and had a big wipe-out, both of them being dumped out into the surf as the dingy tore up and down into the waves and back towards the beach as if driven by a demon. It came close to running them each over several times before it finally went up on the beach, corralled by some other dingy drivers who came out to assist. This was the wind-up of the party, and now everyone was waiting for panga rides back to boats in the failing light. The giant swells continued and the panga operators had to stop making runs when it was too dark to see the surf. This left 50 hahaers, myself included, stranded on the beach until morning. Everyone tried to make of the most of it, warming by a driftwood fire, or huddling together in the kitchen where the food for the party was prepared. I found a old abandoned inflatable dingy and tried to use it as a blanket on the rocky ground. I found if I slept on my back I was comfortable but contact with the ground sucked all the heat from my body and left me shivering, sleeping on my side made it a little warmer but the rocks pressed into my hip and side, again making it most uncomfortable to sleep. Needless to say it was a long night and I was glad when the sun's rays started appearing on the eastern horizon.

The pangas finally delivered me back at the boat and we headed out for the big non-start. There was no organized start because so many boats were scrambling to get their crews back from the beach. We again headed offshore first and then gybed around and headed outside the fleet in what we felt was better wind. We re-ran the spinnaker sheets a different way and this greatly simplified our being able to change the direction of the boat downwind. The boat is by all accounts a heavily loaded cruising boat but nevertheless flys down the course. It reminds me of a boat called Raven that I raced against in Banderas Bay Regatta which also seemed to sail really well for a cruising boat. Raven was a Dacheu design, and while Snow Goose is not a Dacheu it does share the same canoe hull with a fat shallow keel. We were keeping pace with one of the fastest catamarans on the course all day long - I'd love to see how the boat sails upwind.

A few hours after dark we had our first spinnaker wrap. I was off watch so I came up groggy from sleep and the spinnaker and sock control lines were wrapped around the headstay. We tried for a while to untangle the mess to get the sock down over the chute but gave up on that and finally did a traditional spinnaker douse with the three of us gathering the huge section of cloth onto deck and into the bag. Mike got the boat sailing again under genoa while Dave and I brought the chute below to re-pack it. We finally sailed into the tropical tempuratures we'd waited for all race long - so repacking a spinnaker for a 50 foot boat in the cramped and hot cabin was a sweaty job. After running spinnaker from bow to stern and re-socking it we brought it back out on deck, double-checked all the lines in the limited illumination of the spreader lights and then re-launched the spinnaker. It took us about 45 minutes from when it wrapped to re-pack it and re-launch it and most of that time we continued sailing the boat. It felt like we did pretty good considering it was the middle of the night.

Leg Three, day two - november 8th, cabo san lucas

The winds died out at about 4 am, just 30 miles from the finish line, so we motored the rest of the way to the line, then rounded lands-end rock and the arches of Cabo and entered the anchorage. On many of the VHF nets the Poobah suggested the east side of the anchorage was the place to be, presumably because many boats will crowd the west end of the anchorage. Unfortunately there was a Adam Sandler movie being shot on the east end of the anchorage, with the port captain shooing everyone out of the east side. As a result the eastern terminus of the anchorage before the movie set filled with tightly packed boats and after anchoring we watched as boats anchored closer than what we were comfortable with, then boats anchored between the first boats and us, and so on. We thought we wouldn't need a shore boat, we could just walk from deck to deck to get to the beach.

Mike and Dave headed into shore to check us in, and I stayed out at the boat for a swim. I didn't get much sleep the two previous nights; spent one of them hypothermic on santa maria beach, and the other one with a lot of off-watch sail changes and spinnaker repacking. So I was moving pretty slowly to get packed up and ashore.

Cherie spotted snowgoose in the anchorage and came out with a friend - Bob, who'd she had met at the hotel. She collected me and we went to our hotel, stopping at a great fish taco place on the way. I caught up on my sleep for a few hours then we got our things together and headed out for a night at squid roe, the locally famous dance bar in Cabo. We had a great time and ran into many friends from the Haha and years past.

-- November 9th

We got a late start and had fish tacos for breakfast. I got myself a haircut, which I've needed for several weeks, then we headed to the beach party. Cherie and I entered the 'here to eternity kiss contest' which involves rolling around in the surf trying not to drown in the waves while looking as graceful and passionate as possible. It was fitting as this was our sixth anaversary of our first kiss which was here in Cabo after the 2001 HaHa. We finished the night with a nice romantic dinner at the Baja Cantina.

-- November 10th

We packed up our gear, checked out of the hotel, and headed out to Snow Goose. After dropping off our stuff and hanging out with Mike for a while (Dave had flown home earlier in the day) we went over to Moon Tide for a day trip to a snorkeling bay up the coast. Moon Tide is a 47 foot lagoon catamaran captained by Bill Lilly, and his crew of four is Mark, Jennifer, Tessa, and Angelina. Also joining us on the journey was Lynn from Wahoo and Nate from another sailboat. We anchored in this small beautiful cove, with three other boats - mostly filled with snorkeling tourists from nearby resorts. The water was stunningly clear and schools of brightly colored fish swam below our feet. The cove was surrounded on either side by towering jagged tan and red rock formations, with coral covered spires of rock just below the surface, and the inner side of the cove had a gently sloping white sandy beach.

We returned from this beautiful retreat and prepared for the awards ceremony. We managed to secure a second place out of our entire fleet, only losing out to one boat who endured through the calms and sailed the entirety of all three legs.

And that wraps up the 2007 Baja Haha for Snow Goose. After a few days of preparation here in Cabo we'll be heading for Puerto Vallarta by way of Isla Isabella. We had a great time, and am very thankful for getting to meet so many great folks sailing down through Mexico this year.

The End

"From California to Wisconson, 1700 miles in an experimental aircraft"

2007-10-02T00:00:00-07:00

In late July I flew my Long-EZ, a homebuilt experimental aircraft, from California to Wisconson to spend a week with friends at a lake house and to briefly attend the Oshkosh EAA airshow. The journey was 1700 miles and took me over some of the most inhospitable terrain in the country.

My first flight was a very short hop from Hollister to Tracy on Friday
evening. I wanted to get started at first light Saturday, and often
Hollister is fogged in mornings. I picked Tracy because it's the cheapest
gas inside my flight test area and so I've flown in there several times.
The downside is that there's nothing for food or lodging around the
airport. So I filled the tanks and rolled out a sleeping bag on the
tarmac. The wind was blowing and the stars were out. After I got an
abbreviated night of sleep I woke up, had a bit of an MRE and started
trying to get my telephone flight briefing. It was 5AM, and it seemed that
there were very few flight briefers and that probably everyone else was
trying to get a Saturday morning start to Oshkosh. The other unfortunate
thing is that the payphones will hang up after 10 minutes on a 1-800 call.
So after a few 10 minute hold-and-hangups and then going from my cellphone
I was able to get a flight briefer, get the weather, and file a flight
plan. It caused about a half-hour longer than I expected and so I left
with the sun rising in the east.

An unexpected stop in Wyoming

The worries for pilots crossing to Oshkosh is the Sierra and the Rocky
Mountains. High mountains often host turbulence and thunderstorms. But if
you get an early start the heat of the day hasn't had a change to make
those areas dangerous. I chose an altitude that I /thought/ I wouldn't
need oxygen (and below the FAA altitude for oxygen) but high enough to
avoid the winds and turbulence over the high terrain. My worries were
weather, or reliability of the aircraft. I didn't expect physiological
problems with the pilot (me). After four hours in flight I started
developing a slight headache. Crossing the Rockies east of Salt Lake City
I picked up some minor turbulence and I got a bit nauseous. I was a bit
puzzled, as I never gotten motion sickness in many of the rapacious
conditions I'd experienced sailing, but chalked up my nausea to the quick
motion of air turbulence. As I progressed for another half hour my
headache got stronger and I decided I was due for a break, so I landed at
Rock Springs, which is at 7000 feet (over a mile high) but has a 10,000
runway. The landing was unremarkable. I got down and rushed to empty my
bladder, and then tried to sit back for a bit and recover. At the time it
hadn't occurred to me why I was feeling so bad. I thought perhaps I was
dehydrated or had a bit too much sun through the plane's bubble canopy.
But after a half hour I felt worse and threw up. I drank more water and
tried to give it a little longer, but didn't feel any better. I used the
web browser on my phone to look up some stuff on Wikipedia and it looked
like I had Altitude Sickness. It's typically something that mountain
climbers experience. My training as a pilot only covered the short term
effects of the lack of oxygen (hypoxia) but didn't really cover long-term
effects of low-oxygen environments. Cessnas don't fly for five hours at a
time at eleven-thousand feet - which is why I suspect it's not covered
very much for a private pilot's license. So I asked the FBO to get me a
shuttle to a hotel for the night in hopes that a good days/nights sleep
would fix me up.

I stopped by two drug stores looking for altitude sickness medicine. Rock
Springs Wyoming is at 7000 feet altitude, one of the highest places you
can live in the US. It is also 900 miles from the nearest ocean. So if
anyone can tell me why the drug stores carry a half-dozen sea-sickness
medications and nothing for altitude sickness please let me know! On the
suggestion of Cherie I got some acetaminophen based headache medicine
which seemed to help.

At the crack of dawn (again) I caught the hotel shuttle. The weather
outlook for the day's flight was good, clear skies to Mercer with some
thunderstorm activity that was expected to remain north of my flight path.
I still had the fuel I needed to get to Scottsbluff, so off I went. I flew
lower, only a few thousand feet above the broken terrain through the high
Wyoming plain. Soon enough I was happily out of the Rockies and over some
lower-altitude and more hospitable terrain. A quick bathroom break and
some more fuel in Scottsbluff and I were off to Mercer, Wisconsin.

Not your typical California weather

For this leg of the trip the terrain was much more hospitable. Flat
farmer's fields for as far as the eye could see, however I was further
from what I'd consider civilization. No interstate like my trip through
the Rockies. Very few airports as well. I felt off the beaten path as it
were. A few hours into my flight I decided it was a good time to get a
weather update to see what the possible weather was doing ahead of my
route. I dialed up the flight-watch frequency on the radio. Flight-watch
is a service that provides in-route weather updates as pilots request
them. The weather system with the thunderstorms was moving faster than
predicted into the Minneapolis area, and I was advised to divert further
east to avoid it. I continued flying and you could see the dramatic swept
clouds ahead of the front, some of them with light sprinkles underneath.
It was as if the clouds were charcoal drawings and the bottoms were
smudged towards the earth. The sky progressively but very subtly got
darker as I smelled the moisture in the air ahead of the coming storm. The
clouds were not the thin white clouds of California, but towering beefy
clouds of substance. I was apprehensive but knew I could divert, and
unlike a sailboat I could out-run any storm.

As I approached Minneapolis and its surrounding congested airspace I
started talking to the controllers who were busy with routing aircraft
around the weather that had set itself upon Minneapolis ahead of schedule.
I was advised to fly south-of-east, which was even further from my
intended destination. The clouds were getting lower and several times I
found myself informing the controllers that I was descending to get below
the cloud base. The weather continued to get lower and darker and the busy
controller handed me off to another frequency. All I got on that frequency
was dead air, and the weather continued to get worse. I switched back and
the controller told me to go to the new frequency and keep trying... I was
counting on the controllers keeping me away from any really nasty stuff
that showed up on their radars and falling through the cracks of the air
traffic control system was unnerving. I decided I had enough, and I really
needed to LOOK at the weather radar with my own eyes, so I punched the
'Direct To' button on the GPS and headed for the closest paved runway... A
charming place called 'Dodge Center'.

I landed at the quiet unassuming airport, with a short roll-out due to the
stiffening wind. I wondered into the pilot lounge and found a technician
tinkering with the weather terminal. Very friendly folks in dodge, he
helped me navigate the two weather computers... It seems there's this one
particular type of flight planning / weather computer that are present at
many of the pilot lounges I've stopped at. My personal preference is just
a plain old web browser and a high-speed internet connection and I can
find my own way. However internet savvyness and broadband are less common
away from California and these terminals do the trick okay. A line of very
nasty weather stretched from Minneapolis to Rheinlander, blocking my path
but moving southeast. Behind it, still in Canada was another band of very
similar weather. I evaluated my options and it seemed to me the best thing
was to give it a few hours to move south of my flight path and depart
before the line behind it moved into the region. This would give me a few
hours to get some late lunch and explore Dodge Center.

After so many hours in the plane I looked forward to what I was told was a
long walk into town. It's funny, I get lots of offers for rides whenever I
make one of these stops after a long trip, but I can't imagine sitting for
a moment longer after I get on the ground. Now Dodge center is not what
you'd call a metropolis. I walked to the downtown, which was about two
blocks long, with half the stores closed down. I did find the one
restaurant, and it had one thing on the menu. I passed; I wasn't in the
mood for fried chicken and mashed potatoes. Luckily the grocery store on
the way back to the airport had some hot food to satisfy my hunger. Back
to the airport, another look at weather - looks like my window is opening
up. I put a few more gallons of fuel in the plane, make the call that I'm
on my way, and off I went.

I got up in the air and the clouds were still pretty low, as I'm right on
the tail end of the tempest. I close in on the Minneapolis airspace, and
this time I choose to just stay at the edge, clear of the controlled
airspace to give myself more options on dodging clouds as needed. From the
air you can sense the slick water everywhere on the ground... The wetted
cement shines more from the air, and the grass is a darker green. As I
progress behind and away from the storm the clouds get higher, holes
appear to the blue above, and I climb up and into a blue sky with friendly
white puffy clouds... As I climb the terrain below changes from farmland
to pine forest so I am relieved to get more altitude. Altitude is life,
and options... When everything is farmland you can land anywhere; with
airplane-eating pine trees altitude provides the comfort of a further
glide to an airport or clearing if anything were to go wrong. Further into
the pine forest there are less signs of civilization. Towns become fewer
and roads less obvious below the forest canopy. I close in on the quiet
Manitowish towers airport and circle down evaluating the runway below.
Manitowish has a narrow and relatively short runway fringed with 50ft tall
'airplane eating pine trees'. It's important here to pay attention on
landing or takeoff here or you could end up in the trees. As luck would
have it the wind is directly across the runway, adding additional demand
for skill to the landing. I came in for landing and as I get close to the
ground I really realize how narrow the runway is. I'd call it
claustrophobic, when lining up, while being attentive to airspeed and
approach angle, and slipping to one side so the crosswind does not blow me
into the grass. I find it's usually the 'easy' landings that are the
roughest due to not having the same focus as when the landing is
challenging. When it's hard and I'm very focused my landings come out
textbook and so while I was stressed the landing came off perfectly and I
taxied in and shut down; a journey of 1700 miles finally complete.

My weekend searching for Steve Fossett

2007-09-15T00:00:00-07:00

The Stratofox team at Minden]

As I m sure most of you know adventurer and record-breaker Steve Fossett went missing on a flight in western Nevada on Labor Day. A group I m involved in tracks ameture rocket and balloon launches, primarily via ham radio. Several of the folks on the groups mailing list expressed interest in helping for the search for Fossett whatever way was possible, and so Thursday night we met and made plans to go out to Nevada and try to do some searching ourselves.

One of the group, Ian, read in a news report that Fossett went flying to scout locations for his land speed record attempt, and he knew that one salt flat that Fossett was interested in was in Eureka county. So Ian emailed the eureka county sheriff to let him know. Ian got a call back a few hours later that a geologist in a mountainous area east of eureka heard what he believed to be a distressed aircraft. The Sheriff contacted CAP about it, but the CAP was not interested in perusing the tip. So we decided that we should search the area ourselves in the interest of leaving no stone unturned in the search, as CAP had their hands full to the west in the area surrounding Fossett s departure point.

It was decided that I would fly out to Eureka on Friday, and the rest of the team would meet me in central Nevada Saturday morning. So I loaded up my plane with camping & survival gear, fueled up, and headed across the Sierras for Eureka Nevada.

Eureka is on US 50, legendary loneliest highway in the US . The flight across really gelled in my mind how expansive and formidable the terrain in Nevada really is. I flew at fifteen thousand feet to stay well above the arid alkali scrub plains between the rocky tree-covered mountains, the mountains and planes alternating at regular intervals. Civilization was sparse, and most of the area is used by the military for artillery and bombing practice. Having a wide margin of altitude over the angry unforgiving terrain below was a comfort.

I landed at Eureka and was greeted by John and Vincie, the husband and wife who run the airport for the County. They moved out from Missouri to retire, but when the pilots at the surrounding airports learned what a good mechanic John was John ended up with much more work than a retired person should have. The airport itself has a single hangar, attached to the airport office, and also attached to their home. Only one other aircraft is sitting on the tarmac next to mine. All in all a very quiet airstrip.

Of all the folks you meet, you often meet the most interesting folks in out of the way places. The whole county s population is 1900. The city itself has a population of only 650. Most of the residents are involved in Mining operations taking place in the mountains surrounding the town.

This is where I met Dan, at the Eureka Airport. He s the kind of person out in the country that looks a little rumpled and rough, talks in a slow and deliberate manner, and is very bright despite his outward appearance. Dan has lived in Eureka all of his life but Dan and Steve have gone out scouting the Diamond Valley salt flat several times. Dan is a pilot and has done Search and Rescue for the county, so was a wealth of information on where and how to search, and what places a small plane could land in the countryside. It also happened that the airport was out of fuel and would not get more until the next day, so Dan siphoned 10 gallons out of one of his planes for me to use until the fuel shipment arrived.

Sheriff Ken Jones also drove out to the airport to meet me. We talked a little about the tip he got, more about this neat plane I flew into the airstrip and he finished by telling me if I needed anything, or rides into town to give his department a call and he d send someone out to shuttle me around. The generosity of folks continued to amaze me in Eureka.

I originally was supposed to fly to Austin, NV for the night but the lack of fuel at Eureka made me decide to stay the night. John and Vincie offered to call the hotel in town for me. Unfortunately the nice hotel was full, and they said they wouldn t feel right letting me stay at one of the lesser hotels so they put me up and fed me for my stay in eureka! The hospitality they had for a stranger was amazing, and it was great to be treated like family after arriving out of the blue at their home.

In the still morning air I took off with my siphoned fuel and headed for my search area. My plan was to make north/south passes at a one mile interval. With the early morning sun the shadows were long and most of the west slope was in shade, so I did my best looking down into the mountainous forested terrain. I would have spotted someone who found a clearing and had a survivable landing. But if someone went straight in there would have been little chance of seeing him from the air. After actually doing this sort of searching I understand how planes disappear into the mountains and even thorough searches may not turn up anything.

After loading with the newly-arrived fuel and a wonderful lunch prepared by Vincie, I bid farewell to Vincie, John, and Dan, and left for Minden where the rest of the team was located. The trip back was much like the one out, and I climbed up to twelve thousand to avoid the turbulent hot air over the desert.

I landed in Minden, parked the plane. A half dozen satellite trucks sat in the parking lot, the ramp was full of Civil Air Patrol aircraft, the grass across from the airport office had the tent of the Nevada air national guard. The airport was a buzz of activity. Once I met up with the rest of my team we talked to CAP and they had no use for us, and basically no interest in the areas we searched to the east. We were told not to do any flying in their search area so we decided as a team to just go do a driving search of the areas around Mono Lake with the remaining daylight. The conventional wisdom of the day is that he was somewhere south of the Flying M Ranch. We arrived there just as the sun was setting, and the binocular search turned up nothing. We decided to check out a small nearby airstrip on our topographic chart. Off the pavement and through a dirt track and just as we were nearly to the airstrip we ran into private property flying M ranch . The ranch is a big place, stretching all the way down to Mono Lake..

There was little else we could do, so that evening the team disbanded, those who drove up drove home through the night and I flew home Sunday morning.

"Home Fabrication"

2007-05-21T00:00:00-07:00

This weekend I went to the Make Faire here in Silicon Valley, put on by Make Magazine. make is geared towards folks who enjoy making things with their own hands, inventing and creating instead of simply consuming what's available at the store.

The most amazing technology at the fair was the home fabrication / 3D printer technology. There were several units there, but one caught my eye. The Fab @ Home unit, designed for hobbyists, is a unit that can be assembled for just two thousand dollars in parts and a weekend of work.

This will prove to be one of the most distruptive technologies to come along. Home fabrication will make the copyright issues with MP3s look like a cakewalk. When you can print your own furniture, clothing, and other housewares, just by downloading designs from your friends.

"Yahoo Pipes, a very neat app!"

2007-02-08T00:00:00-08:00

Well, web 2.0, for me has a lot to do with making data available in an agnostic manner, wether that be via RSS or via a web services API. Data tied to a presentation layer, such as a traditional website, is data that has no future outside that website. The rise of mash-ups is enabled by data being decoupled from it's presentation. Being combined with other data makes that data more valuable.

Until now you've needed to be a reasonably adept programmer to put together different data sources to create mash-ups. But not now. Yahoo has just launched an application that allows anyone with the most rudementary conceptual knowledge of programming to create new mashups.

Yahoo Pipes is the new application, and it allows anyone to easily string together web data sources and funnel them through some rudimentary filters to create new mash-ups. Yahoo has been a bit absent with the whole innovation thing since Google became the industries' darling but I think this marks their comeback in a big way.

There are a good series of articles on the O'Reilly Radar about why it's important and how it works. Tech crunch has a good mention about Yahoo! Launching Pipes and There's a nice bit about it from Yahoo MySQL guru Jeremy Zawodny.

The excitement about this product is very high in the tech community, resulting in someone as big as yahoo having their new service overwhelmed. So be patient when trying it out until they've got some new servers spun up!

Technorati Tags: pipes, yahoo

"Self-healing networks"

2007-02-01T00:00:00-08:00

Last year I wrote an article on building a self-healing network with off the shelf software components. If you are responsible for managing a large UNIX/Linux network it's a must-read...

An excerpt from the article:

_Computer immunology is a hot topic in system administration. Wouldn't it be great to have our servers solve their own problems? System administrators would be free to work proactively, rather than reactively, to improve the quality of the network.

This is a noble goal, but few solutions have made it out of the lab and into the real world. Most real-world environments automate service monitoring, then notify a human to repair any detected fault. Other sites invest a large amount of time creating and maintaining a custom patchwork of scripts for detecting and repairing frequently recurring faults. This article demonstrates how to build a self-healing network infrastructure using mature open source software components that are widely used by system administrators. These components are NAGIOS and Cfengine._

ONLamp.com -- Building a Self-Healing Network

Technorati Tags: nagios, cfengine, healing

"My client, Avvenu, releases a great app"

2007-01-30T00:00:00-08:00

Avvenu got them selves some very good press after releasing their music player app at CES. They showed up on Techcrunch which linked to the article Go2web2: Your own iTunes PC! They've even gotten themselves well Digged. Their application allows you to listen to the music on your PC from anywhere, on most any device. Works for most mobile devices and web browsers. The coolest feature though, is the ability to share your music with your friends! You can share a playlist and your friends can listen to it via a nifty little flash app.

They posted a funny little demo of the Alpha version of the product.

Technorati Tags: avvenu, itunes, mp3

"Project Management Tools."

2007-01-30T00:00:00-08:00

I've been trying to find some project mangement software lately that's compatable with the Getting Real development methodology. For our new 'Locomotive' project I'd love to find a tool where I can set out all the required tasks, and assign an hour value to them to create a timeline for how long each part of the project will take. This should help me decide if parts of the project should be trimmed. It also jives with the "Getting Real" suggestion that tasks be broken into 4-hour chunks.

So I went off into the web (version 2.0) to evaluate several web-based project management systems. The first one I tried was Basecamp, the 37signals offering. I found there's no time management available in the base product. So I continued on and checked out Zoho Projects . This also was missing the functionality I required. Finally I tried out Devshop. This had really great time mangement tools. I would recommend this one for medium-sized team development projects. The screencasts were very impressive. However it didn't quite do what I wanted as the smallest unit of time it supported was 1 day. So for now it looks like I'm back to the web 0.0 pen-and-paper method, or perhaps an excel spreadsheet. If you have any project management suggestions please leave them in the comments!

"A mid-January update

2007-01-23T00:00:00-08:00

Cherie and I are still in Maryland, where we've had a strange winter so
far. So far we've only experienced snow covered roads and icicles
hanging from trees when watching the news in California... But yesterday
we finally got our first snow of the winter.

I'm borrowing wireless Internet access from as best I can figure Amtrak.
With my High-Powered Wifi card and cantenna pointed at the Amtrak rail
bridge I pick up a very nice low latency high bandwidth wireless access
point.

I've been taking all the time we've been spending inside away from the
cold to catch up on my reading and sharpen some of my skills. To that
end I've again picked up my Ruby on Rails (www.rubyonrails.org) book and
have released my first public RoR application/website... I've been
meaning to produce a website for my Experimental Aircraft club in
Hollister since joining a year ago. With the free time in Maryland I
finally had time to complete it. It's up at http://www.eaahollister.org/

Cherie and I have also been taking the opportunity while out here to get
into better shape. To that end we've been eating much healthier. We've
found the book 'You, the Diet.' very helpful in that regard. It
basically contains the same fundamental eating tips that I usually
follow on a diet... Mainly avoiding high-glycemic foods (i.e. heavily
processed foods like breads and sugars) and eating several small meals a
day. We've also been doing exercise daily; Usually my exercise has been
mostly cardiovascular but this book has taught me that strength training
is also an important part of increasing your metabolism.

Another California Road Trip

2006-11-06T00:00:00-08:00

California Road Trip Photo 1

Heya Everyone!

Cherie and I had Halloween in the Castro district this year, but
leading up to it I had the most interesting journey from Manteca, where
we were staying with Cherie's family, to San Rafael, where we were
staying with our friends Sam and Allison. I think I set a personal best
for the number of modes of transportation taken in one day. Cherie's dad
Stan drove me to the Modesto airport. I then flew my plane to Hollister.
In Hollister I picked up my car and drove to the San Jose airport, where
I put my car into long term parking then took a bus to the train
station. Took a train up to San Francisco, where I then got on the Muni
(the SF 'subway') to within a few blocks of the ferry terminal, walked
to my ferry, which took me to San Rafael, where Sam picked me up and
drove me the rest of the way to their home. So to summarize... truck ->
airplane -> car -> bus -> train -> subway -> feet -> ferry -> car. All
in a single day.

For Halloween we stayed the night with Margaret in SF and carved
pumpkins then met Mark out on at the crazy Castro Street party that's
become a big yearly event. It has become the place to be if you are
into costumes. Some of our favorite costumes were 'Madonna with baby',
'hot dog on a stick', 'my myspace page', and 'Michael Jackson with the
neverland ranch children'. There was a heavy police presence, because
whenever you get together a third of a million people partying within
just a few city blocks you inevitably get some hoodlums. This year was
no exception, and unfortunately an altercation turned into a shoot-out
where 9 people were hurt.

Following Halloween Cherie and I did a nice trip down the cost (by car).
We spent two days in Tahoe hiking and hanging around. Hotels there are
really cheap between summer and the ski season, and if you don't mind it
being a little chilly it's very beautiful in the fall. Then we drove
down 395 and stopped at Mono Lake. These amazing underwater spires
created by minerals in the spring water appeared after LA started
draining the sources for the lake to provide water down south. The
reduced water level revealed the spires in the lake. Now they are going
to restore the lake and that means many of the spires will be underwater
again when they raise the water level. We finished the trip spending a
day in Joshua Tree where the hardy cacti flourish surrounded by rocks
stacked as if a giant child's block set.

Cherie and I are planning to winter in Maryland at my Mom's vacant
condo. We'll be making the trip across the US visiting friends,
relatives, and locations of historical significance along the way. We're
leaving after Thanksgiving. Here's a map of where we'll be:

http://tinyurl.com/yd9vqx

California Road Trip Photo 1 California Road Trip Photo 2 California Road Trip Photo 3 California Road Trip Photo 4 California Road Trip Photo 5 California Road Trip Photo 6 California Road Trip Photo 7 California Road Trip Photo 8

"SoS Framework"

2006-06-27T00:00:00-07:00

I've been playing around with Ruby on Rails lately. RoR is a framework for creating clean web applications. RoR has a lot of promise but I'm not yet up to speed enough with Ruby to really knock out apps with it quickly. I keep finding myself reverting to PHP.

My PHP KungFu is strong. However I've recently gone back and looked at some of my PHP code after playing with the RoR MVC methodology and realize that PHP is really conducive to really UGLY hackish code. It bugged me to try to work on some of the code I'd written in the past. So as a result I decided to give a swipe at writing a lightweight framework in PHP similar to the RoR stuff. It's certainly not as complete as RoR, but it gets done what I need done. You can find my new code at http://www.rage.net/sos_current.zip. The new system is called Spaghetti on Snails. Hope somebody else finds it useful.

EZ does it.. the adventures of N620CS

2005-10-16T00:00:00-07:00

The fuselage of my Long-EZ

I have been working on my single-engine private pilot's license this summer. I'm fairly close to completing it so I've taken another big step . I just bought an airplane. The airplane is a Long-EZ, and am working this next month to finish it and get it flying. It was quite an adventure to get it out from New Jersey to California. It's the third time I've driven from coast to coast, and the third time I saw nothing but highway and service stations in between! One of these days I'm going to ship my car to NJ and take a month off to really see the stuff in the middle.

After arriving in California I talked to Tony & Michelle, and they agreed to let me take over their garage for a month while I finished up the projects that needed to be completed prior to moving to the airport.

Painting over the weekend was a learning experience but ultimately a success! My biggest problem was that the high-build primer sets inside the spray gun very rapidly, and it took me a while to figure out the proper cleaning procedure for the gun to keep it from clogging!

Painting was an 'afternoon project' that ended up taking from Friday evening to Monday morning. However the result is that I now have a plane that's ready to go to the airport in Hollister.

The fuselage of my Long-EZ Trailering across the US, from New Jersey to California Starting the painting process Repairing the nose damage from the trip The primer is done! Burt Rutan: 'You can have it in any color, so long as it's white.' Tony admires the paint job What it looks like when it's done Why was the plane on a trailer? A 9.5 foot wide plane doesn't fit into a 8 foot wide truck! My Long-EZ in the hangar at Hollister

"The upcoming Avvenu beta"

2005-02-18T00:00:00-08:00

I've been brought into a great startup and we've been doing some amazing things designing and deploying a network for a service that'll go into public beta very shortly. The company is Avvenu, and you can sign up for the public beta which will be starting very very soon.

The Avvenu service allows you to share your pictures and other media to friends and remotely access it from anywhere with an Internet connection, and all this without having to upload anything. Everything you share stays on your computer, which connects to the Avvenu network over your broadband connection and you and your friends download directly from your home PC.

There's been several write-ups of the service from PC World, Network Fusion, Jeff Nolan's blog, and Snoop Dogg

And, BTW, you're getting a sneak peak of the technology in action. The images in this blog entry are served through the avvenu service!

"Fort Bragg, California"

2005-01-07T00:00:00-08:00

An angry sea in the foreground, but off in the distance a rainbow decorates the sky

I had broken my golden rule and I was paying for it! It's the middle of winter and I'm not in the tropics; the first time in four years. It's dark by five, the ground is frozen in the mornings, and it's rained for a week.. I was over getting rained on in the Bay Area and I had a long weekend. I was ready to get rained on somewhere else. Not wanting to face travel complications on a holiday weekend I opted to avoid the airports or the roads east to ski areas and instead headed north to Fort Bragg.

It was a blustery winter day as I winded my way along US route 1, a road that clings to the cliffs and snakes it's way along the Pacific shoreline. The rain came down with a violence that was matched by the waves' assult on the rocks and beaches below. Occasionally I'd reach a particularly scenic overlook, the rain would abate slightly and I'd be able to snap a few pictures. In a few moments a new rain cell would blow in and continue the downpour with renewed vigor and I'd continue along my way.

After a full day's drive I reached Fort Bragg, a quiet town of a few thousand, a ghost of it's self when the town was the regional hub for logging the redwood forests. Now the massive sawmills that seperate the shoreline from the city are all skeletal abandoned structures withering in the summer sun and winter rain.

In this sleepy town the only place that a stranger could get in on the new years festivities was a biker bar called the tip-top. Even the biker bars in Fort Bragg are a bit sleepy. I met a couple from Pittsburg (you came all the way from the east coast?? Hah, Pittsburg California!) who spend their winters running youth dirtbike racing. They also got the bug to just 'get away' this weekend and Fort Bragg is their favorate place to do that!

An angry sea in the foreground, but off in the distance a rainbow decorates the sky The rain brings out the vibrant colors of the vegitation along route 1 Bodega bay is particularly peaceful this afternoon The typical (beautiful) coastline along route 1 The Russian river empties into the ocean in a flood of red mud Fort Brag has a small port for the local fishing fleet Here's a dingy that'll never deflate! Fort Bragg is the terminus of an abandoned rail line..

Skate or Die!

2004-12-30T00:00:00-08:00

What happens when you get some professional. skaters. a skateboard, a few beers, and a camera all on the same playground?

The premiere skating video of the 21st century!

Skate or Die!

Skate or Die! (5 Megs, 30 seconds)

"There is no spoon, aparently."

2004-11-30T00:00:00-08:00

Were there
Hundreds of protestors or Dozens of protesters? Who knows.

"a bit on bitorrent."

2004-11-25T00:00:00-08:00

BitTorrent on NPR

and >BlogTorrent

"Something wonderful"

2004-09-20T00:00:00-07:00

Well the state of the technology industry right now is pretty interesting. All the big bellweathers are adrift; Sun, Motorola, Intel, Cisco.. Some people find that scary, but I think it's an indication that everything is about to change. There's always uncertanty before some new amazing 'thing' refocuses the industry in a whole new direction. I'm not sure what that direction is, but I want to be part of the 'mosaic' of that new direction.

The whole thing reminds me of the movie (and I suppose it was once a book) 2010...

Dave Bowman : You see, something's going to happen. You must leave.
Heywood Floyd : What? What's going to happen?
Dave Bowman : Something wonderful.
Heywood Floyd : What?
Dave Bowman : I understand how you feel. You see, it's all very clear to me now. The whole thing. It's wonderful.

The Vineyard Harvest

2004-09-19T00:00:00-07:00

Grape clusters

I spent this Saturday at Dan and Diane's place helping them with their yearly harvest of grapes for their upcoming Cabernet (which I must comment, the best Cabernet I've ever had was from their harvest two years earlier). Dan, Diane, Bernadette, Tom, Teresa, and Carrol spent a wonderful day harvesting this year's crop then enjoying a few bottles of wine from previous year's harvests.

If you are not familiar with the process; The grapes are harvested at the end of summer as their sugar content becomes high enough for a decent alchohol content in the wine, but before the grapes turn to rasins. The harvested grapes are then crushed and de-stemmed, the native yeast is killed off; winemaking yeast is added, then it's stored where it can oxigenate for a few weeks. It's then transferred to a sealed tank where it can off-gas but no oxegen can enter.. Then it's bottled and a year or two after coming off the vine it's a beautiful bottle of wine.

It was a fabulous weather for harvesting, as we had a cold front roll though which kept the tempuratures low and the hot sun at bay with a bank of clouds. Dan and Tom have been working over the years to perfect their home-grown wines and it's amazing to see them working on the chemistry and mathematics involved in producing that oh-so-perfect bottle of wine. If I knew yardwork was so fun I may have reconsidered owning a home and settling down!

Grape clusters Harvesting grapes Vineyard work Harvest day

"Enemy of the state."

2004-09-10T00:00:00-07:00

Apparently it's okay to be 'trading with the enemy' of our government if you are the government.

Vermont made a 8 million dollar deal with Cuba. But I can't buy a six- pack of buccanero without facing time in the federal pen. That's just jacked to me.

"Perhaps..."

2004-09-08T00:00:00-07:00

Perhaps if you can't remember to turn off the lights you should be driving something a little simpler like a VW.

"You'd lose your job for this."

2004-09-08T00:00:00-07:00

If I never showed up to work I'm pretty sure I'd be fired. As would any of us. But I suppose it's different if you work in washington. Glad we are paying folks like John Kerry for doing nothing. He hasn't been present for all the items of legislation voted on as far back as yahoo has tracked it (NV means no vote). Your tax dollars at waste.

Senator Kerry





Greg Retkowski


411 Walnut St #1716


Green Cove Springs, FL 32043








September 8, 2004





Dear John,





Hello, as a citizen of the US and therefore as your employer, I am afraid


I have to notify you that we've decided to terminate your employment for


lack of attendence at work. When we hired you your responsibilities were


to attend sessions of congress and vote on legislation and you've been


unable to complete these duties on a regular basis. We are always seeking


to cut government waste by terminating employees who draw paychecks while


not performing their duties, which is a category you fall under.





We wish you the best of luck in whatever future endevours you undertake.





Sincerely,








Greg Retkowski

"oh, I'm totally disenfranchized right now..."

2004-09-07T00:00:00-07:00

Thank you for visiting the Florida Department of State website.

Due to the impending storm, Department of State Internet services will be unavailable beginning.

6:00 p.m. Friday, September 3, 2004, through Tuesday, September 7, 2004 at 9:00 a.m.

barring any further interference resulting from Hurricane Frances.

We apologize for any inconvenience and hope you will visit again soon.

http://election.dos.state.fl.us/regtovote/regform.shtml

Henry Coe State Park

2004-08-17T00:00:00-07:00

Park landscape

Fourty eight hours of solitude, just what the doctor ordered. When I work in the bay area pull a pretty intense schedule and sometimes I just have the need to get away from it all. And an amazing place to get away from it all is backpacking in Henery Coe State Park. Coe is the largest state park in northern california and is located just east of Morgan Hill. It has sweeping hills, amazing vistas, beautiful flora, and abundant critters.

Not only is it relaxing and a great way to get away from it all, but it's cheap as well. Backpackers permits are only $3 per day, which includes parking. I made a final stop at the safeway off 101 at the Henery Coe exit and stocked up on gatorade, water, and food for my two day sojourn. Then I drove up into the hills were the park begins, threw on my pack and headed out. The fourty eight hours I was out backpacking I only saw three other sets of trekkers, otherwise it was just myself and the wilderness.

Park landscape Rolling hills Park vista Natural beauty

"Dude, where's my angst?!?"

2004-08-05T00:00:00-07:00

Well I've taken to listening to internet radio at work, it's a good way to block out the distractions and concentrate on the tasks at hand when living in cubeland. 75% 'Digitally imported' trance/goa and then the other 25% is industrial/darkwave stuff. I just laughed out loud, when I realized that with my lifestyle I have no right to be listening to anything this angry! Even the failed harddisk on a server today was not angst inducing. Some day I'll encode and post the MP3's from 'Analog Deth', my early-90's industrial band. I had lots to be angstful about then. Not so much now!

"Bittorrent brings cable (almost) on demand"

2004-07-14T00:00:00-07:00

Got Movies? Lots of movement in the BitTorrent space. Torrentocracy could kick Tivo's ass.

Passive Social Networks

2004-07-13T00:00:00-07:00

Gmail is an interesting excersise in the economics of information markets that I mentioned to you yesterday. Google is creating an artificial shortage of gmail accounts by only releasing accounts to people already with an account.

Creating an artificial shortage of something that's in (virtually) unlimited supply is an interesting way to create an awful lot of demand. It's also interesting because by having this system for distributing email addresses google is learning something about the relationships between it's networks. It could, in theory, build a friendster-like relationship database just by watching what parties correspond by email, or over the web by observing which parties link to other parties. What's interesting about this is that they can be able to build this social network structure pasively (by observation) rather than actively (by some person clicking on 'this other person is my friend').

Silent but for the gentle sound of wind across the wings (all photos from www.soarhollister.com)

2004-07-11T00:00:00-07:00

Acrobatic glider in flight

Yesterday I went up with a friend in an acrobatic glider. It was an amazing experience that I think weathering a few storms at sea helped me prepare for. Russ, te pilot had an extensive plan of loops, rolls, hammer-head stalls, and several other manouvers high above the agricultural fields of central california.

As we started the routine and dived sharply to gain the airspeed for the first loop I immediately started feeling queasy; as the contents of my stomach were moving in a different direction than the glider. But then I forgot completely about my stomach as the adrenalin kicked in. Russ put me through the full workout, and then asked how I was doing, I shouted 'more'! So a few more snap- rolls and another loop and we wre finally down to the altitiude at which we'd need to return to the airport. As we calmly glided back twards the airport the excitement/terror of the aerobatics wore off and my stomach re-asserted how unhappy it was about all this. But it settled down gradually without the need to decorate the cockpit with my lunch. On the way back Russ gave me a few moments of instruction then handed the controls over to me. I did a few sweeping turns but for the most part the glider flew itself. Finally as we approached the airport we did one high-speed pass just off the top of the runway, climbed and swept left, circled around and landed.

The Hollister Gliding Club is located 45 minutes south of San Jose. They offer training and glider rides (hey, what a great gift idea!).

For more information visit www.soarhollister.com

The MDM-Fox, Technical data

Acrobatic glider in flight Glider tail panorama Glider in flight Glider on the ground

Dogged on Slashdot

2004-07-07T00:00:00-07:00

1997 Clinton era photo

On a recent article on slashdot my KnowTraffic site gets dogged (again!) because it looks like just walked out of 1997!

I have a friend working on the design. I officially fire myself from all graphics arts endevors! Lemme stick to code and systems!

History in Flight

2004-06-23T00:00:00-07:00

SpaceShipOne takeoff

On June 21st, I witnessed a moment in history. In the Mojave Desert, space ceased to be a place restricted to superpowers. On this date Scaled Composites launched the first civilian spaceflight and made space just a little more accessable to everyone. I filmed the event and have the video available for download here.

The 'launch system' involves two aircraft. The first, White Knight, takes off from a conventional runway and climbs to 50,000 feet with it's two jet engines over the period of an hour. It then drops the second aircraft, SpaceCraftOne from it's belly. SpaceCraftOne then fires it's hybrid rocket (it uses a solid fuel and liquid oxidizer; a system that is extremely pirotechnically stable and safe) and climbs straight up to 100 kilometers above the surface of the earth. On decent, the aft half of the wing folds vertically and the aircraft decends belly first until enough speed is bled off that the wing returns to it's normal configuration and the craft glides to a landing on a conventional runway.

My video of this historic event is titled 'History in Flight' and documents the events of that early summer morning in the Mojave.

Download History in Flight (5meg)

Special note to webmasters, you may not link directly to this movie, however you may either mirror it locally and provide that as a link, or link to this page.

Check out the production stills at the right.
Links
Scaled Composites
X-Prize
Mojave Spaceport

SpaceShipOne takeoff SpaceShipOne and White Knight in formation SpaceShipOne at altitude SpaceShipOne landing SpaceShipOne against the sun Astronaut Michael Melville holds a sign 'SpaceShip One, Government Zero'

"My email makes me happy"

2004-06-16T00:00:00-07:00

I am SO HAPPY to be back to an IMAP based mail system! The thunderbird mail reader rocks. Now I can click on links in my email reader and it does intelligent things; I don\'t have to open up a scp session every time I want to send someone a picture or file, life is good.

My hosting company Hurricane Electric while having a rather full UNIX environment has never offered anything beyond pop3. pop3 is okay for casual users, but I like the ability to keep my mail more elegantly synchonized between the server and whichever client computer I happen to be using at the moment. I\'ve bitched about it to no avail. They probably don\'t want users to be constantly connected to a service (IMAP) on their server; so instead they deal with me constantly ssh\'ed in using pine.

Now I\'m forwarding my mail to my colo server (as well as archiving locally); my colo box runs secure imap and voila, I have email again!

Why does a sharp techie like me depend on a hosting company\'s server for his domain? Pretty simple, if I\'m wandering around mexico for a month I can hardly read my mail much less deal with software/hardware screwups on my servers. It\'s worth it to me to have someone else deal with that headache and ensure that my domain \'just works\'.

"A Nice Day in the Woods"

2004-06-15T00:00:00-07:00

Deer in the woods

I am currently staying with Mike Taht longtime friend and former roomate in his house in the Santa Cruz mountains. It's amazing to wake up to the sounds of birds chirping and to see deer wandering through the yard. It's a bit like living on Endor

Deer in the woods Forest scenery

"Cool linux wifi router"

2004-06-15T00:00:00-07:00

cool gadget or disruptive technology, you be the judge.

"The Caveman Diet"

2004-06-15T00:00:00-07:00

I once weighed 225 pounds; I had a terminal case of hacker-but. But a simple diet I developed helped me lose the weight and now I'm a much more compact 170. I've nicknamed it the 'caveman diet', because many of the principals of my diet are drawn from how we've evolved.

If you want the juicy stuff, without reading the whole deal I'm about to tell you the secret of my success.. Shhh.. It's secret!! So I'll wisper it..
diet
excersise

Eat like caveman

Cavemen didn't have bagels or pie. Cavemen hunted animals or ate what berries, veggies, and fruits they could gather. Thanks to the advent of farming and grinding of grains the human race has been able to feed many more people with much less effort; but our bodies did not evolve to handle the rapid dumping of these sugars into our bloodstream. As a result we react by releasing chemicals to store it all as fat.
Rule: Lower your consumption of simple carbs, increase protien intake

Walk like cavemen

Cavemen didn't drive SUV's. No bones about it, the convienence of modern life has taken a toll on humans. We walk as far as our driveway in the morning, sit at a desk all day long, then walk as far as the parking lot to return home, spend the rest of our evening on the couch in front of 'survivor' before passing out to do it all over again.

If a caveman was lounging around not moving, that meant there was no food to hunt or gather; the body needed to lower it's metabolism in anticipation of famine.

Rule: Avoid a sedentary lifestyle. Walk as much as possible

Run like caveman

Cavemen have fast metabolisms, why? Because they need to have rapid access to energy for extended periods of time, for extended hunting or to escape sabertooth tigers! How do they get these fast metabolism's? By running around, lifting heavy objects, climbing hills; activities us modern humans call aerobic excersise. If you can manage a run or other heart-racing activity in the morning it'll raise your metabolism all day.

Rule: Do cardiovascular excersize in the morning

Snack like caveman

Sometimes caveman feasts, like when his tribe has killed some wolly mamoth, but often he's more snacking throughout the day, gathering berries and veggies, or having an occasional squirrel.

Having a few big meals a day, your body will want to store the exccess as fat.

Rule: Favor many small meals/snacks throughout the day to a few large ones

Rule: Have a healthy breakfast that includes protein

Rage.Net Inc, worldwide corporate headquarters... A laptop, a cellphone, and a pringles-can antenna.

2004-04-11T00:00:00-07:00

Network setup with laptop and antenna

A bit of info on Google\'s network architecture. This is all junk I was doing in the late nineties. Like the way they think over there. The \'kickstart and autoconfigure on boot\' was a system I built for Mediaplex.

Good to see folks throwing brain in combination with brawn at a problem.

"A Day in the Desert"

2004-04-10T00:00:00-07:00

The Anza-Borrego Desert

I thought a good way to spend a weekend would be to visit the Anza-Borrego desert, just a few hours east of San Diego.

One of the things I love about california; you can drive two hours in any direction and be in a completely different climate, geography, and ecosystem. Observe that three days earlier I was snow skiing two hours north.

I was ready for some quiet solitude. So I packed up the car and headed east. Anza-Borrego is the largest state park in the lower 48 states. The park is one of the few places where camping is permitted anywhere and for free. The only rules are that you camp away from the roads and do not have open fires.

The unique environment demands unique adaptations from it's resident creatures. The high heat and low rainfall demand that animals and plants alike make the most of the scarce water available.

The Anza-Borrego Desert To get there I had to pass through the charred remains of the Cleveland national forrest The road twists and winds through a mountain pass to get to the desert floor I finally arrive at Yaqui Pass campground The flowering cactus are amazing! More plantlife The barren desert Another desert bloom Watch your step! Rattlesnakes are common here. This amazing fish can live in salt water or fresh, but one of its last habitats is this desert. Here, lizard lizard. Most of the mamals living in the desert are small. Here is a desert squirel The desert is a favorate place for offroading vehicles. A Black desert beetle Roadways in the desert stretch on forever The town of Borrego Springs, an oasis in the desert This taco stand is one of the most popular stops for travelers This chute growing out of the Agave plant was once a staple of the indigenous people's diet. Spotted on a hiking trail. 'Warning' what?? A small desert bird Beep! Beep! The road runner; no sign of Wile E Coyote! This ain't made for offroading!

"One parting look; see you next season!"

2004-04-07T00:00:00-07:00

Ski slope view

With the last of the now melting and the close of the mountain to skiing within days it was time for one last ski trip to close out the season.

Scott, Dustin, and I drove up the night before our mid-week adventure. We spent the evening hanging out at the Big Bear Lake 'Barn Bowl'. Darts, Beer, greasy food, and bowling. What more could you ask for? We finally made it to our hotel sometime around midnight where we met up with Lisa who'd come up earlier and was doing an afternoon / morning snowboarding outing to make it back in time for work.

Bright an early we crammed in our IHOP breakfast and hit the slopes! Just a few runs were open and conditions were inconsistant, if thats the right word for having in any given moment arbitrary combinations of slush, ice, water, and gravel. Ahh, still the best skiing I've ever done!

Dustin and I were a bit more adventurous, this being our second day ever snow skiing. We visited a variety of runs and did a few black-diamond runs.

Can't wait till next season!

Ski slope view On the slopes Skiing action Group photo Snow conditions Mountain view Last run End of season

"Power Steering Line Replacement on a Z32"

2004-04-05T00:00:00-07:00

Power steering line connection

How to replace a power steering line on a Z32 (second generation 300ZX, Twin Turbo model).

If you notice automatic transmission fluid underneath your vehicle then you might have a power steering leak. If you are lucky (like me) it'll be your power steering hose. If you are unlucky it'll be the pump itself which I'd have no clue on replacing short of pulling the engine.

Diagnosing the problem could be the biggest challenge. Diagnosis requires a mechanic's mirror along with a flashlight. The power steering pump on a Z32 is shoehorned between an alternator, engine block, turbocharger piping, and chassis frame. The best way to inspect it is from the underside, after removing the plastic belly-plate. The hose will leak around the fittings where it goes from steel line to rubber.

Once you've determined you need to replace the line, you can order it online from Courtesy (Part Number
49710-Z32TT) for about $200 with shipping.

Replacement procedure
Special tools: 24MM wrench/socket; Drip Pan
Refer to pictures.

Jack up car and secure with jackstands
Remove bellyplate
Remove two 10MM mounting brackets from below.
Remove two ducts from Intercooler.
Unplug electrical connector from P.S. line in region of steering gear.
Unbolt 24MM and 14MM line bolts from steering gear, drain fluid to drip pan.
Remove hose-clamped P.S. line from radiator area(refer to picture)
Unbolt 24MM line bolt from top of power steering pump.
Wiggle line out from PS pump and gear areas; remove old line.
Installation is reverse of removal.

The most difficult part of installation is getting those darn line washers back on. There should be one above and below the line itself; i.e. your assembly should look like: Bolt-head | Washer | PS Line | Washer | Pump/Gear housing.

After re-assembly fill the power steering resivoir with fluid; to bleed turn steering wheel repeatedly from one side to other until PS pump stops whining and fluid level no longer drops.

Power steering line connection Power steering pump location Power steering line routing Power steering line removal Power steering line installation Power steering line assembly

"The Last Gasp of the Bubble"

2004-03-10T00:00:00-08:00

MP3.com liquidation

Dustin and I broke away from our chronic under-employment to peruse the last of the great liquidations of the internet bubble. MP3.Com, headquartered in San Diego , was being scrapped.

The internet business plan...

1) Make web site


2) ???


3) Profit!

MP3.com liquidation Office equipment sale Bubble aftermath Liquidation sale Office clearance Empty offices Final sale End of an era

"Mexico's Guadalupe Canyon"

2004-03-07T00:00:00-08:00

Spent last weekend camping at Guadalupe Canyon.

Jean's Pictures

"VoIP is here!"

2004-01-31T00:00:00-08:00

I've been spending the weekend setting up Asterisk and putting it together with VoicePulse to give me a full featured virtual phone system.

"Geocaching"

2004-01-27T00:00:00-08:00

We went geocache hunting this weekend. Jean has put some pictures and information on her website...

Cold, Hard Geocaching, 2004

"Haiti 1995: Operation Uphold Democracy"

2004-01-25T00:00:00-08:00

That machine gun (a M249, SAW) is bigger than me!

Well it looks like Haiti is back in the news again, for all the wrong reasons. Lets take a ride through history, to nine years ago. Back when I was a young soldier in the US army, and images of the gulf war (the first one) were fresh in our minds. The dragon of communism, a danger for fifty years, had been vanquished and our nation's military were trying to understand their new role in a changing world.

That machine gun (a M249, SAW) is bigger than me! A map of Haiti Here's the 65th Engineer Battalion in Jacmel. We repaired three bridges in Jacmel A view of the river passing under 'Amber', the biggest of the bridge projects A bulldozer Another view of the bridge Shit burning detail. Yes, that is the technical military term. pour in gasoline, diesel, and stir while burning. Luckily some local was employed in this lowest of occupations The drive from Port Au Prince to Jacmel was a long and windy one, even though they are relatively close together A dismembered truck in one of the villages along the route Haiti, although impoverished, is very beautiful out away from the cities The industrial park that was US Army headquarters, Port Au Prince Biggs, one of the mechanics in my platoon I was on the losing end of a shaving cream fight. This is our 'home', a tent that housed a dozen of us A bus station in Port Au Price; busses are the main way haitians travel around their country This was our own 'bus', a chinook, which took a few mechanics (including me) to Cap Haitien SPC Todd Spelman, CO's driver, C Company A view of the port at Cap Haitien SPC Jeff Novatny, who worked in the signal battalion here in Cap Haitien

"thought for the day."

2004-01-21T00:00:00-08:00

McDonald's Heiress to Donate $1.5 Billion to Salvation Army

Can you name the last capitalism-bashing entrepenuer-hating ralph-nader-voting american socialist who donated $1.5 billion to charity? Didn't think so.

"Doughnut holes and differentiation"

2004-01-20T00:00:00-08:00

Well I went to the local doughnut shoppe for breakfast this morning. Same kind that's in every stip-mall across the united states. Got a ham-and-cheese croissant and a cup of coffee. With the microwaved croissant they gave me a doughnut hole. I am not a big doughnut person but I suppose the gesture struck a chord with me. It made a mental impression on me; I will always remember this doughnut place as the place with the free doughnut hole. It set itself apart from the rest of the strip-mall doughnut shops. I guess all that differentiation stuff isn't just for the fourtune 500.

"How the Weekend Shaped Up"

2004-01-19T00:00:00-08:00

Party scene

Sometimes small get togethers get outa hand. Scott really knows how to put together a party. The street filled with cars, and dozens of revelers wandered the neighborhood. The music was loud and pulsing, the drinking heavy. But it's all fun and games till it gets outa hand and the cops arrive. I suppose that's the ultimate sign of a good house party. Turning lemons to lemonade Scott took the opportunity to take some photographs.

Party scene Group photo Party guests Weekend fun Party memories

Mexico's Copper Canyon

2004-01-13T00:00:00-08:00

One of the most amazing views in the world!

One of the most magical places in Mexico is the Copper Canyon. Reachable from Los Mochis on the coast or Chihuahua inland, a train snakes it's way up into the mountains and into a region that is timeless in it's scenery and ancient in it's native cultures.

While sailing on Cassiopeia we spotted this whale in Careyes The whale dives into the deep Myself, Dustin, Anne, and Rennie overlook Tenacatita bay A catamaran garage?? One of the strange sights in Barra de Navidad The Cassiopeia crew hangs out with Al and Donna at our favorite roof-top bar. A rural train station along the Copper Canyon train route. A small town who's only access to the outside world is the Copper Canyon train. A view of the engines at the head of the train. The trip up is so steeep and twisty that this small train with less than a dozen cars requires multiple engines. Our train against a scenic backdrop. The river valley at the base of the canyon. Looking down on the river valley. We approach one of the dozens of train tunnels along the copper canyon route. As the train approaches Creel it makes a stop at Divisidero, where local vendors sell tacos to the passengers. The town of Creel, at the center of the Copper Canyon region, is overlooked by 'Cristo Rey'. Translated, Christ the king. The cemetery of Creel, where carved marble graves neighbor graves of piled stones. San Ignacio Mission in Creel, located near a Chiapas indian village. Nice Rock. The engine in Creel station, ready to carry me back to the coast. A smiling Chiapas indian girl. Another Chiapas indian girl. The canyons as viewed from Divisidero. One last view of the canyons.

A California Road Trip

2003-10-01T00:00:00-07:00

The ordinary route between the Bay Area and Los Angeles is Interstate 5. But when you've got a T-top performance car you go for the extraordanary. Sticking to breathtaking coastal routes and scenic mountain passes reminds us all why there's more than just 'economy cars' produced by the auto manufacturers.

STUFF!

Four Days on the Chesapeake

2003-07-30T00:00:00-07:00

Well I found myself out of work and was long overdue to visit Mom. I hadn t seen Mom for three years; she d been living in Sarajevo and was home for the summer long enough to study French in preparation for her move back across the pond to Paris. I, Mom, and the rental car made a point of exploring the amazing and historically rich areas around her home on the Chesapeake Bay.

Day 1: Havre De Grace, Maryland

The first day we explored nearby Havre de Grace. Apparently the town s biggest attraction is the Decoy Duck museum. Not much of a waterfowl predator myself I opted to explore the other facets of this charming small town. One small treasure I found here was the local lighthouse. The Havre de Grace Lighthouse is the oldest lighthouse in continuous operation in the United States. It was built in 1827 and has been running ever since. It was maintained by one family, the O Neill s, till 1928. John O Neill, the original lighthouse keeper, was awarded the job for his valiant defense of Havre de Grace when the British overwhelmed the town in the war of 1812.

Skipjack sailing vessels can still be seen plying the waters clamming throughout the Chesapeake here. It s the law actually, to maintain the clam s stocks only vessels under sail are allowed to clam these waters.

Day 2: Newcastle, Delaware

I visited Newcastle and only after arriving did I learn my favorite beer is from England and not this charming Delaware town! Newcastle was founded in 1651 and was a thriving port town for many years. It s where the Quaker William Penn arrived to the new world. The area to the north that Penn s Quakers colonized immortalizes him with its name of Pennsylvania. The town is a picture book example of the colonial period with its cobblestone streets, brick houses, and dark taverns. Here we toured George Read the 2nd s house (no pictures please!). George had grandiose ideas and a bit of envy of his father who was a signer of the Declaration of Independence and the Constitution. George however, after building his gaudy mansion never managed to do well at anything bigger than town politics; when he died his mansion and possessions had to be sold off to pay his debts.

Day 3, Gettysburg, Pennsylvania

Next on our tour was Gettysburg, one of the key battles in the Civil War. It was the south s first incursion into the north. The battle lasted three days through the town and across the hills to the south. The confederate forces made early gains however were stalemated and driven back, finally taking such heavy losses in attempts to rout Federal forces that they were forced to retreat to the south. The closing action of the battle was Picket s Charge , one of the most famous infantry actions in history. Twelve thousand south Carolinians charged the Federal lines in an attempt to break through. They drove into fire from rifles and cannon and when they were forced back only two thousand of them remained.

One of the most widespread weapons in the civil war was the 12-pound smoothbore field gun, model 1857 nicknamed The Napoleon . This weapon was the workhorse of civil war artillery for both sides. It was capable of firing long-range cannonballs over a mile, as well as close-combat support when loaded with grape shot, akin to a shotgun blast.

On the return to Maryland we passed through York, Pennsylvania, the site of the Harley-Davidson factory. Note to visitors, it s closed Sundays, the day we happened to stop by!

Day 4, Baltimore, Maryland

On my final day we traveled to Baltimore which is the home of Ft McHenry, and the Baltimore clipper . The clippers could average 12 knots, and even maintain 10 knots in 10-15 knots of wind! These ships were not built using plans but put together on the fly and from the memories of the shipwrights.

And as our luck would have it we were greeted by a special treat entering the Baltimore Harbor; The Libertad of Argentina dressed in flags was being guided into port. Libertad is a training ship of the Argentinean Navy. Built in 1960 and with an overall length of three hundred forty feet, it is truly a sight to behold. Libertad has the transatlantic record in a ship of its category, set in 1966 and yet to be topped. She is also the eight-time winner of the Boston Teapot trophy.

Next we visited Fort McHenry, the inspiration for the National Anthem. During the war of 1812, after the British sacked Washington (and perhaps also Havre de Grace) they planned to finish off the American militia in Baltimore. The British realized the strategic importance of the fort and so the fleet bombarded the fort repeatedly with numerous attacks. In the early hours of the morning the guns fell silent. And when the sun rose over Fort McHenry the US flag continued to fly and the British thwarted had no choice but to withdrawal.

Our final stop in Baltimore was the US Constellation. Constellation was the last sailing warship built by the US navy and is now a delightful pierside museum and does a great job illustrating the workings of such a vessel.

For example we learned what the phrase able-bodied seaman really means. An ordinary seaman has only three years of experience, whereas to be an able- bodied seaman (an official designation, with the abbreviation of A.B ) you must have six years experience!

STUFF!

All About Me

2003-07-22T00:00:00-07:00

Overlooking my Z's engine at California Speedway

Thanks for visiting my little home on the range. I hope you'll find it informative and entertaining. Feel free to send comments or suggestions to me.

After getting clear of High School I determined if I had to sit in another classroom again I would explode, and so ran off to join the US Army at 17. I was a truck mechanic stationed in Hawaii, but also served in Haiti as a part of 'Operation Uphold Democracy'. While serving in the Military I did my most rewarding work. Not fixing trucks, but delivering pizzas part-time in the evenings. True joy in work is not so much what you do but who you do it with. The rigid lifestyle of the military was not for me and when my re-enlistment came up I opted for civilian life.

I moved by to Ft Lauderdale, Florida to try going back to school and picked up a part-time job delivering pizzas to support myself. I also explored this new world that started gaining popularity when my enlistment ended in 1995, The Internet... Soon I had surpassed all the knowledge I was gaining in school by exploring the web and learning the operating systems that powered the networks. I got lucky, in that I was offered an entry-level job over IRC at a new ISP forming in Ft Lauderdale, Safari Internet. I was their first employee and both the company and I grew in leaps and bounds over the two years I worked there. However all good things come to an end and I decided that if I wanted to be where the action was I had to move out west.

Since arriving in Silicon Valley four years ago I've done a wide variety of projects for various clients through my consulting practice Raging Network Services Inc.. Rage.Net is also a vehicle for my other open source projects.

For the winter of 2001/2 I spent my time sailing my 41-ft sailboat Scirocco (pictured below) through Mexico and Central america, with a side-trip to the French Riviera for a wedding. Most of my travels over the last few years have been with Cherie Sogsti, a travel writer and closest of friends.

So now that you know the past, you want the scoop on the present?

Ohhh, letz see.. Panama, Cayman Islands, Cuba, Turks & Caicos, Puerto Rico.. Another sailing season has come and gone and I am back in California to recharge the bank account, and appreciate how great this country really is. They don't have Starbucks everywhere you know!

Overlooking my Z's engine at California Speedway Me (on the left) and friends taking Scirocco through the Panama Canal

NASCAR for the Rest of Us

2003-07-22T00:00:00-07:00

Where s the motorhome? I heard one of the organizers say. The late motorhome was only a motorhome on the outside. Inside it contained all the electronic instrumentation, audiovisual gear, and roughly one thousand pylons (red cones, for us laypeople) that would transform this airport tarmac into the local version of Daytona Speedway .

Ever hear of Autocross? While NASCAR has turned into a household name, Autocross remains largely obscure. But this little-known competition is actually the largest motor-sport in participants topping even Drag Racing. Thousands of Americans gather on their weekends, converting mall parking lots and airport tarmacs into coned race courses. They compete in everything from open-wheel racecars to family station wagons. Autocross is a safe and economical way to access the thrill of motor-sport competition.

Autocross is a timed event where a single driver and vehicle navigate a course marked out with chalk and pylons. The course contains straight-aways, slalom, S , 180-degree, and hairpin turns. Drivers compete against each other by trying to turn] the best time. Each driver has three runs to try to make the best time for their class. Drivers are divided into classes of similar vehicles to make the contest fair and competitive.

All that is required for autocross is a car, the $25 entry fee, and a helmet; and often you can borrow the helmet from the event organizers.

Autocrossing s heritage derives from the gymkhana and slalom competitions popular in Europe in the 1950 s and 1960 s. These competitions crossed the atlantic as the demand for European sports cars increased in America. The sport has evolved from it s early beginnings into the modern Autocross. Autocrossing s rise in popularity has caught the automakers attention. One automaker, Mazda has modeled it s Rev It Up national competition after the autocross format.

Autocrossing draws a wide range of people from all walks of life. Some are college students with souped up sport-compact cars who d otherwise be out street-racing fast-and-furious style; Others are retirees with their classic wheels who didn t manage to die young but still live fast ; Still others are white-collar workers and home-keepers who find autocrossing a welcome alternative to a day at the park.

At the registration table you pay your event entry fee, show a valid driver s license and sign the waiver. With little fanfare you are issued a wristband and welcomed to the world of racing.

One of the critical parts of the autocross takes place after the course has been coned and chalked. This is the walkthrough , where each autocrosser has the chance to walk the course. Since there are no practice runs in autocross competitors create their strategy at a leasurely 3MPH walking pace. Courtesy dictates that an experienced autocrosser walk the novices through the course. Where are you looking? You should be looking over there! pointing to a point behind my shoulder. The successful autocrossers are always thinking about the turn three turns ahead, not the one they are approaching. If you are thinking about the turn you are approaching you ve already passed it. In this sport things happen that fast!

Autocrossing also helps drivers develop skills they can apply to everyday driving. By competing they learn the limits of their vehicle and driving abilities in a safe environment. The result is that they are more confident and better prepared to handle unexpected situations they encounter on the public roads.

Screeeech! The green flag is waved I do my best to lauch off the starting line with as little wheel-spin as possible. The course itself takes under sixty seconds to navigate, but in this sixty seconds there are close to two dozen turns, several straightaways, and a slalom. My first run is noisy, I must have been quite a site as I screeched and skidded around the course. By the time I stop after the finish my heart is racing and the adrenaline has really kicked in.

I am handed my timeslip, I m about 5 seconds off the lead time of my class. It was quite dramatic. But any decent autocrosser will tell you noisy and dramatic is slow. The secret to autocross is to be smooth and in control.

I can t expect to conquer this sport my first time on the track. A good goal to set at the first event is to simply make each run better than the last. Which is what I did, shaving about s of a second off of each remaining run.

Almost as exciting as competing in the autocross is working the course. Part of what makes the autocross work is that the competitors are also the course workers who man the stations placed strategicly around the track. Stationed between turns with a radio and a red flag, it is these workers responsibility to reposition the pilons as they are thrown yards into the air by errant autocrossers who miss their turns. Several autocrossers are on the track at one time, spaced twenty seconds apart, giving the course workers small windows in which to replace pilons then return to safety before the next racer bears down on them. However, accidents in these events are unheard of as a professional manner is maintained by everyone. If any course worker sees an unsafe situation they can wave the red flag to bring the competition to a halt.

Autocrossing has novice classes, which allow newcomers to compete against other newcomers before graduating to the hardcore competition. The Autocross season starts in January and runs through the summer. After the sixteen events of the series the scores from each event are tallied and the top competitors from each class receive awards. There is a winter slush series run as well for those who can t wait till the start of the next season.

The Sports Car Club of America organizes autocross events across the United States. The San Francisco regional group runs events in many venues throughout the bay area including Oakland Coleseum and Candlestick Park. NASA/X also organizes events across the Bay Area over summers. For more information visit the SCCA SF Region at http://www.sfrscca.org/solo2/

STUFF!

"Going old-school!"

1969-12-31T00:00:00-08:00

Some old-school internet stuff! feeling a bit nostalgic.

Elian WHASSUP!?!
ALL YOUR BASE are belong to us
Okay, this one is just anoying

"Photo 90.jpg

1969-12-31T00:00:00-08:00