They told me to.

Pretty accurate instructions, for today…

Thu, 10 Dec 2009 12:47:25 -0600

Pretty accurate instructions, for today…

Experience with 1and1...

Fri, 28 Aug 2009 15:04:07 -0500

I recently received an invoice for a domain renewal from 1and1.com — however, I never asked for the domain to be renewed, nor had any expectation that it would be. In fact, when I first purchased the domain, I specifically was purchasing it for a single year and didn’t get the impression that my billing information would be stored on their servers.

I’ll freely admit that they sent an email about a month and a half ago that said they’d be auto-renewing the domain. However, the subject was “1&1 domain renewal notification,” and glancing at it, I read “The domains listed below are due to expire in 42 days on [date].” Since I knew I had not requested an automatic renewal I simply ignored the email so that the domain would expire.

So anyway, I called ‘em up and asked about it, and they said they could send a link to a refund/cancel form. Which of course I assumed would be an online form behind secure authentication that would let me simply take care of it all. Instead, it was a link to a PDF that I was supposed to fill out and mail or fax in. And they wanted my account information, credit card information, and a copy of a government-issued photo ID. So considering that I already lost all trust in the company due to the way they charged me for a renewal and stored my billing info without me realizing it, there’s certainly no way I would send them all that information together, especially in a completely insecure manner. (Later another customer support representative told me he doesn’t believe the copy of the ID is necessary, but the form says you have to include it.)

The second customer support representative I talked to put me on hold and transferred me to the cancellations department without asking me first — “I’ll have to transfer you to cancellations [click]”… Talking to someone in cancellations (who of course says they have no ability to deal with refunds), she actually asked me to verify my password for her over the phone. I wasn’t about to do that, but it decreased my trust of them even further.

So having hit complete dead-ends with their customer support folks, who say that I agreed in the first place to auto-renewal, and that they can’t refund the charge without me mailing/faxing that form, I gave up and decided to just cancel my account, eating the $9 for the renewal. (Side note: the first year was $7 — having it auto-renew at a higher rate is one more reason I don’t trust them.) Unfortunately, when I requested that they delete all my personal, private billing information from their servers after I cancel (because they have no right to store it without my consent), the representative told me that it’s not possible to delete such information from their system. (!!!)

The only way to cancel the account is to look up in their FAQ that you go to cancel.1and1.com to process cancellations, which I did. Of course, then, it makes you go through an email verification (sending a link in email) to verify the cancellation. However it also says that they won’t process your cancellation until you print, sign, and mail/fax another form to them (albeit a simpler one than the refund form). I have a hard time believing they stay in business, except for all the charges they receive from people who didn’t want an auto-renew, and haven’t gone through the ordeal of cancelling.

So, if you’re thinking of buying a domain from 1and1.com, make sure you read all the fine print, and make sure you really want it to auto-renew (at whatever rate they decide is appropriate for subsequent years), and be prepared for some hassle if you ever want to cancel your account.

Tab Completion for EC2 Domains

Fri, 10 Apr 2009 14:02:00 -0500

Little-known unix command: complete

At least for the bash shell, this command controls the programmable tab-completion, allowing you to configure certain programs so their tab-completion only matches files, dirs, some arbitrary set of terms, etc.

A common example is to set up “svn” so that bash knows its commands, and tab-completes them for you. A very simplistic version would be like the following:

  complete -W "add blame checkout commit copy delete diff log revert status switch update" svn

Granted, for brevity I left out a lot of commands, including their shortened versions, etc. However it turns out it’s pretty cool, because then you can type “svn u” and hit tab, and suddenly it’s got “svn update” in there for you, rather than trying to match filenames.

Of course, that brings up an interesting issue — you probably want to match a directory or filename for the second argument… And if you want to get crazy, complete lets you do that by passing it a shell function that returns possible completions for the current situation. There’s a whole package out there with specialized functions for common unix commands, which you can find at http://www.caliban.org/bash/.

However, this blog post is about abusing the tab-completion to match certain domain names. Existing EC2 instances, that is. At work we use DynDns to provide a dynamic domain name to each of our EC2 instances, and of course we often want to ssh into them by the dynamic names rather than the default Amazon ones, which look like “ec2-67-123-123-123.compute-1.amazonaws.com”. For that, first of all, we need to figure out which domain names are currently being used by instances.

That’s done with a combination of parsing IP addresses out of the data returned from ec2-describe-instances, and looking up the current IP addresses of each of our dynamic domain names. The ec2 IP parsing looks kinda’ like this (in Ruby):

  running_ips = []

  `ec2-describe-instances`.each_line { |line|

    running_ips << $1.gsub('-','.') if line =~ /\bec2-([0-9-]+)/

  }

The domain lookup is only marginally more complicated…

  domains = ("001".."010").collect { |n| "server-#{n}.example.com" }

  domain_lookup = {}

  `dig #{domains.join(' ')}`.each_line { |line|

    domain_lookup[$2] = $1 if line =~ /^\s*(server-\d+\.example\.com).+?\s+IN\s+A\s+([\d.]+)/

  }

And then we can get a list of currently running domains by pulling the relevant slice of IPs from domain_lookup, and excluding any nils:

  running_domains = domain_lookup.values_at(*running_ips).compact

Printing that out looks something like this:

  >> puts running_domains.join(' ')

  server-001.example.com server-002.example.com server-004.example.com

So … Coming back to the bash completion, we could put all that in a script and then do something of this sort:

  complete -W '$(ruby ec2_domain_lookup.rb)' ssh

But of course, that’d limit ssh’s tab-completion to only these servers, and would leave out useful things like the username, etc. Helping that, however is the fact that I’ve already got two aliases in my profile for working with ec2 boxes:

  alias ec2='ssh -i ~/.ec2/ssh-keypair -o StrictHostKeyChecking=no -o ServerAliveInterval=240 -l root'

  alias escp='scp -i ~/.ec2/ssh-keypair -o User=root'

These let me connect to an ec2 box with “ec2 hostname” or scp files to it with “escp file hostname:” (assuming you get the path to your keypair correct, etc.)

So coming back to the tab-completion, we finally get this:

  complete -W '$(ruby ec2_domain_lookup.rb)' ec2

And then since the escp command should also include directories and filenames in its completion, we use the following for it:

  complete -W '$(ruby ec2_domain_lookup.rb)' -d -f escp

So that’s the basics… The next step is to set it up so you always have those available in your shell, and don’t have to wait 10-15 seconds for all the IP/domain lookups to complete, as it generates the completion lists. So I have a cron job that periodically runs the lookup script, and outputs a file with the two complete commands in it. My .bash_login script just sources the file like so:

  . ~/.ec2_tab_completion_commands

And it’s always available without delays. Enjoy. :)

Photo

Tue, 23 Dec 2008 14:17:06 -0600

Finally!

Thu, 04 Dec 2008 20:20:18 -0600

Google Labs Gmail features are finally integrated into Google Apps for your Domain accounts! Yay! Yay!

More Backtracing

Mon, 20 Oct 2008 13:27:05 -0500

Related to the earlier post on doing backtraces, I figured out how to make some portions work on the Mac version of ruby and/or gdb, so you can do it even when you can’t easily capture stdout…

$ gdb -p
call (int)rb_eval_string(“$_old_stdout, $stdout = $stdout, File.open(‘/tmp/ruby-debug.’ + Process.pid.to_s, ‘a’); $stdout.sync = true”)
call (int)rb_eval_string(“puts caller.join(\”\\n\”)”)

Yay. Now it’ll throw the backtrace into a file in your tmp dir. I’m assuming the same would still work on linux or other unix variants, too.

"Your mind is precise and discriminating."

Sun, 17 Aug 2008 22:56:24 -0500

“Your mind is precise and discriminating.”

- Fortune cookie…

A Tale of Two Batteries

Wed, 13 Aug 2008 13:06:00 -0500

We all know that laptop batteries lose some of their capacity over time. Apple generally warrants them for a year, if I remember correctly, but after that you’re on your own if the battery barely holds a charge. But how do you get real info on your battery, other than doing time comparisons?

Luckily the Mac provides some fairly detailed info. There’s an application called coconutBattery that will display this for you, but personally I prefer to grab it from the command line (the same way coconutBattery gets its info).

I have an alias set up, called “battery”:

$ alias battery=’ioreg -w0 -l | grep Capacity | cut -d ” ” -f 17-50’

This just sets it up so when you type “battery”, it asks the mac to list off a bunch of info and looks for the lines talking about “Capacity” (and then strips off some extra whitespace).

So here are two examples, from different batteries:

$ battery
“CurrentCapacity” = 4388
“LegacyBatteryInfo” = {“Capacity”=4388,”Voltage”=12583,”Flags”=5,”Current”=4388,”Amperage”=131,”Cycle Count”=152}
“DesignCapacity” = 5500
“MaxCapacity” = 4388

$ battery
“CurrentCapacity” = 1608
“LegacyBatteryInfo” = {“Capacity”=1925,”Voltage”=12260,”Flags”=7,”Current”=1608,”Amperage”=12,”Cycle Count”=102}
“DesignCapacity” = 5500
“MaxCapacity” = 1925

Basically this shows the current capacity (charge), max capacity, and the capacity the battery was designed for. The LegacyBatteryInfo line has some other useful info, like flags that tell whether the power cord is plugged in, if the battery is currently charging, and the number of complete recharge cycles the battery has effectively been through.

What’s interesting here is to see that the first battery is still doing pretty well — it currently can store about 80% of its original charge (4388/5500). That’s after 152 cycles (and approximately a year of use). The other battery is a pathetic example, which holds 35% of what it originally could have, after only 102 cycles (and being used less than the other battery over the course of the year). Personally I’d call it a manufacturing defect, but unfortunately being a bit more than a year old, Apple won’t take responsibility for it.

Mallards Game

Fri, 01 Aug 2008 21:51:13 -0500

Mallards Game

Photos of the LHC

Fri, 01 Aug 2008 15:24:00 -0500

Photos of the LHC:

via http://simonwillison.net/2008/Aug/1/lhc/

Sending specific traffic over a VPN connection on OSX

Fri, 01 Aug 2008 13:53:28 -0500

For a while I’ve had to go back and forth between using the “Send all traffic over VPN connection” setting and not using it. Generally I want the setting off, so that all my network traffic doesn’t get slowed down as it bounces through extra servers. However, sometimes I need to access computers in the office that aren’t on the default VPN subnet.

Recently I found this link: http://blog.liip.ch/archive/2006/01/07/changing-default-routes-on-os-x-on-vpn.html which describes how to tweak the routing settings. However, it’s a bit out of date; as of that writing, any time you used the built-in VPN setup, all traffic would go over the connection.

In any case, let’s say you have a VPN where the default subnet is 192.168.2.x but some servers are on 192.168.3.x addresses. When you connect to the VPN, requests to 192.168.2.x addresses work just fine, but trying to access the others ends up trying to send the requests over the Internet, which of course won’t work.

To fix this, create a file called /etc/ppp/ip-up and put the following in it:

#!/bin/sh
/sbin/route -n add -net 192.168.3 $IPREMOTE >> /tmp/ppp.log 2>&1

You’ll need to create the file as root, and then set it executable. Next time you connect to your VPN, the system will automatically run that file, having set the IPREMOTE environment variable to the IP address of the VPN gateway server. It will make a log of these changes in /tmp/ppp.log for future reference.

This tells your computer to always try routing 192.168.3.x addresses through that gateway. And you can stop sending *all* your network connections through the VPN. :)

When you disconnect from the VPN, it automatically removes the route, so you shouldn’t need to do any other cleanup.

"top" for filesystem usage...

Thu, 31 Jul 2008 16:46:55 -0500

Does your mac sometimes slow down to a crawl due to hard drive usage, and you can’t tell what’s causing it?

I whipped up a little ruby script to act like “top” for filesystem usage.

#!/usr/bin/env ruby

data_read = ''
process_times = Hash.new(0.0)

# Clear screen
print 27.chr + '[2J'

# Open a pipe to fs_usage for file calls.
data = IO.popen('fs_usage -wf filesys')

while true

  # Keep responsive to fs_usage output by reading 10 times per second.
  10.times do
    sleep 0.1

    data_read << data.read_nonblock(1048576) rescue ''
    lines = data_read.split("\n",-1)
    data_read = lines.pop.to_s

    lines.each do |line|
      elapsed,process = line[138..148],line[152..-1]
      process_times[process] += elapsed.to_f
    end
  end
  
  # Jump up to the top of the screen and print the header.
  print 27.chr + '[f'
  puts "PROCESS".ljust(30) + ' ' + 'TIME'
  puts "------------------------------ ---------"

  # Print in order of which used the most time.
  process_times.sort { |a,b| b[1] <=> a[1] }[0..19].each do |name,time|
    puts name[0,30].ljust(30) + ' ' + sprintf('%0.6f',time).rjust(9,'0')
  end

  # Clear the times for the next loop.
  process_times.each_key { |k| process_times[k] = 0.0 }

end

However, it uses “fs_usage” to get the data, which has to be run as root, so:

$ sudo ruby fs_top.rb

PROCESS                        TIME
------------------------------ ---------
ruby                           00.000542
pppd                           00.000145
firefox-bin                    00.000000
iTunes                         00.000000
update                         00.000000
cron                           00.000000
thunderbird-bin                00.000000
notifyd                        00.000000
SystemUIServer                 00.000000

Then just hit ctrl+c to quit. Enjoy.

I'm just sayin'...

Mon, 28 Jul 2008 21:05:37 -0500

Cheetahs Never Lose: 8
Ring of Fire: 9

http://www.sandlotsports.biz/leagues/teamview.asp?teamId=1254&leagueid=144

Ruby 1.8.5, REXML, and You

Mon, 28 Jul 2008 09:43:47 -0500

In my previous post, I mentioned that I was trying to track down why our mongrels (running Rails) at work were spinning out of control for no apparent reason. However, finally being able to generate a backtrace from those mongrels led right to the issue.

The relevant portion of the backtrace looked like this:

from /usr/lib/ruby/1.8/rexml/encoding.rb:59:in `check_encoding’
from /usr/lib/ruby/1.8/rexml/source.rb:40:in `initialize’
…
from /usr/lib/ruby/1.8/rexml/document.rb:45:in `initialize’
…
from ./current/config/../vendor/rails/actionpack/lib/action_controller/cgi_ext/cgi_methods.rb:53:in `parse_formatted_request_parameters’

So apparently REXML was the culprit, and it was happening while trying to parse the parameters that were passed to our Rails app. Before any of our (non-framework) code was even called. From some of my investigation, I knew some of the spinning mongrels corresponded to requests with 200k or more of POST data, so this made some sense. But other requests with that much data went through just fine.

In any case, tracking down that line revealed this code, as part of what REXML uses to detect the encoding of the XML data:

str =~ /^\s* return $1.upcase if $1

And, well, that’s just wrong. The non-escaped question mark, the two “\2” backreferences, the “return $1” instead of “return $3”, etc. However, our servers are currently running on Ruby 1.8.5. So I took a look at the corresponding code in the version of REXML that comes with Ruby 1.8.6, and found that it’s fixed:

str =~ /^\s*<\?xml\s+version\s*=\s*([‘”]).*?\1\s+encoding\s*=\s*([“’])(.*?)\2/um
return $3.upcase if $3

So it just happens that the sequences of data we were receiving in the rails app were playing unkindly with the broken regular expression, going exponentially out of control with backreferences.

Changing the old REXML code to the new code brought encoding-detection time down to 52 microseconds instead of 2 *hours* on some of the test data. So, we fixed that on our servers, and the problem is gone.

Another solution: Upgrade to Ruby 1.8.6 or better. :)

Backtraces from Live Ruby Processes

Thu, 24 Jul 2008 16:19:00 -0500

We had some out-of-control mongrel processes at work recently… They wouldn’t respond to anything except a “kill -9”, were taking 100% cpu time, and spinning for hours before responding to any new requests.

Unfortunately we had no idea what could be causing it, either. Loading up gdb and printing out a backtrace only gives a bunch of “rb_call” type entries, the C-level code rather than anything at the ruby level. The obvious solution is to go ask Google how to get a backtrace.

Presumed answers:

http://weblog.jamisbuck.org/2006/9/22/inspecting-a-live-ruby-process and http://eigenclass.org/hiki.rb?ruby+live+process+introspection

However, they must have been working with a different version of Ruby, because that doesn’t work on my local Mac or the Linux servers; I just get errors about accessing invalid memory. So after a few frustrating hours I downloaded the source code for Ruby 1.8.6, and found rb_backtrace(). It’s a function that takes no arguments, and prints a backtrace to stdout rather than trying to return an array (which would be semi-difficult to interpret in gdb).

So …

$ gdb -p

(gdb) call (int)rb_backtrace()

Forces ruby to print a backtrace out to the process’ stdout. In our case, that’s a daemontools log file, so I could finally track down what was making the mongrel spin forever!

There’s probably still a great way to do this within gdb, so you don’t have to go find the stdout, or possibly get debug data where it shouldn’t be. But it’s a starting point.