Versioning RESTful APIs

There seems to be general (but definitely not complete) consensus among REST experts that the proper way to version a REST API is via custom MIME/media types.  You can Google this for yourself to find the discussions on the rest-discuss mailing list, Stack Overflow, various blogs, etc.

There was one subtlety to the discussion that I found interesting and I spent some time researching.  Most folks propose media types like:

"application/vnd.mycompany.myapp-v2+xml"

However a few folks, based on examples in the section of the HTTP spec for the Accept header, suggest a format like:

"application/vnd.mycompany.myapp+xml; level=2"

That's often countered with the argument that an "accept-extension" like "level" is optional (with the implication that it can be ignored if present, which seems like an unfounded assumption), or that it should only be used to indicate minor versions.

However my reading of the examples in the HTTP spec and related historical documents leads me to believe that using "level" to indicate major versions was exactly the intended purpose. All usage of the "level" extension historically seems to have been in relation to the “text/html” MIME type, and specifically to indicate major versions of the HTML spec (HTML 2, HTML 3, etc.) This page (http://www.w3.org/MarkUp/table-deployment) dated 1995 clearly shows usage of a "text/html; level=3" media type to indicate to HTML 2 only clients that they won’t understand the markup and should not attempt to do so.

That said, this is largely an academic exercise. In the absence of specific definition it seems like media types must be treated as opaque strings, so "application/vnd.mycompany.myapp-v2+xml" and "application/vnd.mycompany.myapp+xml; level=2" both serve the same purpose. Since there seems to be consensus that the former is the better format to use I’ll be going that way, I just don’t see any reason that the later format is any less valid. And in fact using the "level" extension for minor versioning seems counter to historical precedent, but (IMHO) acceptable and harmless if used on a custom MIME time.

Ruby, OptionParser, and Switches with Optional Arguments

Ruby's OptionParser supports switches with optional arguments. Both the official documentation and the pickaxe book give examples of various syntax choices like these:

"--switch [OPTIONAL]"
"--switch [=OPTIONAL]"
"--switch" "=[OPTIONAL]"

Somewhere along the way the first time I needed a switch with an optional argument I copied the third syntax, where the argument is specified as a separate string.

Later I noticed odd behavior. While my switches with required arguments could be used with or without an equals sign, my switches with optional arguments required an equals sign. I.e. "--switch foo" did not work for the switches with optional arguments, while "--switch=foo" did work.

I've lived with this oddity for several years. But then recently I picked up a copy of the new "Build Awesome Command-Line Applications in Ruby" book and was giving it a read. In their section about OptionParser they only listed the first syntax (all one string, no equals sign). That got me wondering if the
equals sign made a difference.

I went back and re-read the OptionParser section of the pickaxe book and noticed an interesting statement differentiating "--switch=ARG" from "--switch ARG". Specifically they say the later syntax "specifies the argument follows the switch."

Hmm, that sounds promising. Write a quick test. Yes! "--switch [ARG]" makes the argument optional, and with or without the equals sign. I should have stopped there, but now I was curious. So I tried "--switch=[ARG]", and oddly it works the same way. Optional argument, optional equals sign. Hmm, that's odd. Try the "--switch" "=[ARG]" syntax again just to make sure I'm not going crazy. Nope, not crazy. Argument is optional, but equals sign is not.

So, a bit more pondering and I come up with 9 possible syntaxes and set about to try them all:

"--switch [OPTIONAL]"
"--switch [=OPTIONAL]"
"--switch =[OPTIONAL]"
"--switch" "=[OPTIONAL]"
"--switch" "[=OPTIONAL]"
"--switch" "[OPTIONAL]"
"--switch[OPTIONAL]"
"--switch[=OPTIONAL]"
"--switch=[OPTIONAL]"

Interesting, all 9 are valid syntax for OptionParser. 8 work in terms of the argument being optional, "--switch =[OPTIONAL]" being the odd one where the argument is required despite the square brackets (tested with both ruby 1.8.7 and 1.9.2).

And finally, drumroll please, the distinction that actually makes the equals sign required is when you specify the argument as a separate string. The presence or position of the equals sign when specifying the switch to OptionParser makes no difference, except as noted with the "--switch =[OPTIONAL]" syntax. "--switch [=OPTIONAL]"? No equals sign required. "--switch" "[OPTIONAL]"? Equals sign required.  Wacky.

+1 to the Pickaxe book for prompting me to ponder the different syntaxes, but -1 for being somewhere between misleading and wrong in actually documenting the difference.

FIOS Router CLI and DHCP


I recently got Verizon FIOS service.  The router Verizon provides is an Actiontec MI424WR.  Verizon goes to some trouble to hide the Actiontec branding; but, for example, the Model Name field in the System Monitoring tab on mine says "MI424WR-GEN2".

Anyway, I wanted to get the DHCP lease database out of the router in order to populate that information into my own DNS server.  I initially looked at scraping the web interface, but the router does some weird Javascript hashing of the password before submitting it to the server.  I figured there had to be a better way.

In the advanced options you can turn on telnet access to the router.  That gets you into a command line administration utility, which in turn will let you access a shell (via "system shell").  However I was unable to find a DHCP lease database in the filesystem.  There was a /var/state/dhcp directory, but it was empty.

Some searching on the web turned up the fact that the Actiontec MI424WR runs a commercial Linux distro called OpenRG.  More searching turned up an OpenRG Configuration Entries Guide.  A bit more poking and I found that this command will dump out the DHCP lease database:

conf print dev/br0/dhcps/lease

Python Packages

I'm new to the Python and Django world, having primarily worked with Ruby and Ruby on Rails for the last few years.  In reading about how to manage dependencies several folks referred to pip and distribute as the new hotness (literally).  Ok, fine, they're probably something like ruby gems, but what's the difference?

Looking over their respective pages (http://pypi.python.org/pypi/pip and http://pypi.python.org/pypi/distribute) it is confusing to see that there appears to be some sort of circular dependency.  The pip page says "In order to use pip, you must first install setuptools or distribute."  The distribute page gives instructions for installing distribute via pip.  Wha?!

The heading at the top of the pip page says "pip installs packages. Python packages."  The heading of the distribute page says "Easily download, build, install, upgrade, and uninstall Python packages".  Argh, still confused.

Pip is more clearly documented, and serves in the roll of package downloader, dependency resolver and installer.  This would be analogous to yum or apt.  I.e. pip does not create packages.  Pip apparently replaces something called easy_install.

So if you're just using existing packages you'll just use pip, possibly in combination with virtualenv for isolation (similar to bundler in the Ruby world).

So what about distribute?  The distribute documentation is terribly confusing.  It says it replaces setuptools, but then seems to imply that it is setuptools.  It looks like distribute is a fork of setuptools because the setuptools maintainer stopped maintenance.  And I guess in parallel there's another tool called distutils.  These all are used to make packages.  As is usual for package making there's a metadata file (setup.py in this case) and setuptools/distribute/distutils take that and a directory full of files and makes a package.

And apparently because setuptools, distribute and distutils all have problems there's work underway on distutils2 to replace them all.

On a side note, you'll also find references to eggs.  This is apparently a packaging format introduced by setuptools.  It seems to have lost favor.  The PyPI repository distributes packages as generic tarballs.

So, in summary if you're making a package use distribute or distutils, and if you're installing a package use pip.  I think.  :)

Other references:
http://guide.python-distribute.org/
http://packages.python.org/distribute/

Upgraded to Mac OS 10.7 (Lion)?  Tried to update Macports with sudo port selfupdate and get:

Error: /opt/local/bin/port: port selfupdate failed: Error installing new MacPorts base: shell command failed (see log for details)


You probably still have the Snow Leopard version of Xcode installed.  Try to run Xcode, it will probably tell you that you need to upgrade.  Go delete the Install Xcode app from your Applications folder, fire up the App Store app and get Xcode again.  That will get you the installer for Lion, which you can then run to upgrade Xcode to the Lion compatible version.  Then the Macport update should work.

I'm working on a MacRuby project in Xcode.  I upgraded Xcode from 4.0 to 4.0.1 and Interface Builder stopped working right for my project.  New outlets weren't showing up so I couldn't connect them to UI elements.  I eventually realized that in the Xcode upgrade the MacRuby templates got wiped out.  A reinstall of MacRuby got things working again.

Filesystem Hierarchy Standard for Mac OS X

I frequently refer to the Filesystem Hierarchy Standard when trying to figure out where to put files on Linux systems.

I recently started doing some Mac development and was looking for an equivalent document. Apple's File System Overview seems to cover similar ground.

User-Friendly RESTful Routes in Rails

RESTful routes created via map.resources use the :id field for URLs for individual resources.  I.e. if user 'joebob' has an id of 42 then the URL for his resource is http://example.com/users/42

I find those URLs rather user-unfriendly.  Wouldn't http://example.com/users/joebob be nicer?

If you plan for this from the start you can fake out rails by using a non-standard id column.  I.e.

    create_table :users, :id => false do |t|
      t.string :id, :null => false
    end
    add_index :users, :id, :unique => true

Judging by blog posts there are at least a few folks out there who have taken this approach.  Here are a couple that point out issues you might run into if your user names (or whatever identifier you use for id) contain special characters:  here and here

But what if you already have a project up and running and don't want to rearrange your database?  Or just think that faking out Rails that way is kinda ugly?

It is possible you can do this somehow with map.resources and its :requirements parameter, but I haven't been able to figure out a syntax that works.  It seems that map.resources really expects to map the identifier to the id column.

So for now I'm settling with a partial solution.  In routes.rb:

  map.connect 'users/:name', :controller => 'users', :action => 'show', :name => /[a-z]+[a-z0-9]*/
  map.resources :users

The trick with the connect route is that the regex for name must not match ids.  I.e. you want /users/joebob to use the connect route, but /users/42 to use the RESTful route.  Of course if your usernames are alpha only this is easier.  Note that Rails does not allow anchor characters in the regex, so a regex like /^[a-z][a-z0-9]*/ doesn't work.

OK, but if you have a standard RESTful controller the first line of the show method is going to look like:

    @user = User.find(params[:id]

But that method is going to get passed params[:name] for calls using the connect route.  So you'll need to replace that line with something like:

    @user = nil
    if params[:id]
      @user = User.find(params[:id])
    elsif params[:name]
      @user = User.find(:first, :conditions => ['name = ?', params[:name]])
    end

At this point at least the basics are working, but I'd welcome feedback and suggestions for improving this.

Update:  See http://railscasts.com/episodes/314-pretty-urls-with-friendlyid

I needed to set the UID for an account on my Mac OS X 10.6 (Snow Leopard) box. Various posts online said that you can right click on an account in System Preferences -> Accounts to access the Advanced Options. But right clicking wasn't doing anything for me. Finally I tried a Ctrl-click, the standard way to "right click" when you only have a one button mouse, and that worked. My right button works everywhere else, but for some reason in this instance it didn't.

If your Ruby app using https or some other form of SSL is complaining about:

net/http.rb:567: warning: can't set verify locations

Then you probably passed in a bogus path to your CA file or path. In my case I had:

http.ca_file = '/etc/pki/tls/certs/ca-bundle.crt'
http.verify_mode = OpenSSL::SSL::VERIFY_PEER

That path is right on Red Hat, but I was running the script on my Mac laptop. Whoops. Fixed the path to something valid and the error went away.

I've been using the workaround shown at http://blog.zenspider.com/2008/05/httpsssl-warning-cleanup.html for a while now to eliminate the "using default DH parameters" warning you get when using https in Ruby. However, I recently tried to reference two of my libraries which contained the same workaround in a script and ran into a "stack level too deep" error as the two sets of code both tried to alias the use_ssl= method. I spent some time working on this today and I think I've come up with a better method.

Net::HTTP.ssl_context_accessor(:tmp_dh_callback)
http = Net::HTTP.new('www.example.com', 443)
http.tmp_dh_callback = proc {
  OpenSSL::PKey::DH.new(IO.read('dhparams')) }
http.use_ssl = true

There are a few options for how you create the Diffie-Hellman params. The code above assumes you've got a file with pre-generated params (this is perfectly acceptable from a security standpoint). You can create the file with the "openssl dhparam" command, or a little Ruby script. You could also generate the DH params on the fly, although this can be quite slow:

Net::HTTP.ssl_context_accessor(:tmp_dh_callback)
http = Net::HTTP.new('www.example.com', 443)
http.tmp_dh_callback = proc { OpenSSL::PKey::DH.new(2048) }
http.use_ssl = true

Or if you just want the warning to go away and aren't concerned about the security implications you can just use as empty proc as zenspider did. I've tried to look at the Ruby OpenSSL library source and figure out what happens when the proc doesn't return anything but the code is a bit too obtuse for me to figure out.

Net::HTTP.ssl_context_accessor(:tmp_dh_callback)
http = Net::HTTP.new('www.example.com', 443)
http.tmp_dh_callback = proc {}
http.use_ssl = true

Excel on the Mac defaults to portrait mode. Nearly all spreadsheets I create I want in landscape and finally tracked down how to set this as the default.

• Open up a new document
• Set it to landscape via File -> Page Setup
• Go to File -> Save As
• Switch to the /Applications/Microsoft Office 2008/Office/Startup/Excel folder
• Set the filename to Workbook
• Uncheck the Append file extension checkbox
• Save
• Quit and restart Excel

Your workbooks should now be landscape by default.

You can find this in the Excel help. Don't use the Help menu item on the main Mac menu bar, Excel's help isn't available there. Instead click the Help button on the toolbar in an Excel document to bring up the Excel help system and look for the "Control how workbooks and sheets are created" document, available under Contents -> Customizing Excel.

How to insert a new line above the current line in TextMate? A la capital O in vi? I couldn't find it in the TextMate book, nor searching around online. A few seconds of fiddling on the keyboard turned it up though: command-option-return

This makes sense, as command-return inserts a new line below the current line. Having now looked around some more this command-option-return key sequence is documented as "move to end of line and insert end-of-line symbol", e.g. a semicolon in C or Perl. Conveniently, in Ruby and presumably other languages with no end-of-line symbol it has been redefined to the "new line above current line behavior". It's disappointing that there isn't a universal shortcut.

Update: SourceForge no longer offers the Wikispaces wiki, this post is left up for historical curiosity.

The default wiki for SourceForge projects is used to be called Wikispaces. The CSS stylesheet configuration on the SourceForge Wikispaces is such that list items (HTML "li") are smaller than normal, and shrink as they are nested. Top level list items are smaller than I would care for, and by three or four levels of nesting they are unreadable. This has bugged me for a while, but today I was trying to create a page which had up to five or so levels of nesting (to represent a directory tree) and decided to see if I could fix this.

Using the wonderful Web Developer extension for Firefox I was able to pinpoint the source of this shrinkage, this line of http://static.sourceforge.net/css/sfx.php?secure=0&20080417-1657:

#frame li, #fadbtm li { font-size: 82.3%; }

Wikispaces uses about five CSS stylesheets including sfx.php, and they allow you to edit one of the other CSS stylesheets under Manage Space -> Look and Feel -> Edit your wiki stylesheet.

When editing that stylesheet Wikispaces restricts you to applying styles to the "wiki" class. Your wiki content is placed in an HTML div of class "wiki" and id "content_view". So I tried to add a line to the stylesheet which sets the font size for list items in class "wiki":

.wiki li { font-size: 100% }

However, there is a higher level div with id "frame" which the Wikispaces-provided CSS line applies to. Per the CSS specification styles applied to a specific id (#frame) take precedence over styles applied to a class (.wiki). As such the style for the id "frame" overrides my style for the "wiki" class.

Another read through the CSS spec reveals the important flag to override the normal priorities. And that did the trick, adding the following line eliminates the shrinking list items:

.wiki li { font-size: 100% ! important }

With the Oracle client 10.2.0.1.0 installer if you are automating the install with a response file, set UNIX_GROUP_NAME to something like say 'dba', then run the install but you are not in the dba group the install will fail with:

SEVERE:S_OWNER_SYSTEM_EPERM

I found nothing useful via a search for that error so I thought I'd post something. It would seem that you need to set UNIX_GROUP_NAME to a group you are in. Oddly none of the files that are written out end up owned by that group, so it doesn't seem like the installer really needs those privileges.

I'm trying to package the Oracle client for our internal packaging system, I'm not a DBA and wouldn't normally be a member of the dba group.

I was working on building the mysql ruby gem. My mysql install is in a non-standard location. I could get the gem to build with syntax like:

gem install mysql -- --with-mysql-config=/path/to/bin/mysql_config

However, I noticed that the resulting library did not embed the path to the mysql library, ldd on the mysql.so indicated:

libmysqlclient.so.16 => not found

I did not want to have to fiddle with setting an LD_LIBRARY_PATH at runtime. The way to address this is to pass -R or -rpath to the linker at compile time to embed the path to the library. The mysql gem uses the standard extconf.rb to build a Makefile to build the library. extconf.rb uses the standard Ruby mkmf library to do the heavy lifting. Thanks to the woeful lack of documentation for mkmf it took me several hours to figure out how to convince mkmf to include the proper option in the compiler flags. The answer is:

gem install mysql -- --with-mysql-config=/path/to/bin/mysql_config --with-dldflags="-Wl,-rpath,/path/to/lib/mysql"

--with-dldflags appends anything you specify to the DLDFLAGS variable in the generated Makefile. FYI, --with-ldflags overwrites DLDFLAGS with what you specify, possibly wiping out other linker flags you want to keep. Also FYI, the -Wl,-rpath,etc. syntax is required to pass the linker option through the compiler.

ldd now shows that the path is embeded in the resulting mysql.so:

libmysqlclient.so.16 => /path/to/lib/mysql/libmysqlclient.so.16 (0x00002ad5e81d1000)

Speaking of Perl, Ruby and sorting, sort in both Perl and Ruby allow you to specify a code block to implement your own arbitrary sorting routine. Perl's documentation shows you how you can define this as a subroutine and just reference the subroutine when you call sort. This is handy both for re-use and when your sort routine is complex and doesn't easily fit on one line. Ruby's sort documentation? Eh, no mention of such a thing.

In fact it is possible, but delves into a whole area of Ruby that is somewhat under-documented. The trick is that sort wants a code block, but code blocks are not first class entities. You can't name and reference a code block. However, Ruby has a couple of similar entities, Proc and lambda, that can be converted to/from code blocks and can be named and referenced. The unary ampersand operator will convert a Proc or lambda to a code block. The syntax works like this:

names = [
  {:fname => 'John', :lname => 'Smith'},
  {:fname => 'Jane', :lname => 'Smith'} ]

sorter = lambda do |a,b|
  [a[:lname], a[:fname]] <=> [b[:lname], b[:fname]]
end

names.sort(&sorter)

When sorting complex structures in Perl is is common to use syntax like:

sort { $lname{$a} <=> $lname{$b} || $fname{$a} <=> $fname{$b} } @names

The Perl sort documentation lists examples with this syntax. True to form the Ruby sort documentation doesn't show examples of anything that complex. Best I can tell anything like this doesn't work in Ruby:

a <=> b || c <=> d

Any way I write that I get back the result of the first comparison, even if the result is zero (which should trigger a fall-through to the second comparison).

Since Ruby's Array implements <=> you can achieve the desired result with:

[a, c] <=> [b, d]

A couple of MacPorts errors I just encountered:

Error: Unable to execute port: can't read "frameworks_dir": no such variable

I had version 1.6 and it seems the recommendation to fix this is to update to 1.7 (via port selfupdate). I started port in interactive mode, ran selfupdate, then tried my install again and got:

Error: Unable to open port: can't read "porturl": no such variable

I wasn't sure if it was necessary, so I ran 'sync' at this point. The install then failed with:

Error: Unable to execute port: can't read "PortInfo(name)": no such element in array

Couldn't find any info about that online, so on a lark I exited port and restarted it, after which the install proceeded.

It's a mystery to me why the FreeBSD folks don't publish this better (for example, a mention on http://www.freebsd.org/security/ doesn't seem unreasonable), but FreeBSD has an easy to use system for updating the base system:

# freebsd-update fetch

Review the changes to be applied, then

# freebsd-update install

If there's a new kernel in there then you should reboot. That's it. Not quite sure when this was fully integrated into the base OS, I'm only an occasional FreeBSD user, you used to have to install a few ports to get this working.

Even their official security advisories don't mention this, instead referring to PITA methods of patching and building from source. I installed the OS via pre-built binaries, why would I then update by patching the source? Sigh.