Skip to content

Announcing Resourceful

Resourceful is an advanced http library for Ruby that does all the fancy stuff that makes HTTP an amazing protocol. I’m pleased to announce the initial release of Resourceful, 0.2. It already has some pretty cool features, with more to come.

This library is intended to make it easier for you to write your next whiz-bang Web2.0 app by performing the next level of HTTP features for you. There’s some pretty nice stuff in the HTTP1.1 spec, but so far (at least in Ruby), everyone has has to roll their own. There has been some amazing stuff done on the server side of the HTTP spec in ruby, like mongrel, thin and rack, but the client side has been stuck with Net::HTTP for too long. We hope to remedy that.

Basic Example

Here’s how you perform a very simple HTTP GET request:

require 'resourceful'
http = Resourceful::HttpAccessor.new
resp = http.resource('http://rubyforge.org').get
puts resp.body

Yeah, yeah, big deal, right? Every Yet-Another HTTP Library can do that. What makes Resourceful different is the additional features we added on.

Features that should make you want to use it

I plan to write some full-length articles about these features in the future, to show how we’re using them. For now, a brief description will have to suffice:

  • Redirection callbacks - GET requests automatically follow redirects, PUT, POST and DELETE do not. All allow callbacks to be set, that get called upon redirection. Should the callback return false, the redirection will not be followed. This will allow you to, for example, notify a local storage mechanism to update any links you might be storing with the new location.

  • Pluggable Authentication modules - Basic is built in, as is a very simple Digest one (but it’s probably too simple to be really useful at this point. However, its very easy to roll your own, it only has to provide a couple of methods, and be registered with the @accessor.

  • Support for HTTP Caching - Most of the important parts of HTTP Caching, like storage, expiration, and validation are all handled for you. This is a simple in-memory store for the cached documents, but this is easily extensible. Some possible caching backends are a database, disk store, or memcached.

Update: My cohort on this also made a blog post about Resourceful.

Tagged ,
2008 06 30 Paul Sadauskas Resourceful
Ruby Comments (3) Permalink

We’re looking for a Ruby Developer

We’re looking for another Ruby & Web developer at my company, Absolute Performance. Here’s the job description. Email me and/or my boss if you’re interested, and want to work in beautiful Boulder, CO, at an awesome company.

Ruby and Web Developer

Description

We are looking for a Ruby & Rails developer to work primarily on our flagship product, System Shepherd.

Responsibilities

  • Create and maintain web management tools for our product
  • Work with a development team, and designers and managers, as needed
  • Modify and integrate existing open-source tools
  • Must be able to work on-site at our office in Boulder, CO

Requirements

You must have experience with:

  • Ruby
  • Rails Deployment
  • ReST & Web Services
  • Linux
  • Git & Subversion
  • MySQL & PostgreSQL
  • Ajax & Javascript
  • HTML, CSS

Bonus points for experience with:

  • Merb/Datamapper
  • Shell scripting
  • System Administration (Apache, monit, xmpp)
  • Agile/Scrum development practices
  • Java/J2EE & Application Server deployment
  • C++ and modern coding standards/libraries (STL, Boost, etc)

Other requirements:

  • Ability to create, read and follow written specifications
  • Strong analytical and creative problem solving skills
  • Ability to work self-directed in a fast-paced environment with minimal supervision
  • Strong work ethic, reliable and detail-oriented
  • Excellent written and oral communication skills
  • Ability to multi-task, prioritize work and meet deadlines
  • Flexibility and a desire to take on responsibilities
  • Ability to work in a team environment
2008 05 23 Paul Sadauskas Ruby Comments (0) Permalink

Spec’ing Migrations (A Tutorial)

I realized I haven’t blogged about (IMHO) the neatest feature of DataMapper’s migrations yet. One of the more harrowing experiences for me in Rails is upgrading a production server with live data, and hoping your migration handles all the existing data correctly. You can dump the database, and attempt the migration on a clone, and hand-examine the data to make sure it was correct, but that feels very non-ruby-like to me. With the spec groups and matchers available in DataMapper migrations, though, you can spec your migrations and be certain that it will work correctly, and translate all your edge-case data correctly.

Lets start off with a simple example. We’ll start backwards for this example, but in real life, you’re probably better off writing the spec first, then the migration itself, as is normal in iterative development. But regardless, lets say we have the following simple migration we want to spec:

migration 1, :create_people_table do
  up do
    create_table :people do
      column :id,     "integer"
      column :name,   "varchar(255)"
      column :age,    "integer"
    end
  end
end

Note that I’ve used string as the column types here. I hope that one day migrations will support dm-types, but until those stabilize post-0.9, I’m not going to try to implement it. Anyways, this just creates a pretty typical people table.

Now lets start writing the spec:

describe :create_people_table, :type => :migration do
 
  before do
    run_migration
  end
 
end

Just some boilerplate here. You put the migration name as the name of the describe block, and pass the additional option of :type => :migration. This informs rspec to run the migration group-specific routines as part of this spec. After that, we have a before block. You must specify the run_migration at the end of this block. This allows you to insert any data you want before the migration gets run, so that you can test it got migrated correctly. The way these work is that for every describe block, the database is dropped and recreated, then the migrations run up_to the migration specified. Then the before block is executed, the migration is performed, then the examples are executed. There are some subtle differences between the various DO adapters as to how that all works, but the results are the same.

Alright, lets look at an example now:

it 'should create a people table' do
  repository(:default).should have_table(:people)
end

Pretty self explanatory, right? Here we use the #have_table matcher to check that we do, in fact, have a table called people.

How about a more complicated one?

it 'should have an id column as the primary key' do
  table(:people).should have_column(:id)
  table(:people).column(:id).type.should == 'integer'
  #table(:people).column(:id).should be_primary_key
end

Ah, some meat in this one. Pretty obvious what it does, too. First we check that the people table has a column called id. Then we look to see that the column’s type is ‘integer’. (I plan on writing better matchers for these. Right now you have to string-match the column type of your RDBMS. Someday, you will be able to do column(:id).should have_type(:integer) or column(:id).type.should be_integer. I haven’t decided which I like better.) Finally, we check that the column is a primary key. (This matcher hasn’t been written yet, either. Feel free to contribute patches to any of this.)

And that’s really all there is to it. Take a look at the [sample migration spec]: http://github.com/sam/dm-more/tree/master/dm-migrations/examples/sample_migration_spec.rb to see the whole thing with more examples.

Some things to be aware of

  • The matchers don’t work at all for MySQL. If anyone wants to contribute, please feel free. Take a look at the postgres & sqlite files under lib/sql/ to see how it should work.
  • I couldn’t find a way to drop/create a database while inside a DO adapter connection. Rather than trying, I just drop/recreate the ‘test’ schema inside the database specified in the adapter.
  • In Sqlite, I just delete the file, and let the adapter re-create it on its own.
  • Postgres is the best-tested, and the most feature-complete. Everything in the examples works on both pg and sqlite. The spec can be initialized in run in mysql, but none of the matchers have been written yet.
Tagged , , ,
2008 05 22 Paul Sadauskas Datamapper
Ruby Comments (0) Permalink

Moving on from Gentoo…

Updated: I added a few more paragraphs that I had originally commented on in response to Mr. Berkholz’s comment.

I recently saw the announcement of the ‘fork’ of Gentoo, Exherbo. Its not really a fork, because there isn’t any shared stuff. The package manager used is Paludis, one of the alternatives to Portage in Gentoo. Several of the developers on Exherbo are from Gentoo, and they profess similar goals. I’m a bit wary, though, because their webpage is pretty dickish. Thats fine, Gentoo never claimed to be a everyman’s distro, but I kind of wish they’d waited to announce their project until after they wanted people to start using it.

I shouldn’t be surprised, though, since Gentoo dev and author of Paludis, Ciaran McCreesh, is one of the Exherbo devs. I once tried to use Paludis about a year ago, when portage was in one of its broken states ( one of the reasons I’m moving on, more on that later…) I read the install page for Paludis, got it installed, but couldn’t get it working. I fiddled with it for awhile, read some conflicting documentation on the Paludis website, and finally hopped on the IRC channel to ask for help. I was informed that the documentation was out of date, and when I asked what I needed to do, showing them my error messages and everything, I was told by Mr. McCreesh to just wait a few weeks. That’s a crazy way to treat your users and potential contributors. Maybe I’m just used to the Ruby community, where everyone is helpful and supportive.

I’ve been an idle user of Ubuntu for awhile, its great for my older laptop so I don’t have to spend 5 days compiling X, and all the ACPI and wireless just work without me having to spend an afternoon reading documentation on the gentoo wiki or forums. It installs an awful lot of crap I don’t want, like PAM and games, but that’s an acceptable trade-off to me, for a completely usable desktop system. I can’t stand it for development work, though. All the packages are crazily-named, and you have to install an extra 500MB of stuff to be able to compile anything of your own. And the header files for all the packages are named differently, so you have to search or remember if its -dev, -devel, lib{pkg}-dev, and so on. Gentoo was great, because if you installed a package, you got the header files you needed to develop against it, too. When I got my new laptop, I just skipped even trying Gentoo, and stuck with the Dell Ubuntu that came with it, then a few weeks later replaced it with Ubuntu 8.04.

I’ve tinkered with several other distros for awhile, Slack, Suse, Fedora, but nothing met my development needs as well as Gentoo. Then I saw a post on Zed Shaw’s blog (that I can’t seem to find right now) and he mentioned that he used Arch Linux for development. I’d played with Arch some, its actually being used on my Slice that I’m hosting this blog from. It seemed nice as a server OS, extremely lightweight (after install, you get not much more than bash and vi, just like Gentoo). After reading Zed’s post, though, I decided to give it a shot (I had just melted down my Gentoo machine because portage was broken. Again.)

It’s great. Barebones, extremely configurable, you install only the stuff you want, everything gets installed where I want it, with sane package names, in sane locations. The core repository is rock solid, and the user-submitted package repository (AUR) is well-supported. In the 2 months I’ve been running it, no upgrade has broken my system. Unlike Ubuntu and like Gentoo, Arch doesn’t have distro releases, they just put out an updated live CD every few months. The packages themselves stay pretty close to the lastest release from upstream. You don’t have to wait for a new version of the distro to upgrade to the lastest xorg or kernel version. Overall, its been extremely stable, and the problems I have had received attention from the developers really quickly, and they were extremely helpful.

I think that until Gentoo manages to get its act back together, and be the awesome distro it was for the first few years of its existence, I’m going to stick with Arch for development, and Ubuntu for my laptop. I’m still stuck with Centos for server installs at work, they’re afraid if we tell clients we’re not running some name-brand linux, they won’t buy our product. I tried explaining that if a client cares what OS we run we probably don’t want to sell to them anyways. But as far as rpm-based distros go, Centos isn’t too bad.

My issues with Gentoo are all the breakage, even in non-keyworded packages. If you don’t stay on top of it, and emerge -avu world every week or two, you’re just about guaranteed to spend an afternoon getting everything working again. If you wait more than a month, as a coworker of mine does, you have to spend a whole day getting it to the point where you can upgrade even a single package, because of all the dependencies. I have a server at home that I first installed about two years ago, and haven’t upgraded since. I’m probably better of at this point to just reinstall it.

Not to mention the countless time (at least once a month), where I do my daily emerge world and it fails, and I have to track down on the forums for the answer, if there is even one. Most of the times it can be solved by emerge –sync, but the fact that a package even slipped past QA long enough to make it into stable is appalling.

I love to tweak, and Gentoo still is awesome for that. But Arch is just as good, and the packages are just as fresh, or more so (Where’s postgres 8.3 in Gentoo? It’s been out for 6 months now, and I still have to use an overlay that’s broken more often than not? Is was the same story with upgrading to 8.2, and I thought the plan was to get it slotted to fix all these problems.)

Probably the biggest problem of all is the detachment of the Gentoo leaders from the users. This has been blogged about elsewhere, probably most famously by Daniel Robbins himself. The devs lead the show, which is how it has always been, but over the last few years they’ve been working more on pet projects, rather than making things better for the users. When’s the last time we had a livecd? The Gentoo forums used to be a great resource, but in the same time span, the atmosphere has changed for the worse. There’s fewer people interested in helping out and answering questions these days. The Arch forums remind me of the Gentoo forums of old, its nice being able to find answers to your questions, without having to sift through the rude responses. Maybe it has gotten better, I haven’t visited the forum in several months.

I guess my biggest gripe is the community. The focus has changed away from the users, which is the whole reason Gentoo is in existence. I see on your blog that you’re now a member of the council, maybe you can help make the changes that need to be made.

Tagged , , ,
2008 05 21 Paul Sadauskas Linux Comments (2) Permalink

I might as well get in on it, too

Modified for zshell:

paul@rando64 ~ % cat .zsh_history | awk -F \; '{print $2}' | awk '{a[$1]++}END{for(i in a){print a[i] ” ” i}}’ | sort -rn | head
1435 cd
1057 git
1048 ls
950 vim
438 ruby
405 sudo
323 spec
272 emerge
265 rake
211 make



paul@rando64 ~ % wc -l .zsh_history
10027 .zsh_history

Tagged ,
2008 04 15 Paul Sadauskas Linux Comments (0) Permalink

MySQL Enterprise

I just came across this link about some new MySQL features will be for Enterprise customers only. The feature they mention here is online backups. I think I’m just going to stop using MySQL for any new projects.

I started using MySQL several years ago, near the end of the 3.x series. MySQL at that time was the easiest to get running, and at that point in my career, I wasn’t too interested in SQL-compliance. I started playing around with postgres, but didn’t use it for anything major. I found a lot of its concepts confusing, as my only real db experience was with MySQL (and Access, I’m ashamed to admit).

About 2 years ago I started working for my current employer. They used Postgres, so I bucked down and started learning it. I was impressed with what I saw, but still preferred MySQL. After I was there for a few months, we needed to develop a project that had 10,000s for INSERTs and DELETEs an hour. Testing showed that Postgres spent 50% of its time auto- or manual vacuuming. I switched the dbms over to MySQL using MyISAM tables (we didn’t need transactions or anything fancy for this). It was a huge speed improvement, and we’ve been running both MySQL and Postgres to this day.

Its coming time to rewrite the project, to account for some changes to our design. I’ve been playing around, and the new auto vacuuming stuff in Postgres 8.3 is pretty good. I was going to do some further benchmarking to see if we should consider switching back to just Postgres, but after reading the post I linked above, I’m not going to even bother. Postgres is fast enough these days, its within a few percentage points of MySQL at most common things, and its much more standards compliant. Hopefully Sun will see the light, and realize that continuing down this path will destroy MySQL and the community. Free software developers (including myself) are a fickle bunch, and can jump ship or fork a project with startling speed.

Tagged , ,
2008 04 15 Paul Sadauskas Linux Comments (0) Permalink

MWRC

Well, I’m off to Mountain West Ruby Conf. If any of you Datamapper freaks are going, leave a comment, or email/jabber me at psadauskas@gmail.com. Maybe we can meet up for a beer.

Tagged
2008 03 27 Paul Sadauskas Datamapper Comments (0) Permalink

Birthday!

paul = Person.find_by_name("Paul")
paul.age += 1
paul.save!
Tagged
2008 03 10 Paul Sadauskas Uncategorized Comments (1) Permalink

Rails Logging to Syslog using Logging gem

When using a mongrel cluster, you can either log to a separate file for each mongrel instance, or you can log them all to the same file, but on a loaded cluster, there’s a good chance your logged lines will get interleaved and be unreadable. Luckily, there’s another way. The new replacement for log4r Logging can take care of this. It has a built-in way of not interleaving the lines, but (I think) its using lockfiles to do so, and if so, that’s going to be detrimental to performance. The best solution has been around for 25 years, syslog. And with one of the more recent syslog daemons (syslog-ng, or rsyslog), you can set it up to log your mongrel log wherever you like.

First, install the logging gem: sudo gem install logging

Then, in config/environments/production.rb:

require 'logging'
config.logger = returning Logging::Logger['mongrel'] do |l|
  l.add_appenders( Logging::Appenders::Syslog.new('my_rails_app') )
  l.level = :info
end

For more details, check out the Logging docs, but all I’ve done here is set the process name that it gets logged to as “my_rails_app”, so change this to whatever your app name is.

Then, you can filter it in rsyslogd.conf:

:msg, startswith, " my_rails_app" /var/log/rails/production.log

And you’re done!

Tagged
2008 03 05 Paul Sadauskas Rails
Ruby Comments (0) Permalink

DM Migrations (now with 100% more helpers!)

Added some helpers to the DataMapper Migrations I’ve been writing. These helpers just build up some SQL, and feed it into #execute.

migration 1, :create_people_table do
  up do
    create_table :people do 
      column :name,   :string
      column :gender, :string
    end
  end
  down do
    drop_table :people
  end
end
 
migration 2, :add_age_and_dob_to_people do
  up do
    modify_table :people do
      add_column :age, :integer
      add_column :dob, :datetime
    end
  end
  down do
    modify_table :people do
      drop_columns :age, :dob
    end
  end
end

A caveat: The ALTER TABLE stuff in SQLite is pretty weak. To do anything other than rename the table, or add a column, you have to create a new table with the schema you want, copy the data, then drop the old table. Since these helpers just build the SQL, but don’t execute it, I can’t run any queries against the table at load time, because some previous migration may have altered the schema in between the load and execution of this one. You’re better off just writing the SQL yourself in this case. This is only a factor in SQLite3, though, since Postgres & MySQL support the full ALTER TABLE stuff. In SQLite3, trying to call one of the broken helpers will result in a NotImplemented exception.

Tagged , ,
2008 02 11 Paul Sadauskas Datamapper
Ruby Comments (4) Permalink