As a note, this isn’t about these specific flaws: It’s that these kinds of flaws are endemic in ruby and particularly rails.

Set Processing

devon@chronos ~/Projects/sets $ cat sets1.rb
#!/usr/bin/env ruby
require 'set'
require 'pp'
large = Set.new(Range.new(1,10000000))
small = Set.new(Range.new(1,10))
pp small.intersection(large)

real 0m35.421s
user 0m23.900s
sys 0m9.910s



devon@chronos ~/Projects/sets $ cat sets2.rb
#!/usr/bin/env ruby
require 'set'
require 'pp'
large = Set.new(Range.new(1,10000000))
small = Set.new(Range.new(1,10))
pp large.intersection(small)

real 0m16.917s
user 0m12.720s
sys 0m3.430s


So the order that I intersect the sets drastically impacts the speed of the algorithm (2x). Let’s take a look at python to see whether a language in the same class behaves similarly.


devon@chronos ~/Projects/sets $ cat sets1.py
#!/usr/bin/env python

real 0m2.583s
user 0m1.850s
sys 0m0.670s



devon@chronos ~/Projects/sets $ cat sets2.py
#!/usr/bin/env python

real 0m2.598s
user 0m1.760s
sys 0m0.710s


So no, python handles the choice of which set to intersect with the other gracefully. This is a good example of how the ruby set api is immature. I’m ignoring that it’s taking between 13.7 and 6.5 times as long to execute, because I think that it’s reasonable for people to do code in something that is easier for them to work in and then optimize. My problem here is that based on the direction you do the intersection, it can take 2x longer in ruby. This is shoddy engineering and is the kind of thing that gets solved in a language as it matures. The problem is that the language was immature when it exploded into widespread use, and no one seems to be doing anything to make it ready for prime time.

So what about 1.9? 1.9 is supposed to be where the energy is going to make the language faster and more mature.

devon@chronos ~/Projects/sets $ cat sets1.rb
#!/usr/bin/env ruby1.9
require 'set'
require 'pp'
large = Set.new(Range.new(1,10000000))
small = Set.new(Range.new(1,10))
pp small.intersection(large)

real 0m13.310s
user 0m11.240s
sys 0m0.680s



devon@chronos ~/Projects/sets $ cat sets2.rb
#!/usr/bin/env ruby1.9
require 'set'
require 'pp'
large = Set.new(Range.new(1,10000000))
small = Set.new(Range.new(1,10))
pp large.intersection(small)

real 0m9.181s
user 0m8.280s
sys 0m0.480s


While it’s faster (which wasn’t really my complaint anyway), it still is very different based on which direction you do your intersection from. I see evidence of speed work being done on ruby, but not on maturing the language so that it meets tried and true engineering principals like the rule of least surprise. (Yes, code is a user interface for coders, so the rule of least surprise applies) (1)

Nil as an object
How about this one that has actually done damage to our database:


devon@chronos ~/Projects/sets $ irb
irb(main):001:0> nil.id
(irb):1: warning: Object#id will be deprecated; use Object#object_id
=> 4


It certainly explains why we have a bizarrely large number of items in our database with foreign keys set to 4. This conflicts directly with the rails standard for naming a database primary key as ‘id’, which you assess on a model at obj.id. Yes, code should have lots of null checks, but real world code has flaws. Sure it throws a warning, but since it doesn’t break on nil when you call .id, you get data corruption. Gee, thanks, ruby.

Ruby in general
These are just a few examples, but ruby is rife with happy path coding. The language needed another 5 years of development before going into widespread use. Instead, I think it’s getting worse because a set of rails engineers who don’t know anything about languages are at the helm, adding lots of syntactic sugar but with a distinct lack of substance behind that candied shell.

_{1. Perhaps it’s not fair to compare ruby to the unix philosophy, because the ruby design philosophy violates the 17 rules in ways that are clearly intentional. Specifically, ruby and rails egregiously violate the rules of clarity and transparency, and rails in particular violates the rules of separation, composition and representation.}

1. Conservatives would be more comfortable with a browser delivered by a major corporation that has faced antitrust charges over that browser then liberals would.
2. Liberals anecdotally are more prone to counter-cultural choices, and thus would be more likely to seek out an alternative to the default browser.

The Google interviews made me realize one other thing. In testing for this kind of effect, you would need to eliminate people who didn’t know what a browser was. Clearly, a person who doesn’t even know what a browser is is highly unlikely to proactively switch from their system’s default browser. Further, if a person’s political views are correlated at all with their likelihood of understanding what a browser is, not eliminating people who don’t understand could hide real results.

The company I work for, Knewton Inc, is becoming known in certain circles of its clever usage for the vastly underused (IMHO) Amazon Mechanical Turk service. When I mentioned my contention to our guy who’s been pioneering our MTurk usage, Dahn Tamir, he suggested that we build an MTurk task and get some real data to find out whether my hypothesis had any basis in reality. The rest of this post represents my findings.

The Experiment
We posted the following poll to MTurk, allowing users the ability to do the task only once (making it a lot harder to stuff the ballot box, or, at the very least, require a hell of a lot more effort on the part of the users):

The options for political affiliation were:

• Republican
• Independant
• Democrat

The options for political philosophy were:

• Strongly Liberal
• Liberal
• Moderately Liberal
• Moderate
• Moderately Conservative
• Conservative
• Strongly Conservative
• Socialist
• Libertarian

For the web browser, we considered a person to have passed the test if he or she answered Internet Explorer or Firefox, but no others. We considered them to have failed if they answered Yahoo, Google or MSN. It was pointed out that AOL did have a browser that could be reasonably referred to as AOL, so we ignored this field. Finally, we recorded the workers’ User Agent, letting us see what browser/os they were using.

Raw Results
Observations:

• Clearly, MTurkers are going to be more likely to know what a browser is, and it shows in the data. Google found that 8% of people in Times Square knew what a browser was, but 86% of respondents here did.
• We appear to have more Democrats/Liberals than Republicans/Conservatives one would expect in the normal population.
• I only added Socialist on a lark, and I was pretty amazed to see that it still got 2.24% in a poll of Americans
• Firefox polled a lot higher here than one would expect in the wild. This should make it a lot easier to see any effects, along with the high pass rate of the test.

Party Affiliation
Total % Republican 244 23.71% Independent 397 38.58% Democrat 388 37.71%

Political Philosophy
Total % Strongly Liberal 92 8.94% Liberal 188 18.27% Moderately Liberal 169 16.42% Moderate 197 19.14% Moderately Conservative 157 15.26% Conservative 105 10.20% Strongly Conservative 34 3.30% Socialist 23 2.24% Libertarian 64 6.22%

Browsers
Total % Firefox 479 46.55% Internet Explorer 439 42.66% Safari 63 6.12% Chrome 41 3.98% Opera 7 0.68%

Browser Test Results
Total % Pass 886 86.10% Fail 143 13.90%

Test Results
Next, after pulling in the raw results, we compared browser, affiliation and philosophy to the test results.

Observations:

• Unsurprisingly, IE fared better amongst people who failed the test. Firefox did substantially better amongst people who passed the test. Safari also did better amongst people who passed the test, which was a little surprising, because we expected all default browsers to do better with people who failed. The sample, however, is small enough that the Safari factor could just be noise. It has been mentioned that this could have been caused by Safari for windows. That supposition can potentially be answered in the data.
• Democrats did worse on the test, while Republicans and Independents did basically the same. We see the likely reason why in the political philosophy section. The further left you go, the worse you tend to do on the browser test, until you hit strongly liberal. Strong Liberals reversed the trend entirely and tested better than any other group.

Test Results by Party Affiliation
Pass % Fail % Republican 87.70% 12.30% Independent 88.16% 11.84% Democrat 82.99% 17.01%

Test Results By Political Philosophy
Pass % Fail % Strongly Liberal 91.30% 8.70% Liberal 81.91% 18.09% Moderately Liberal 84.62% 15.38% Moderate 86.29% 13.71% Moderately Conservative 86.62% 13.38% Conservative 87.62% 12.38% Strongly Conservative 88.24% 11.76% Socialist 86.96% 13.04% Libertarian 89.06% 10.94%

Browser by Test Results
Pass % Fail % Internet Explorer 39.16% 64.34% Firefox 49.89% 25.87% Safari 6.21% 5.59% Chrome 4.06% 3.50% Opera 0.68% 0.70%

Browser choice by political opinion
Here we have the final analysis. Looking at how browser choice changes across the political spectrum.

Observations:

• My initial hypothesis appears to be supported by this data set. IE falls off sharply the further left you go on the spectrum from Moderate. Strong liberals use IE at half the rate of strong conservatives.
• Both Libertarians and Socialists appear to fit in on the left end of the spectrum at least where browser choice is concerned. Libertarians appear to loath IE and are nuts for Firefox. They have the highest Firefox rating of anyone with an astounding 70.18%
• Firefox’s fall as you go to the right is less pronounced as IE’s fall as you go to the left. This is probably caused by the generally higher likelyhood of picking Safari as you go leftward.
• Opera appears to have its ’stronghold’ with Socialists, who were as likely to pick it as Chrome or Safari.
• Libertarians also appear to have an exaggerated preference for Opera, Safari and Chrome. Basically, Libertarians really don’t like IE.

Browser by Party Affiliation
IE % Firefox % Safari % Chrome % Opera % Republican 45.33% 45.79% 5.14% 3.27% 0.47% Independent 36.00% 52.86% 6.57% 3.71% 0.86% Democrat 38.51% 49.38% 6.52% 4.97% 0.62%

Browser By Political Philosophy
IE % Firefox % Safari % Chrome % Opera % Strongly Liberal 25.00% 63.10% 8.33% 3.57% 0.00% Liberal 35.71% 50.65% 9.74% 2.60% 1.30% Moderately Liberal 41.96% 48.25% 4.20% 5.59% 0.00% Moderate 45.88% 44.12% 4.12% 5.88% 0.00% Moderately Conservative 47.06% 43.38% 7.35% 2.21% 0.00% Conservative 44.57% 46.74% 4.35% 3.26% 1.09% Strongly Conservative 50.00% 46.67% 0.00% 3.33% 0.00% Socalist 30.00% 55.00% 5.00% 5.00% 5.00% Libertarian 12.28% 70.18% 8.77% 5.26% 3.51%

Edit
Some have found the graph confusing, so here are two other versions. Both horizontal, showing the data in one by browser, and in the other by political philosophy. Click on them to enlarge

(enlarge)

(enlarge)

Conclusion
There appear to be a few strong correlations in data between a person’s political thinking and his or her browser choice. Still, this data is not representative of the population as a whole. It is, instead, a sample of people who are web savvy enough to use MTurk and who are motivated to take polls like this for small amounts of money. Also, the strong bias of the tech and open source communities toward libertarianism is probably causing the strong bias of libertarianism toward Firefox in this sample. Still, I thought these results were too interesting to keep to myself. Enjoy.

Notes
If you want to toy with the raw data and get access to the source I used for experimenting with the data, as well as my analysis spreadsheets, those are available here. We did this in 3 separate data gatherings, and those samples are still separated into 3 data files the raw data.

var Bar,QClose,QOpen:integer;
var gap,Qgap:float;
QClose:=GetExternalSeries(‘QQQQ’,#Close);
QOpen:=GetExternalSeries(‘QQQQ’,#Open);

for Bar:=2 to BarCount-1 do
begin
Qgap:=100*(@QClose[Bar-1]-@QOpen[Bar])/@QClose
[Bar-1];
gap:= 100*(PriceClose(Bar-1)-PriceOpen(Bar))/
PriceClose(Bar-1);

if not LastPositionActive then
if (gap > 5) and (Qgap > 0.5) then
if PriceClose(Bar-1)

The above script (code) encodes the following rules:

“Buy a stock when the stock is down the day before, QQQ is gapping down more than half a percent, and the stock is gapping down more than 5 percent.”

“Hold the stock at least until the next morning.”

“Sell when the stock goes lower than the prior day’s close.”

Without examining this language and the whole site in great detail (which I assure you gentle reader, I will be looking into this…) I gather that this script is run at the start and end of each trade day to determine what action should take place for a given stock. This is fantastic, it takes the emotional element out of stock trades. As anybody who has ridden a stock all the way to it being delisted just hoping that it will turn around, you know how important it is to have specific rational trading rules for when to sell and when to buy. ]]> http://www.evilsoft.org/2005/12/09/wealth-lab/feed 0 http://www.evilsoft.org/2005/12/02/why-i-still-use-delicious http://www.evilsoft.org/2005/12/02/why-i-still-use-delicious#comments Fri, 02 Dec 2005 18:41:28 +0000 http://www.evilsoft.org/?p=49 See I still use del.icio.us and here is why. Now my compatriot has been telling me how cool simpy is, but I have so much already invested in del.icio.us and frankly now that it has even tigher integration into firefox I am less likely to change. I mean its got handy buttons (See Fig 1)
and lets you take notes (See Fig 2) by highlighting portions of the current viewed page.

Fig 1

Fig 2

I will be using it for the next week or so and write a more formal review then, in the meantime if you use del.icio.us check it out.

Del.icio.us better watchout, because there is a new kid on the block: Simpy. While I have not noticed many new features out of del.icio.us since the experimental posting page, simpy is regularly adding useful new features. Thing I have noticed so far that Simpy does that del.icio.us doesn’t do (or I have failed to find):

1. It lets you essentially make Boolean expressions with tags. So instead of clicking on Development, and then seeing that it shares some links with java – I would expect when I click on java, I would only see links that are *only* java AND development – not so on delicious – in stead I’ll see all of java. Being able to narrow down tags is a *huge* improvement. Not only can you focus in on tags, but you can also remove links from the results based on tag. Very useful. I pretty much always wished del.icio.us had this feature.
2. Private links. Del.icio.us is great, and I love being able to share links with everyone – but there are some links that I just want to share across my browsers – and not other people. Say like the link to my bank and credit card company – or links from my company’s intranet.
3. Groups. Simpy lets the user join groups, and post links for the consumption of the other people in the group – once a link has been posted, you can then mark it as read – allowing you to track what you ahve looked at. Sweet.
4. Topics. Ever want to track someone else’s links on delicious? Simpy apparently lets you do this in what it calls a topic.
5. Notes. Simpy lets you take notes with their system that you can tag the same way you tag links. Not much use to me yet, but an interesting idea – since it’s centralized, it offers the same benefits for notes that both systems offer for links.
6. The killer one: Sync from del.icio.us. All the effort you have put into tagging things on del.icio.us is not lost – simpy can suck all the links out of your del.icio.us account, and it will tag them appropriately, meaning you lose nothing. This means testing Simpy is pretty risk free, because you can take your existing links, pull them in and see if you like it.

Now, it’s not all better then del.icio.us. If there is one single thing I think del.icio.us does far better it’s the link add form. The experimental interface for del.icio.us deals with tags far more eloquently then simpy’s tagging interface. Where del.icio.us will recommend tags to you from *your* tags, as well as showing you all your tags, and showing you popular ones, Simpy has a truncated list of your tags, and only tells you popular ones. This makes it *really* hard to consistantly use the same tags in your link collection. Furthur, del.icio.us’s link add page is far more eloquent allowing you to simply click or unclick the tag to add or remove tags. Simpy divides it across the page so that your tags are in a textbox, but there are a series of checkboxes on the side of the screen where you can add predefined tags. Unlike delicious, when you check one of those boxes, it doesn’t populate the link textbox, and unchecking it doesn’t remove one. The Simpy developers would do well to take a page from del.icio.us on tagging interface – the one area where del.icio.us far outshines simpy.

You can find me there are soulcatcher.

I am not sure how I am going to use this, but I am going to find a way, damnit. The ability to embed a web browser in a web page is just such a cool, little idea. As they say on their website, its like TV in TV for your web browser. Now I just gotta think of a way to use that here…

• Easy to install (if like me you have a web server running linux. Gentoo makes installing mediawiki a breeze)
• You have access to it from anywhere, using anyone’s computer
• You don’t have to worry as much about losing your data. A server is easy to keep backed up. When is the last time *you* backed up your thumb drive?
• Rock solid – this is after all the software that runs the gigantic Wikipedia.
• Support for media – so I can upload related photos/drawings/other media and attach that to my notes.
• Lots of good patches are available, like the one I installed that lets you make pages private.
• Running this on a webserver means it’s simple to make any part of it accessible to other people – and you can give them accounts if you want to collaborate with people on an idea.

Course, now the software engineer in me wants to do all sorts of stuff with it, which is always dangerous. Ideas so far include finding a way to integrate it with Gallery, and writing a wiki that allows Game information to be stored in a wiki – with privledged accounts for referees/GMs, and accounts with lower permissions for players.

Hrm… Wonder if I can write a plugin for PCGen.

Bar none the best experience I have had with this was at work. Being a contractor, I have to punch in and out multiple times per day, but my employer’s IT department doesn’t support anything other then IE internally (this seems to be the norm). Well, they made 2 small mistakes in their html and javascript that made the site unusable on anything other then IE, which of course means that I have to use two browsers. Well, Greasemonkey came to my rescue. Because I can use greasemonkey to dynamically re-write sites, the timesheet app now works perfectly in firefox for me, and IT didn’t have to lift the finger they refused to.

So, what I have installed in my firefox:
Web Development:

• Venkman: A javascript debugger for firefox. This package is fantastic for javascript developers, and provides you with a number of useful features. Must have if you do javascript.
• Web Developer: Adds a menu bar

Google:

• Google Pagerank Status: Google Pagerank is the algorithm that pretty much controls the web, well now you can see in your status bar just how popular a particular site is according to google.
• GooglePreview: This extension is surprisingly useful. All it does is adds a snapshot of the web page into the google results. If you are looking for a link to a page you have been to before, the fact that you can see it is invaluable.

Just Plain Cool:

• FoxyTunes: Lets me control Winamp from the status bar of firefox. No more fumbling around to which desktop my winamp is on, just find the nearest browser.
• BlogThis: A plugin that well…lets me post here more easily.
• del.icio.us: A fantastic replacement for the bookmarklets that let you post a link to deli.icio.us. All the functionality of the experimental del.icio.us interface, but with a xul window. Yum.
• SessionSaver 2: I keep a lot of tabs open. I mean a *lot*. This makes rebooting, or for that matter shutting down firefix a 10 minute ordeal as I determing which pages I still need to read, and which ones I ma done with – as I save the links somewhere for after my reboot. Well, no more. Sessions saver grabs the state from the last Firefox session, and opens the same windows/tabs.
• Greasemonkey: This plugin isn’t interesting, it’s web changing. Greasemonkey is a paradigm shift in how the web works. No longer are we constrained by the piss poor design decisions other people make in their websites. Using Greasemonkey, you can write or download scripts that completely change the way that a web page works. You can alter the content, layout, javascript, whatever you need. Some really clever thing shave been done with this like the hack that overlays the subway system of Chicago on google maps.
• Aardvark: Sort of a cross between Greasemonkey and webdeveloper, Aardvark lets you select elements of a web page, and either remove them, or alter their formatting to make them easier to read/print. Perfect for printing sites that make getting a good hard copy a pain in the ass.