However, there’s a problem with EC2. A big one: performance. Bang for your buck. My friend James Golick (@jamesgolick) has often publicly talked about how crappy performance was on EC2, both in terms of CPU and IO. I knew he was right, but didn’t know how right he was really.

I recently found out about Joe’s Data Center, a low-cost dedicated hosting provider so I decided to give them a shot for a pet project I’m working on. They certainly don’t have the infrastructure of Amazon, but that’s a trade-off I can live with given the scope of my project. They have great reviews all over the interwebs so that made me feel at ease.

I leased one of the cheapest machines they are offering: a dual processor AMD Opteron SledgeHammer 2.2Ghz for $55/mo. I went crazy and added 2GB of Ram for an extra $10/mo. Total damage: $65. Roughly the same price as a small instance on EC2 (m1.small). The SledgeHammer is by today’s standards a pretty old and crappy CPU, but I figured it would be plenty for my needs.

I decided to run some benchmarks against an m1.small to see how “bad” the SledgeHammer really was. I stopped paying attention to CPUs the minute I bought my first Mac 6 or 7 years ago. All these fancy CPU names don’t mean much to me anymore.

After some Googling, I settled on UnixBench for my benchmarks. I didn’t need very precise or scientific numbers, just a ballpark to see what I was getting at Joe’s.

I WAS BLOWN AWAY!!! In a nutshell, my dedicated server is 10 times faster than an m1.small instance. TEN TIMES!!! For the same price! IO is also about 5 times faster  than EC2′s local storage. And as a bonus, I get 4GB of Ram instead of just 1.7GB.

After such a surprise, I decided to benchmark more expensive EC2 instance types to see how well my new best friend fares against my olf virtual friends. The results were equally surprising: my dedicated box is faster than a m1.large ($250/mo), faster than a m1.xlarge ($500/mo) and roughly as fast as a High CPU c1.xlarge ($500/mo). More explicitly, the c1.xlarge did ~50% better when running UnixBench with 8 threads, but the Opteron box is 50% faster when single threaded. Also, to get IO roughly equivalent to my dedicated box on EC2, you have to shell out $500/mo for a c1.xlarge. Even the m1.xlarge doesn’t cut it.

EDIT: Since publishing this article, I rented a EX 4S at Hetzner. It is effectively 58 times faster than an m1.small and has 32 GB of Ram. IO throughput is 29 times higher. All this for just $13 more a month.

Here’s a chart of my results (UnixBench index):

The raw numbers (including IO) can be found here: https://gist.github.com/1566734.

I’ll let you draw your own conclusions, but to me, it’s clear that Amazon needs to improve its performance. The bang-for-your-buck ratio of EC2 is completely out of whack.

EDIT: Title was initially “EC2 is basically one big scam”, which was a bad choice of word. I changed to word scam to rip-off, which is more appropriate I believe.

EDIT 2: Hetzner.de has a new amazing machine, the EX 4S (Intel Core i7 2600, 32 GB, 3TB RAID 1). It’s just ~$78/month so I decided to benchmark it as well. HOLY FUCKING SHIT. Check this out: Hetzner EX 4S Benchmarks. I added the numbers in the table above for comparison. Looks like we have a winner!

A fellow Montrealer entrepreneur who recently moved to The City invited me over for Schartz’s Smoked Meat. He somehow managed to import smoked meat from Montreal. Crazy Canuck. Turns out he had enough meat to invite a few other entrepreneurs. I accepted the invitation because experience has taught me that these impromptu parties and gatherings are often the best source of inspiration, validation, information and networking Silicon Valley has to offer. And of course, cooking Montreal smoked meat in San Francisco sounded like a lot of fun!

A relatively well-known and successful startup guy was at the party. We “knew” each other over Twitter and email, but we had never met in person. I started by congratulating him on his recent acquisition, something not quite uncommon in the Valley. I then proceeded to ask a few questions about his startup and mainly why they had to pivot from their awesome and successful first product, since I’ve been toying with an idea that relates to what they were doing.

He asks me what I’m working on and I unleash the elevator pitch. He quickly interrupts me. No need to babble on and on, he already fully understands what I’m talking about, what I’m building, why and who the target market is.

He says “Check out company XYZ. They built pretty much just that 4 years ago but they were too early. I’ll introduce you to the founder if you want.” Awesome! Years of insider knowledge now within reach! He then goes on mentioning a few companies who’s APIs will save me months of research and development.

These serendipitous meetings are what make Silicon Valley so awesome. I went to a smoked meat party “after work” and ended up getting more work done than I would have in 6 months or a year sitting at my desk in Montreal. If you are an entrepreneur outside of Silicon Valley and don’t visit the Bay Area on a regular basis, you are way behind. The world moves fast here.

Oh… and as I’m writing this, Ron Conway just walked by… Hi Ron!

(this actually happened a while ago. I wrote this article the next day but didn’t publish it until now.)

There are numerous reasons to incorporate your startup in the USA, particularly in Delaware. I won’t bore you with the details, since I assume you did your homework. Right? Read this great thread on Quora explaining the benefits of Delaware incorporation for startups. Read the whole thing. It’s a goldmine.

The reasons for me to incorporate in the US were very simple:

• I thought I might raise capital from US investors
• Exit was most likely going to be an acquisition by a US company (that’s what eventually happened)
• Wanted the option to move to the US (possibly with an L-1 visa)
• I expected to sell our product mostly to US-based customers
• Wanted to escape Quebec’s stupid French-language laws

After weighing different options with my excellent US/Canada tax accountant Robert Chayer, it was clear that I had to incorporate in Delaware. If you are starting a tech startup, you are most likely doing it wrong if you incorporate in a US state other than Delaware.  60% of Fortune 500 companies, 50% of publicly traded companies and 50% of all American corporations are based in Delaware.

Facebook? Google? LinkedIn? Yahoo? Amazon? YouTube? Sendgrid? Quora? Yup. Delaware. (search here)  This didn’t happen by accident. But it’s not the point of this post; I’ll let you figure out on your own why Delaware is usually the right choice.

I finally incorporated Karabunga as a Delaware C Corporation on March 6, 2007 through Delaware Intercorp. I also used them as my Registered Agent until I sold Karabunga to Websense in 2009. I do not hesitate to recommend them. They were very professional and knowledgable every time I had to deal with them. I also used their mail forwarding service which was as advertised.

I did not use Delaware Intercorp’s generic Certificate of Incorporation since it didn’t fit the requirements of a tech startup. I had a good idea of what I wanted/needed in terms of number of shares, par value and so on, but I thought hiring a lawyer to draft the legal document was the smart thing to do. YOU SHOULD ALWAYS consult with an expert before incorporating a company. If you screw up, you may have to pay a ridiculous $165,000 franchise tax to the State of Delaware. Doing it right only cost me an extra $150. Ryan Roberts recently published an awesome incorporation checklist. You should need to follow his advice and discuss it with your attorney.

With Karabunga incorporated, I was ready to do business in the US. I was however running the business from a small town near Montreal, Canada. Having Canadian employees working from Canada for a US entity is a pain in the ass and a tongue-twister. Moreover, American companies cannot benefit from the generous startup tax credits offered in Canada.

This is where it gets fun! I ended up incorporating a second company in Canada to deal with these issues. In my case, I incorporated in the province of Quebec for simplicity. Let’s call this new company “QC-inc”.

Both Karabunga and QC-inc would be owned directly by me. According to Robert, this is critical. A Canadian individual owning a US company that owns a Canadian company forms what he called a “sandwich” (CA – US – CA), and although sandwiches are yummy, they have important tax implications that you should in most cases avoid. In retrospect, I wonder what would have happened if I owned QC-inc, and QC-inc owned Karabunga (CA – CA – US). This could have had interesting tax-minimizing benefits after the acquisition. It’s definitely something to bring up with your tax accountant. There was probably a reason why we didn’t do it this way, I forget.

Here’s what the corporate structure looked like:

Karabunga owned all the IP, the Defensio name, the trademarks, the code and the servers (in our case, EC2). Karabunga owned and controlled all the value.

QC-inc was a simple consulting firm that had only one client: Karabunga. Our employees, office, dev computers, ping-pong table and our infamous Dev1 development server all belonged to QC-inc. The idea is to keep both companies as independent as possible. If QC-inc went out of business for whatever reason, it would not have impacted Karabunga in any way (aside from losing all the employees). QC-inc also obeyed Quebec’s French-language laws such as Bill 101. Hopefully you don’t have that problem where you live.

Karabunga is the company we sold to Websense and the employees became Websense employees. I later dismantled QC-inc since it lost its only client and no longer had a purpose.

Having two independent companies also meant treating them as such. It was important for QC-inc to invoice Karabunga at a fair market rate for consulting services. For example, it would have been illegal to take a loss in QC-inc to keep Karabunga’s expenses low. From what I understand, as long as QC-inc breaks even or makes a small profit, you should be fine. But then again, talk to your tax accountant!

Because QC-inc is just like any other Canadian entity, it was eligible for SR&ED. We dealt with Raymond Luk and his fantastic team at Flow Ventures. I advice you to work with a consultant for your tax credits and grants. They’ll take a small cut, but it’s worth it. As a startup founder, you shouldn’t be wasting your time filling out paperwork. I don’t hesitate to recommend Raymond and Flow every occasion I get.

I mentioned earlier that our employees worked for QC-inc rather than Karabunga. “What about stock options?” I can hear you all say! Well, turns out, that was never an issue. QC-inc and Karabunga had an agreement to grant Karabunga stock options to QC-inc employees. End of story.

The US/Canada setup worked very well and was easy to manage. Believe it or not, most headaches came from dealing with the provincial government, not from the State of Delaware. Incorporating and maintaining an incorporation in Delaware is also very affordable.

The first problem I ran into was opening a bank account in the US for Karabunga. Since the shareholders (i.e.: me) were Canadian, banks wouldn’t let me open and account. This is because of some new anti-terrorism laws since 9/11. I applied at 2 or 3 banks and was turned down every time. I finally found RBC Centura (now called RBC Bank), which at the time was owned by RBC in Canada. The closest branch was in Virginia Beach, so I went there an opened an account without any problems. I later learned that I could have opened an account with RBC Centura through a Canadian branch, but I could not have expensed my beach vacation. RBC Bank was recently sold and is no longer owned by a Canadian bank, but according to their website, they will maintain US banking for Canadians.

Another problem arose when I had to file Karabunga’s tax return. I called a few people specialized in US taxes in Canada but they all quoted me thousands of dollars, even though we were not yet profitable. I ended up hiring a small family-run accounting firm in Burlington, VT who did everything over email for just a few hundred dollars. This goes for lawyers as well. US-specialized lawyers in Canada can be ridiculously expensive. US-based lawyers are just expensive.

If you think a good reason to incorporate in Delaware is to save money and avoid taxation, think again. As far as I know, corporate taxes are the same, or maybe even a little higher in the US. Moreover, selling a US corporation instead of a Canadian one could cost you an extra ~$180,000 CAD in taxes. Let me explain.

Every Canadian has a $750,000 lifetime capital gain exemption. It can only be used once. If you sell a Canadian company for $750,000. You pay $0 taxes. If you sell a Canadian company for $1M, you’ll pay taxes on $250k. Only half your capital gain is taxable, and this amount would be taxed at approximately 48%. In this case, you’d pay $60k in taxes ($250k / 2 * 0.48).

If the same million-dollar company was incorporated in the US, the capital gain exemption does not apply. The formula would be $1M / 2 * 0.48. You would have to pay $240k in taxes. These numbers are all approximate, obviously.

The following table illustrates the impact of the capital gain exemption. As you can see, the impact (in percent) decreases as the value of your company goes up. Nevertheless, it’s something to keep in mind.

This dual-incorporation structure is not for everyone. I think it’s a great way to get the best of both worlds, however. It was relatively easy to manage; much easier than I originally envisioned.  Now that you have all this information, it should be relatively easy to replicate. If I started another Internet startup, I would definitely consider replicating the Karabunga setup.

I hope this was helpful. I’m open to questions or comments.

UPDATE: This article is being discussed on Hacker News. Join the conversation.

It was exciting to see these two super smart guys taking the plunge into the crazy world of startups. I was obviously willing to give them as much advice as I could to help them achieve their goals.

However, I don’t believe in giving “empty advice”. That’s useless and a waste of everybody’s time. Instead, I exposed all the problems and roadblocks I identified in their idea and go-to-market strategy. And there were many. Not ridiculous-many, but first-time-entrepreneur-many.

There’s a very fine line between providing valuable advice and killing somebody’s dream. Giving blunt honest opinions and useful advice is hard. However, it’s probably the most valuable things young entrepreneurs can get from more experienced people; novice entrepreneurs tend to repeat the same mistakes.

At the end of our lunch meeting, I told the guys that I didn’t think their business would grow as fast and as big as they were hoping, mainly because of the competitive landscape, but that they should give it a shot anyways. After all, they’re rockstar developers… it wouldn’t take them long to build it and they’d learn a ton. Worst case, they’ll waste just a couple of months of salary, best case, they’ll be rich. It was worth a shot.

And of course, they went for it. They had passed my “got some balls?” test.

Fast forward a few months, I get a phone call from my entrepreneur friends. They said that what I had predicted was happening: competition was tougher than previously anticipated, the user base wasn’t growing as fast as expected, and more importantly, they still couldn’t afford their private zoos.

They once again sought my advice and mentioned that they’d be willing to sell to the right buyer, at the right price. I gave them a few pointers on what to do, who to talk to, and how to play their cards right to make things happen.

April 20, 2010, TeamBox acquires Talker. The guys made a decent chunk of change, certainly not FU money, but definitely something to be more than happy with.

I had lunch with them again yesterday and they told me how much my support and advice had inspired and motivated them. Really!? I felt like a fraud. I didn’t do anything special… I basically (unintentionally) crushed their dreams into oblivion. The advice I gave was so obvious to me, yet, it made a huge difference. Apparently.

Their immense appreciation made me realize something: the smallest things can have a significant impact on other people’s lives. What I did was easy: I just had lunch and a couple phone calls with friends and gave them my 2 cents. It inspired them to work harder, faster and to aim for success.

This is something veteran entrepreneurs need to do more with up-and-coming guys. Grab a coffee and give your honest opinion. Expose flaws, problems and help find solutions. Apparently, it helps a lot, or at least, it gives them hope.

It sure made my day.

Don’t worry, there’s hope! I have saved the old black icon and you can return to the old school way pretty easily.

How to update Twitter for Mac’s icon from blue to black:

1. Download the Black Twitter Icon Pack
2. Navigate to your Applications folder
3. Right click Twitter and select Show Package Content
4. Go to Contents, then Resources
5. Update the file Icon.icns with the one included in the Black Twitter Icon Pack zip file.
6. Profit!

As a bonus, you’ll find an alternative icon that Twitter never used. You may use it if you wish.

Enjoy!

I ran a few benchmarks under 1.8.7 MRI, 1.8.7 REE, 1.9.2 and Rubinius to see how they compare. I also benchmarked to_json to have an order of magnitude. My benchmarks consist of serializing/deserializing a relatively complex Hash (many nested levels). The YAML representation of the Hash is ~3kb.

Conclusions:

• YAML is really slow
• YAML is twice as fast under 1.9.2 vs 1.8.7
• ZAML is much faster than YAML under 1.8.7, but about the same in 1.9.2
• REE (no GC tweaks) isn’t faster in this situation
• Rubinius is ridiculously slow
• Psych under Ruby 1.9.2 is disappointing

Here are the results:

ruby 1.8.7 (2009-06-12 patchlevel 174) [universal-darwin10.0]
--------------------------------------------------------------------
YAML.load:	0.7924
Marshal.load:	0.0742
to_yaml:	5.1661
ZAML.dump:	1.4574
to_json:	0.1152
Marshal.dump:	0.0845

ruby 1.8.7 (2010-04-19 patchlevel 253) [i686-darwin10.4.0], MBARI 0x6770, Ruby Enterprise Edition 2010.02
--------------------------------------------------------------------
YAML.load:	0.7976
Marshal.load:	0.0864
to_yaml:	4.9012
ZAML.dump:	1.3508
to_json:	0.0940
Marshal.dump:	0.0812

ruby 1.9.2p0 (2010-08-18 revision 29036) [x86_64-darwin10.5.0]
--------------------------------------------------------------------
YAML.load:		0.7079
YAML.load (Psych):	2.6055
Marshal.load:		0.0928
to_yaml:		2.1682
to_load (Psych):	2.2924
ZAML.dump:		5.0259
to_json:		0.1133
Marshal.dump:		0.0908



rubinius 1.1.1 (1.8.7 4018ebda 2010-11-16 JI) [x86_64-apple-darwin10.5.0]
--------------------------------------------------------------------
YAML.load:	2.7777
Marshal.load:	0.8560
to_yaml:	23.5828
ZAML.dump:	4.0745
to_json:	3.6620
Marshal.dump:	3.3797

If you want to benchmark on your machine, here’s the source code. Sorry, I couldn’t include the YAML string as it is confidential data.

require 'rubygems'
require 'benchmark'
require 'yaml'
require 'zaml'
require 'json'

yaml_str = <<-eos
yaml goes here
eos

h = nil
GC.start
yaml_load_time = Benchmark::realtime {
1000.times { h = YAML.load(yaml_str) }
}

GC.start
to_yaml_time = Benchmark::realtime {
1000.times { h.to_yaml }
}

begin
  YAML::ENGINE.yamler = 'psych' if RUBY_VERSION == "1.9.2"
rescue
  puts "Cannot load Psych. Is libyaml installed? Install it and recompile Ruby."
  puts "On OS X with RVM, install it with MacPorts and run:"
  puts "rvm install ruby-1.9.2 -C --with-libyaml-dir=/opt/local"
  exit 1
end

GC.start
psych_load_time = Benchmark::realtime {
  1000.times { YAML.load(yaml_str) }
}

GC.start
to_yaml_psych_time  = Benchmark::realtime {
  1000.times { h.to_yaml }
}

GC.start
to_json_time = Benchmark::realtime {
1000.times { h.to_json }
}

GC.start
zaml_dump_time = Benchmark::realtime {
1000.times { ZAML.dump(h) }
}

marshaled = nil
GC.start
marshal_dump_time = Benchmark::realtime {
1000.times { marshaled = Marshal.dump(h) }
}

GC.start
marshall_load_time = Benchmark::realtime {
1000.times { h = Marshal.load(marshaled) }
}

puts `ruby -v`
puts "--------------------------------------------------------------------"
puts "YAML.load:\t\t#{'%.4f' % yaml_load_time}"
puts "YAML.load (Psych):\t#{'%.4f' % psych_load_time}" if RUBY_VERSION == "1.9.2"
puts "Marshal.load:\t\t#{'%.4f' % marshall_load_time}"
puts "to_yaml:\t\t#{'%.4f' % to_yaml_time}"
puts "to_load (Psych):\t#{'%.4f' % to_yaml_psych_time}" if RUBY_VERSION == "1.9.2"
puts "ZAML.dump:\t\t#{'%.4f' % zaml_dump_time}"
puts "to_json:\t\t#{'%.4f' % to_json_time}"
puts "Marshal.dump:\t\t#{'%.4f' % marshal_dump_time}"

Alex Coles (@myabc) volunteered and is now the new maintainer. He already shared with me a couple good ideas for the site, which he’ll hopefully implement soon.

Thank you, Alex, and congrats!

Although I didn’t change the UI (because it’s so gorgeous already!), I added the following features:

• Gem page lists dependencies
• Warning when Gem contains C extensions (since they don’t work on JRuby)
• Ability to specify the JRuby version against which the Gem was tested

I accept patches, so fork away!

HD Video on iTunes (missing the last 20-25 mins unfortunately):

MeshU - Part 1

MeshU - Part 2

• A quick recap of what I said.
• And Jonathan Keebler live-blogged the event at ScribbleLive.
• My slides:

I guess this is as close as it gets to being there in person! Looking forward to your comments!