EC2 is basically one big rip-off
by Carl Mercier
First of all, let me set this straight: I LOVE Amazon Web Services. I’ve been using EC2 for about 6 years and Defensio has always been entirely running on AWS. The flexibility is unparalleled and the variety of services such as EC2, ELB, EBS, S3 and CloudFront is a joy to use.
However, there’s a problem with EC2. A big one: performance. Bang for your buck. My friend James Golick (@jamesgolick) has often publicly talked about how crappy performance was on EC2, both in terms of CPU and IO. I knew he was right, but didn’t know how right he was really.
I recently found out about Joe’s Data Center, a low-cost dedicated hosting provider so I decided to give them a shot for a pet project I’m working on. They certainly don’t have the infrastructure of Amazon, but that’s a trade-off I can live with given the scope of my project. They have great reviews all over the interwebs so that made me feel at ease.
I leased one of the cheapest machines they are offering: a dual processor AMD Opteron SledgeHammer 2.2Ghz for $55/mo. I went crazy and added 2GB of Ram for an extra $10/mo. Total damage: $65. Roughly the same price as a small instance on EC2 (m1.small). The SledgeHammer is by today’s standards a pretty old and crappy CPU, but I figured it would be plenty for my needs.
I decided to run some benchmarks against an m1.small to see how “bad” the SledgeHammer really was. I stopped paying attention to CPUs the minute I bought my first Mac 6 or 7 years ago. All these fancy CPU names don’t mean much to me anymore.
After some Googling, I settled on UnixBench for my benchmarks. I didn’t need very precise or scientific numbers, just a ballpark to see what I was getting at Joe’s.
I WAS BLOWN AWAY!!! In a nutshell, my dedicated server is 10 times faster than an m1.small instance. TEN TIMES!!! For the same price! IO is also about 5 times faster than EC2′s local storage. And as a bonus, I get 4GB of Ram instead of just 1.7GB.
After such a surprise, I decided to benchmark more expensive EC2 instance types to see how well my new best friend fares against my olf virtual friends. The results were equally surprising: my dedicated box is faster than a m1.large ($250/mo), faster than a m1.xlarge ($500/mo) and roughly as fast as a High CPU c1.xlarge ($500/mo). More explicitly, the c1.xlarge did ~50% better when running UnixBench with 8 threads, but the Opteron box is 50% faster when single threaded. Also, to get IO roughly equivalent to my dedicated box on EC2, you have to shell out $500/mo for a c1.xlarge. Even the m1.xlarge doesn’t cut it.
EDIT: Since publishing this article, I rented a EX 4S at Hetzner. It is effectively 58 times faster than an m1.small and has 32 GB of Ram. IO throughput is 29 times higher. All this for just $13 more a month.
Here’s a chart of my results (UnixBench index):
| m1.small | m1.large | m1.xlarge | c1.xlarge | Joe’s Dedicated | Hetzner EX 4S | |
| Monthly Price | $65.00 | $250.00 | $500.00 | $500.00 | $65.00 | $78.00 |
| Architecture (Ubuntu) | 32-bit | 64-bit | 64-bit | 64-bit | 64-bit | 64-bit |
| CPUs/Cores | 1 | 2 | 4 | 8 | 2 | 8 |
| RAM (GB) | 1.7 | 7.5 | 15 | 7 | 4 | 32 |
| Index (1 Thread) | 116 | 357.5 | 438.1 | 494.4 | 777.6 | 1803.3 |
| Index (2 Threads) | 571.5 | 1210.5 | ||||
| Index (4 Threads) | 1070.5 | |||||
| Index (8 Threads) | 1746.3 | 6696.7 |
The raw numbers (including IO) can be found here: https://gist.github.com/1566734.
I’ll let you draw your own conclusions, but to me, it’s clear that Amazon needs to improve its performance. The bang-for-your-buck ratio of EC2 is completely out of whack.
EDIT: Title was initially “EC2 is basically one big scam”, which was a bad choice of word. I changed to word scam to rip-off, which is more appropriate I believe.
EDIT 2: Hetzner.de has a new amazing machine, the EX 4S (Intel Core i7 2600, 32 GB, 3TB RAID 1). It’s just ~$78/month so I decided to benchmark it as well. HOLY FUCKING SHIT. Check this out: Hetzner EX 4S Benchmarks. I added the numbers in the table above for comparison. Looks like we have a winner!
L’offre de Joe’s datacenter semble intéressante. Je vais définitivement checker ça si Linode vient à être trop peu pour mes besoins.
Julien: Linode performe beaucoup mieux qu’EC2 dans tous les benchmarks que j’ai vu. C’est domage qu’EC2 soit si lent parce que à part ça, c’est la plateforme idéale à mon avis!
you’re not buying performance with EC2, you’re buying the flexibility, that’s something which is clear.
what also clear is that they cannot never be as fast as a dedicated server, unless there is a breakthrough in both virtualization technology and CPU architecture
EC2 basically costs $500/mo per instance for the equivalent of a 9 year old CPU. Such a computer could probably be purchased for about $100-$200 on eBay. Regardless how flexible it is, something just doesn’t make sense.
EDIT: Actually, you can buy the same server I’m leasing for $40 on eBay! http://bit.ly/zZoAq8
I think its pretty obvious to anyone that has any experience with EC2, or most of AWS, that you’re not paying for (or going to get) the best bang for your buck. Hands down price wise its one of the most expensive. This should be evident to anyone who has worked with EC2 for any given amount of time. What you pay for is flexibility, scale, and all the feature set and configuration you’re able to play with. Price and performance “per instance” clearly is a deadweight argument. Unless you aim to do some serious scaling and require actual usage of the “cloud”, most VPS or dedicated servers will win price/performance any day of the week.
Pour du hosting, je suis très d’accord.
Par contre, pour le traitement de données, c’est difficile à battre comme offre. Au travail, on utilise plusieurs dizaines d’instances xlarge en hadoop pour traiter nos données durant 7-8 heures par semaine. Le spot pricing nous permet de sauver également beaucoup d’argent (habituellement le 1/3 du prix).
Yeah I don’t think that eBay server you mention has the same amount of RAM as that m1.xlarge you’re referring to.
But hey, if you honestly believe that you’re better off with a dedicated machine, then I’m not going to stop you.
Is it any surprise? The moment you go VM with current technology, you get a 1/10 penalty.
Virtualization incurs very little performance penalty. The results I saw is definitely NOT caused by the virtualization overhead.
That may or may not be true. It’d be great to see you run the same comparison with an online VM service such as Linode.
This is actually a very well known fact of life. Many companies virtualize their own co-located servers for flexibility. If there was a 90% performance penalty, nobody would do it.
I did some tests with GeekBench and AWS is a rip-off. The underlying hardware can be quite new, but VM density is exaggerated – I estimate well over 100 Small Instances per 2 socket server.
Hi Carl,
The specs that Joe’s Datacenter is providing for this money is nothing. Check the Hetzner smallest dedicated server offer for EUR 49 (~ $62):
http://www.hetzner.de/en/hosting/produkte_rootserver/ex4
Basically the specs are:
Intel® Core™ i7-2600 Quadcore
incl. Hyper-Threading Technology
RAM16 GB DDR3 RAM
Hard disks 2 x 3 TB SATA 6 Gb/s HDD
7200 rpm (Software-RAID 1)
1 GBit OnBoard connected at 100 MBit
Backup Space100 GB
TrafficUnlimited
What do you think
Regards,
Yas
Yeah I know about Hetzner, but they’re in Europe unfortunately…
Yeah you’re comparing an _elastic_ compute instance to running a _single_ compute instance. Your argument is fine, but it only applies to anyone who is silly enough to pay for an elastic computing instance when all they require is a simple server. If you don’t have the requirements for scale-out and/or enormous computational problems, then why would you use EC2 at all?
That’s what most startups do on EC2.
ash, that is true. But at what cost? If you have access to elastic computing that cost 10x are you truly getting the benefits of being able to scale efficiently? Would not an option be to simple expand your current architecture 10x thereby getting 10 times the theoretical performance of the single instance?
With these types of numbers being bantered about the scope of elasticity required would need to be quite high, no?
Finally someone who gets my point!
And keep it mind that the 10x performance improvement is compared to very old and crappy hardware. My CPU at Joe’s is 9 years old. For not much more money you can get 50x speed improvement.
This is not only way cheaper than 50 m1.small instances, but also much easier to manage.
How is it easier to manage? What happens if the computer at Joe’s locks up? Or if the hardware goes bad?
Locks up? Reboot it. Hardware goes bad? They change it. These problems also happen on virtual machines.
Managing 1 box instead of 10 is 10 times less work.
And when your dedicated server goes down, then what? Performance only goes so far. High availability is a hard problem that many are willing to pay a premium to EC2 for.
An EC2 instance is actually more likely to go down than a dedicated server. Unless you have many EC2 instances, it’s not more highly available than a dedicated box. So yeah, you need redundancy in both scenarios.
EC2 is the easier choice but not the better one.
Even the argument about flexibility is a nonsense.
There are hundreds of hosting companies that were in the hosting business long before Amazon.
Amazon is new to this business and primarily relies on it’s name and the so called cloud marketing hype to to bring what pretty much every big hosting company had and has in it’s datacenter.
Amazon spent some time to build web-based apps that bridge gap between virtualization and some management tools so that the customer can manage VM via the browser – and that’s it…
I used to look at them as potential hosting (cloud) vendor but as Carl pointed out paying for under-performing VMs is not fun! Not at all!
There’s a huge spike in traffic on this article. My stats seem to be indicating that people are being sent here from their email client. Was this feature in some sort of newsletter or mailing list? Which one? I’m curious!
This article showed up on CodeProject mailing list this morning.
Your article was mentioned on the Code Project mailing list http://www.codeproject.com/script/News/List.aspx. Possibly on others. Very interesting comparison by the way.
Yes, you were featured in the CodeProject daily newsletter.
http://www.codeproject.com/script/Mailouts/View.aspx?mlid=9444
Code Project email today:
EC2 is basically one big rip-off
Your cloud may vary.
From the CodeProject | Daily News daily developer news e-mail Carl.
Thanks all! That’s what I thought but I wasn’t sure. I used to be a huge CodeProject fan but I no longer program in .NET so I haven’t visited the site in a while. Kinda miss The Lounge though
You’re on the Code Project daily feed.
“CodeProject | Daily News” newsletter. Cheers.
CodeProject | Daily News
CodeProject Daily News
Carl, I came to read this article because it was linked from The Code Project’s Daily News email.
CodeProject | Daily News
http://www.codeproject.com/script/Mailouts/View.aspx?mlid=9444
This artical was mentioned in the CodeProject | Daily News news letter It’s good to see articles like this I think most of us tring out the cloud are are seeing stuff like this I’m tring out Azure and I’m finding even shared hosting plans have the cloud beat I tried the 90 day free trial from Microsoft some how used the first month up in ten days other people used theirs up in three days had to release credit card to continue free trial is not free any more and I’m starting to think this whole cloud thing is one big rip off the hosting companies can rest easy for now
Here’s the sad thing people don’t realize: it’s pretty easy to buy THREE servers and VMWare, and create your own private cloud for a smidgeon of the price you pay for Amazon’s virtservers. You get some scaling (add a server) and most of the availability. If you stick it in a good datacenter, you can pay a reasonable hourly rate for any hands-on you may need.
Amazon used to be cheap. Not anymore. The drive of every startup and their uncle to that service, plus the plethora of growing virt options in the market, makes “go EC2 or go home” a bit outdated.
It’s useful for real big scaling issues, such as an ad on the Super Bowl may create, or other edge cases. But, startups should reconsider their spending…
Have you tried Rackspace? I’ve heard that you get more bang for your buck when compared to EC2, but I’ve never compared them.
Rackspace is indeed faster according to the benchmarks I’ve seen. Linode is supposed to be even faster. But they’re still nowhere as fast as a dedicated box. So yes, the price/performance is better, but you don’t get all the advantages of AWS.
In your costs analysis do you consider?
- Energy consumption
- Security services (Firewalls, physical security, etc)
- Backup service
- Aviavility
- Maintinance
- Energy: I’m not paying for it. And actually, faster CPUs typically require less energy for the same performance
- Security: Even on EC2, you have to deal with your own firewalls. Security Groups are awesome though. That’s one thing that I dearly miss on a dedicated box. Physical security-wise, any decent data center has good security. I don’t think the security of Softlayer vs Amazon is much different.
- Backup services: that’s called S3, which I can still use from a dedicated box. EC2 does not come with any backup tools whatsoever, although EBS snapshots are great and very convenient
- Availability: it’s not true to say that an EC2 instance is “more available” than any other type of machine. If anything, they are less reliable. It’s all about redundancy. Obviously, a single dedicated box is likely less reliable than 2 redundant EC2 instances.
- Maintenance: Maintaining 1 machine instead of 10 EC2 instances is much easier, so this is actually a big win for dedicated hosting.
As I mentioned before, the main thing that I dearly miss is the AWS ecosystem. EBS, S3 (locally), RDS, Elastic IPs, the API, Security Groups, etc.
What’s also interesting is that, given the current cloud environments, why none of them publish an “equivalence” list; that is, a list that maps what the cloud environment offers to the equivalent physical spec. For example, 1 small EC2 instance = .02 physical (dedicated). From there it is simple math to get the performance you want to pay for, plus it blatantly spells out the performance divergence (instead of letting folks realize it through arduous testing and lots of “WTF” moments).
I think you mean “I WAS BLOWN AWAY!!!”
lol indeed. fixed.
One would purchase EC2 for its cloud capability(on the fly scalability).
Dedicated servers would no way match the huge loads that cloud can handle, what say?
Maybe a different type of test might make sense: real traffic.
What I mean: with a dedicated box, you almost never max out the CPU, unless you run a very high traffic site. Usually in web serving it’s the disk IO that gets maxed out. So Amazon’s cloud may be optimized to a specific type of workload, one which synthetic test don’t necessarily reproduce. It would be easy for Amazon to build an offering where you get 4 cores for each GB, but does it make sense?
Defensio actually does > 50 million requests per day on EC2 and we use way too many machines for what it is.
Amazon is definitely NOT optimized for IO. IO is actually horrible on EC2. Terrible.
BTW, they don’t publicly advertise this, but if you ask nicely, you can also get a private network (dedicated gigabit switch, extra NICs in the servers) from hetzner at no additional monthly charge (just a setup fee). i repeat: free gigabit private network, with stellar ping times:
ubuntu@panama:~$ ping 192.168.1.199
PING 192.168.1.199 (192.168.1.199) 56(84) bytes of data.
64 bytes from 192.168.1.199: icmp_seq=1 ttl=64 time=0.062 ms
64 bytes from 192.168.1.199: icmp_seq=2 ttl=64 time=0.052 ms
64 bytes from 192.168.1.199: icmp_seq=3 ttl=64 time=0.052 ms
that is also (at least) 10x better than what you typically see at amazon.
This is really great information Tim, thanks!
I can’t wait to move my stuff to my new EX 4S. My box at Joe’s Datacenter was actually very decent, but my project is seeing a lot more load/traction than expected, so Hetzner will be perfect.