Comments on Tom White: Elastic Hadoop Clusters with Amazon's Elastic Bloc...

Any more insight on this front (Hadoop on EC2 with...

2010-04-06T23:00:34.785+01:00

Any more insight on this front (Hadoop on EC2 with EBS or not)?

I just finished reading your book and about to finish up my Cloudera training in NY (tomorrow) and looking to setup Hadoop in our production network as well as an instance on EC2 (which I need to have persistant).

I saw a beta [maybe ok for our AWS deploy] with EBS on the Cloudera website for persistance and was not sure that was the latest and greatest?

Tom, I am looking heavily into EBS volumes. Quest...

2009-09-23T15:51:15.256+01:00

Tom,

I am looking heavily into EBS volumes. Question. An EBS volume is in itself RAIDED. Typically hadoop datanodes and taskstrackers are suggested with no RAID and individual disks. What is the EBS suggestion? If you need 800GB of storage should you construct 8 10GB volumes or 1 800GB or 2 400TB?
Is there a way of knowing where the EBS volume physically lives?

Thanks Edward

Jerry,I was referring to the arbitrary user data y...

2009-01-30T21:02:00.000Z

Jerry,

I was referring to the arbitrary user data you can pass to an EC2 instance on boot, and whether that would work as a way of telling the instance which volume to attach. Another way would be to attach the volume to the instance from the client.

Tom

Thank you, Tom for this post. It is very helpful. ...

2009-01-29T05:18:00.000Z

Thank you, Tom for this post. It is very helpful.

The only thing I am confused about is how to tell an instance what EBS volume it should use. You mentioned "user data" yet I couldn't find any documentation on how to specify the EBS volume. Can you kindly shed some light on that?

Thanks,
Jerry Lewis

Perhaps a new section for the book?I just bought t...

2008-09-30T23:05:00.000+01:00

Perhaps a new section for the book?

I just bought the 'rough drafts' on Safari -- keep the revisions coming.

Yousef,Hmm, you're right. I think someone needs to...

2008-09-12T20:49:00.000+01:00

Yousef,

Hmm, you're right. I think someone needs to implement Hadoop on EBS and run a benchmark to compare it to EC2 local disks.

You do realize that these two statements are contr...

2008-09-07T02:21:00.000+01:00

You do realize that these two statements are contradictory, right?

EBS does not have the bandwidth of local disks

...

The bottom line though is that performance exceeds what we’ve seen for filesystems striped across the four local drives of x-large instances.

Thanks Tom, this is very timely and helpful.

2008-08-26T03:14:00.000+01:00

Thanks Tom, this is very timely and helpful.

EBS is replicated, so HDFS replication is redundan...

2008-08-24T10:53:00.000+01:00

EBS is replicated, so HDFS replication is redundant. One could simply set the replication level to one, to inhibit HDFS replication, but then if an EC2 instance were to crash files would be incomplete. One could instead set the replication level to two, to reduce redundant replicaton, or perhaps leave replication at one, closely monitoring datanode instances and re-mount volumes elsewhere when one dies.