Comments on Tom White: "Disks have become tapes"

sameerds:rate = size/timefile size is not a consta...

2008-04-02T04:26:00.000+01:00

sameerds:

rate = size/time

file size is not a constant over disk generations

vicaya:with current wear leveling techniques, mode...

2008-04-02T04:21:00.000+01:00

vicaya:

with current wear leveling techniques, modern (high end) flash drives has a roughly equivalent MTBF of hard drives. That will only go up with time.

The text is totally confused about the terms "foob...

2008-03-21T05:06:00.000Z

The text is totally confused about the terms "foobar rate" and "foobar time". At one point the text says "transfer rate is increasing" and somewhere else it says "transfer time is increasing". It would seem to me, that if time and rate would be inversely proportional to each other, and definitely not interchangeable like this.

SSD does not compete with disk but live as part of...

2008-03-21T02:31:00.000Z

SSD does not compete with disk but live as part of the cache hirerchy (L1->L2->RAM->SSD->DISK->TAPE). They actually solve a very real problem (at X10 price). As an analogy: going from RAM->DISK feels like traveling 100 miles back and forth each time you need some book from your library, while accessing SSD should feel like traveling a mile.

I dont believe that SSDs will catch up to disks an...

2008-03-20T18:29:00.000Z

I dont believe that SSDs will catch up to disks anytime in the next 5 years. And if they do then people will invent aplications that need more storage, magnetic being cheaper than solid state. For example, there is no way that I can fit my music collection on any SSD (> 256GB), and it is still growing, and I still have not started with video, and my digital pics are 10 Megapixels now. Sure enough, the Seagate drives I use to store my MP3 feel slower with every new increase in capacity. Finally, I work in IT at a big firm. It used to be the case that 2GB file size limit was OK, our customers didn't seem to want to keep so m uch data around. Of course, that was then, this is now: 4GB files are the norm (we built our own tiny DB access library for these using LFS under Linux).

vicaya...Disk are never accessed per record. The s...

2008-03-20T15:34:00.000Z

vicaya...

Disk are never accessed per record. The smallest unit of I/O for direct I/O is a page size (8K).
Also, The N in N way merge is usually much larger than 10 -
In the sort phase, you make runs in the size of your memory which is, for example, 4G. So if you have 1TB file, you would have 250 runs.
Also, I am not talking about BTREE. BTREE are good for few record lookup but are bad for large record scans (Unless the BTREE covers your search keys, in which case you can scan the BTREE).

I do not understand how SSD are more benefical to mapreduce than to RDBMS. The key here is to look at the I/O patterns of the alg and not at the specific hardware.

To follow up on what I said earlier:Yes, deteriora...

2008-03-20T13:46:00.000Z

To follow up on what I said earlier:

Yes, deterioration is a concern. That's why I said 3-5 years to become standard equipment so those issues can be resolved / improved. Besides, it is the 'data'. not the drives that hold the value. Your data is backed up. right?

The other benefit that SSD devices offer is that their reads (transfer / seek) are consistent across the entire chip. Hard drives are not, transfer / seek times vary according to where on the platter you are reading from.

Interesting stuff though.

What about the deterioration of SSDs. As I unders...

2008-03-20T11:57:00.000Z

What about the deterioration of SSDs. As I understand it, there is a finite amount of flippability for the bits, so SSD is better for something where you are going to write less frequently and read more often.

@2 regarding external sort/merge, N way merging (w...

2008-03-20T10:50:00.000Z

@2 regarding external sort/merge, N way merging (when N typically < 10) doesn't incur a seek per record but per buffered read per spindle, while typical BTree nodes are much smaller than the read buffers of these streams. So BTree indeed incur many more seeks when read/write large tables.

Also map-reduce typically runs on a DFS, which means indeed that the streams are more likely from many spindles.

That's how Bigtable and similar new generation DBs like Hypertable (and theoretically HBase) can achieve disk transfer rate even for random writes to petabyte sized table.

Flash SSD is an interesting diversion, for read-mostly apps, as frequent writes can wear it out quickly.

Hopefully MRAM will save the day, which is about 10 years away to reach the feasible cost and density.

Reddit comments.

2008-03-20T10:42:00.000Z

Reddit comments.

Uh, exactly what the fuck does all this MEAN?Is it...

2008-03-20T06:08:00.000Z

Uh, exactly what the fuck does all this MEAN?

Is it a translation of the original Sanskrit, or what?

eric14!

2008-03-20T05:45:00.000Z

eric14!

Not to mention that in 3-5 years SSD drives will b...

2008-03-20T04:38:00.000Z

Not to mention that in 3-5 years SSD drives will be standard equipment. Their random vs. sequential reads are about equal.

Perhaps the idea of drives = tapes is applicable in the sense that drives with their hugely increasing capacities will turn into more of a backup medium.

You are simply not correct. I suggest that you rea...

2008-03-20T02:24:00.000Z

You are simply not correct. I suggest that you read more on how disk based sorting alg works. Each disk based sorting alg have a sort phase (run formation) and a merge phase (sorting the differnt runs). However, during the merge phase the alg does not know where the smallest number will reside, hence it must do a seek to read small amount of I/O from each run file (this is the N way merge). So unless you store each run file on a seperate disk you will do a seek. The same alg will be used on any external disk based sorting alg regardless of its name (RDBMS or map reduce)

IOPs have gone up two orders of magnitude in the p...

2008-03-20T00:41:00.000Z

IOPs have gone up two orders of magnitude in the past three years. Currently transfer rates for SSD are usually below that of top grade hard drives, but that will surely change. HD's have been "just above" 100MBps for quite some time and not making dramatic shifts, they will certainly be overtaken.

Are hard drives out? Not at all, for now their cost & data density is unparalleled. But I think the MapReduce situation changes considerably when you can think about chunking half a terabyte datasets onto very-high-IOPs nearline storage. Theres still a role for MapReduce to linearly chunk and streaming data onto SSD, but the chunk-size mapreduce will deal with will be able to be far bigger, giving new life to random access.

Which is a good thing, because as great as mapreduce is theres a lot of places where random access is unavoidable. My concurrency professor liked to say that there were a lot of problems that were "embarrassingly parallel," and MapReduce is a wonderful way to chunk up these jobs and distribute them, but not all problems can localize and subdivide themselves so.