Comments on Tom White: RPC and Serialization with Hadoop, Thrift, and Pro...

Has anybody tried Etch?http://developer.cisco.com/...

2008-08-19T22:53:00.000+01:00

Has anybody tried Etch?

http://developer.cisco.com/web/cuae/etch
http://developer.cisco.com/web/cuae/devconf2008_session_3

Thanks Jeff,Yes, you're right that Thrift provides...

2008-07-08T20:18:00.000+01:00

Thanks Jeff,

Yes, you're right that Thrift provides more in the way of providing code for doing network services. I didn't know that you could use Thrift transport with Protocol Buffers - that's pretty neat.

I'm sure I heard Google say that there isn't RPC support in Protocol Buffers, but there does seem to be some RPC Java-related classes in there. Haven't tried it though.

And well done on making Hive support lots of serialization formats. As I said, I think we need to support all of them, and let people choose what they want to use.

hey tom,good post, this is an area in need of clar...

2008-07-08T20:00:00.000+01:00

hey tom,

good post, this is an area in need of clarification.

protocol buffers and record io are merely data exchange formats, while thrift includes a lot more.

in addition to a data exchange format, thrift includes a code generator, networking transport code, and a suite of robust server skeleton implementations in a variety of languages. it should be noted that you can use the servers and networking code to exchange data in any format, including protocol buffers.

there's also been a lot of debate about serialization formats for putting data on the wire versus putting data on disk.

for the latter case, we've built support into hive for general serialization formats, so while a lot of our data is serialized in thrift's format, we can support record io, protocol buffers, csv, or whatever.