Comments on: Hadoop Digest, February 2010

By: Alex Baranau

Alex Baranau — Tue, 13 Jul 2010 05:08:41 +0000

I think I know what the question is about.

In case the split ends with only some part of the record, the framework (this is up to InputFormat actually) requests for additional bytes (which are in the beginning of the next split). The call is made over HTTP and it usually transmits very little piece of data so, given the size of each split (usually at least 64Mb) it doesn’t affect the performance at all even if the next split is on another physical box.

I hope I answered your question.

By: sematext

sematext — Tue, 13 Jul 2010 03:37:07 +0000

I’m not sure which part of the post you are referring to, but this sounds like a question for the mailing list.
Alternatively, have you looked for the answer on http://www.search-hadoop.com/ ?

By: yuanke

yuanke — Tue, 13 Jul 2010 02:59:49 +0000

how the framework makes sure complete records are presented in ?could you give a more detailed description? all you said I know.thanks

By: slim tebourbi

slim tebourbi — Sat, 20 Feb 2010 09:03:42 +0000

great work. Thanks.

By: Jakob Homan

Jakob Homan — Fri, 19 Feb 2010 05:37:37 +0000

Otis-
This is an excellent resource; thank you very much for compiling this information. Very much appreciated.

Thanks,
Jakob Homan
Hadoop @ Yahoo!

By: Otis Gospodnetic

Otis Gospodnetic — Thu, 18 Feb 2010 13:06:04 +0000

Hi Todd. I didn’t personally put this one together. But yes, digest posts will be monthly. Thanks for the feedback!

By: Todd Lipcon

Todd Lipcon — Thu, 18 Feb 2010 08:14:14 +0000

Hey Otis

Thanks for writing this up! Will this be monthly? Should be a great resource for the community.

-Todd