Comments on Tom White: Hadoop and Log File Analysis

Hey Tony, here's a fairly easy doc with how to...

2011-06-01T17:58:10.046+01:00

Hey Tony, here's a fairly easy doc with how to load your system and app logs into Hadoop (Apache Hive) for SQL-style analysis:
Log analytics with Hadoop and Hive

The doc is written for customers of our hosted "log aggregator in the cloud" service, Papertrail, but it covers how the data gets there, the formatting (TSV), and Hive's LOAD DATA INFILE process.

We've been spoiled by Amazon Elastic MapReduce, but Cloudera's distro is very powerful and would work fine.

Whether or not you're logging to Papertrail, something like that would be my recommendation for reporting.

At Logg...

2010-04-03T20:23:23.834+01:00

At Loggly we are building a cloud-based log management an analysis platform. You will not only be able to run large-scale processes (through map-reduce), but you can do much more based on our real-time indexing and data processing capabilities. In order to enable you to build your own use-cases around your log data, we provide an API that you can use to access all the data that you have Loggly manage for you. Would be curious to get your feedback!

Tom, what would be needed for an enterprise log ma...

2010-03-09T16:26:51.381Z

Tom, what would be needed for an enterprise log management system using Hadoop? Can it source any log type including custom? Do I need to build the interfaces? Does Hadoop include a query reporting system? Otherwise, what are your recommendations for query/reporting?

Shortly after I wrote this, High Scalability covered the story with extra details from Mailtrust CTO Bill Boebel. Well worth a read.