"Fluentd: Open Source Log Management" by Sadayuki Furuhashi, Treasure Data at SF Metric Meetup

Sadayuki Furuhashi talks about Fluentd: Open Source Log Management from Librato on Vimeo.

This is a participation report for the SF Meetup. This meetup is probably data analysis for server administration. A log management is really far from my current job but I have been interested in fluentd since I knew that it is good approach to make data collection easy. And this is a good chance to hear about it from the founder. So, I participated in this.

About SF Metric Meetup

In the 21st Century successful teams are data-driven. This monthly meetup provides a forum for monitoring geeks to gather and trade new ideas, data, and war stories. If you love data, this meetup is for you! (cited from meetup.com) 

About the speaker

Sadayuki Furuhashi is an architect and founder of Treasure Data. He is also a founder of some open source projects including fluentd.

About fluentd

Fluentd is extensive log management tool written by Ruby. Nowadays, the major web tech companes produce millions of lines of logs every day and their analysis sever operational status with the logs. Fluentd makes this collection/shape/analysis process easier. 

What they tried to do in this meetup

This time he came to the meetup with his coworkers of Treasure Data. I talked with some his coworkers about their business. They are trying to make his open source projects, including Fluentd, MessagePack popular through some meetups, before providing charged service. I realized that it is important to have this kind of networking opportunity and talk about own products to potential customers. I keep paying attention to their products and services.

Apache Lucene: Then and Now, Java User Group meetup at Twitter HQ

Meetup.com - Apache Lucene: Then and Now

This time, Doug Cutting (@cutting) talked about the history of Apache Lucene and how Apache Lucene is used for the implementation of Internet search engines and local, single-site searching in major tech companies like Linkedin or Twitter. He also mentioned this project is integrated with Hadoop and still evolving.

Doug Cutting (@cutting) is the founder of numerous successful open source projects, including Lucene, Nutch, Avro, and Hadoop. Doug joined Cloudera in 2009 from Yahoo!, where he was a key member of the team that built and deployed a production Hadoop storage and analysis cluster for mission-critical business analytics. Doug holds a Bachelor’s degree from Stanford University and sits on the Board of the Apache Software Foundation. (cited from Meetup.com) 

His talk is based on the following blog entries.

• Cloudera Search: The Newest Hadoop Framework for CDH Users and Developers
• The Blur Project: Marrying Hadoop with Lucene

As you see, his roles in his company (Cloudara) is the implementation of the new open source project, Blur.

Blur is an Apache Incubator project that provides distributed search functionality on top of Apache Hadoop, Apache Lucene, Apache ZooKeeper, and Apache Thrift. When I started building Blur three years ago, there wasn’t a search solution that had a solid integration with the Hadoop ecosystem. Our initial needs were to be able to index our data using MapReduce, store indexes in HDFS, and serve those indexes from clusters of commodity servers while remaining fault tolerant. Blur was built specifically for Hadoop — taking scalability, redundancy, and performance into consideration from the very start — while leveraging all the great features that already exist in the Hadoop stack. (cited from blog.cloudera.com)

And Cloudera is providing a better way for non-programming users interact with Hadoop data.

In the context of our platform, CDH (Cloudera’s Distribution including Apache Hadoop), Cloudera Search is another framework much like MapReduce and Cloudera Impala. It’s another way for users to interact with Hadoop data and for developers to build Hadoop applications. Each framework in our platform is designed to cater to different families of applications and users (cited from blog.cloudera.com)

 See Cloudera blog for more details.

It seems that there are meetup of Java user group in SF once or twice a month. I am planning to continue joining meetup.