Apache Lucene: Then and Now, Java User Group meetup at Twitter HQ

Meetup.com - Apache Lucene: Then and Now

This time, Doug Cutting (@cutting) talked about the history of Apache Lucene and how Apache Lucene is used for the implementation of Internet search engines and local, single-site searching in major tech companies like Linkedin or Twitter. He also mentioned this project is integrated with Hadoop and still evolving.

Doug Cutting (@cutting) is the founder of numerous successful open source projects, including Lucene, Nutch, Avro, and Hadoop. Doug joined Cloudera in 2009 from Yahoo!, where he was a key member of the team that built and deployed a production Hadoop storage and analysis cluster for mission-critical business analytics. Doug holds a Bachelor’s degree from Stanford University and sits on the Board of the Apache Software Foundation. (cited from Meetup.com) 

His talk is based on the following blog entries.

• Cloudera Search: The Newest Hadoop Framework for CDH Users and Developers
• The Blur Project: Marrying Hadoop with Lucene

As you see, his roles in his company (Cloudara) is the implementation of the new open source project, Blur.

Blur is an Apache Incubator project that provides distributed search functionality on top of Apache Hadoop, Apache Lucene, Apache ZooKeeper, and Apache Thrift. When I started building Blur three years ago, there wasn’t a search solution that had a solid integration with the Hadoop ecosystem. Our initial needs were to be able to index our data using MapReduce, store indexes in HDFS, and serve those indexes from clusters of commodity servers while remaining fault tolerant. Blur was built specifically for Hadoop — taking scalability, redundancy, and performance into consideration from the very start — while leveraging all the great features that already exist in the Hadoop stack. (cited from blog.cloudera.com)

And Cloudera is providing a better way for non-programming users interact with Hadoop data.

In the context of our platform, CDH (Cloudera’s Distribution including Apache Hadoop), Cloudera Search is another framework much like MapReduce and Cloudera Impala. It’s another way for users to interact with Hadoop data and for developers to build Hadoop applications. Each framework in our platform is designed to cater to different families of applications and users (cited from blog.cloudera.com)

 See Cloudera blog for more details.

It seems that there are meetup of Java user group in SF once or twice a month. I am planning to continue joining meetup.

JSON Lint - validation tool for JSON

There are several validation tools for JSON. These tools help you to write and test JSON data or schema.

• JSONLint
• This tool validates that JSON data is valid or not.
• It does not deal with JSON schema.
• JSON Schema Lint
• This tool validates
• JSON schema is valid
• JSON data is valid
• JSON data is valid against JSON schema

INSERT VS COPY: The fastest way to do a bulk insert into PostgreSQL

It takes time to populate lots of data in database by many "INSERT" statements. The official PostgreSQL documentation mentions the topic.

PostgreSQL 9.3.2 Documentation Chapter 14. Performance Tips 14.4. Populating a Database

The chapter shows many options, but I am paying attention to first two.

Disable autocommit

PostgreSQL commits each statement automatically. That means if you run an INSERT statement, it runs like this for each statement:

1. Open a transaction
2. Insert data
3. Close the transaction

It is redundant. In this case, it becomes faster if you use autocommit like this:

BEGIN; -- the beginning of the transaction
INSERT xxxx
INSERT xxxx
END;    -- the end of the transaction

PostgreSQL does not commit the statements between BEGIN and END.

COPY

COPY is also to use populate data, but the different point is that COPY does not offer autocommit. It means that PostgreSQL commit after all data is populated on a database.
Please refer to the document about how to use COPY statement.

Auto increment primary key in PostgreSQL

If you want to increment primary key automatically in Postgres, "serial" is a good option.

• PostgreSQL 9.3.2 Documentation Chapter 8. Data Types 8.1.4. Serial Types 
• Stackoverflow - Set auto increment primary key in Postgresql

Please that you do not need to specify a primary key to insert data to your database.

How to fix an error "Unable to find a $JAVA_HOME at “/usr”, continuing with system-provided Java"

Somehow, I have got this error whenever I run Java command:

$java -version
Unable to find a $JAVA_HOME at "/usr", continuing with system-provided Java...
java version "1.7.0_25"
Java(TM) SE Runtime Environment (build 1.7.0_25-b15)
Java HotSpot(TM) 64-Bit Server VM (build 23.25-b01, mixed mode)

I found the same error as my case at the stackoverflow.

Unable to find a $JAVA_HOME at “/usr”, continuing with system-provided Java

This happens when $JAVA_HOME is not set correctly. In my case, the following works form me.

export JAVA_HOME="$(/usr/libexec/java_home)"

My environment is as follows:

• OS X 10.9
• Java 1.7.0_25 (installed by brew)

Install R with homebrew under Montain Lion (Mac OS X 10.8.X)

You need to install gfortran before R. 

$ brew install gfortran

You might find that you don't have XQuartx. Please download and install it at https://xquartz.macosforge.org.

$ brew install R
Unsatisfied dependency: XQuartz
Homebrew does not package XQuartz. Installers may be found at:
https://xquartz.macosforge.org
Error: An unsatisfied requirement failed this build.

After you install XQuartz, you can install R like this:

$ brew install R

Ant 1.8 "warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds"

I installed Ant 1.8 and build Java applications with Ant. Then, I found the following message:

warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds

You can find a good explanation of this warning with following link:

http://ant.1045680.n5.nabble.com/warning-includeantruntime-was-not-set-td2639463.html

There were incompatible change on Ant version 1.8. So, you may find the same message if you build old ant file with Ant 1.8.

Please just add includeantruntime="false" to javac task.