Friend to friend car sharing service implementation with Kinoma Create

 I joined AT&T Hackathon @ Super Mobility Week - Code for Car & Home at Las Vegas on September 6-7 with my two other team members Rama and Kishore. This hackathon is aimed towards developing new applications for connected car and home industry by utilizing AT&T api’s and IoT hardware and software provided by multiple vendors present here. In the hackathon, we tried to make a friend to friend car sharing system which is more secure and interactive than existing services. We intended to use Kinoma Create platform as in car unit interface that interacts with a driver by visual and audio information. We joined Kinoma Create meetup before the hackathon and Kinoma team lend us their pre-release version to be used with our Hackathon. So, we would like to share our experience in this entry.

Our motivation for the AT&T hackathon

Most people in United states use their car less than 2 hours per day, that is less than 10% utilization rate. At the same time when you need an additional car for a weekend trip or travel outside of home town it is expensive to rent a car. Our concept takes advantage connected car technology to easily share this underutilized car between friends and family. Both parties in this transaction will be benefited as borrower can use car for fraction of rental car cost while owner can use incentive (monetary or points) from this transaction for his future travel. As new P2P car sharing application like getaround, relayrides are getting popular but they will not be useful for sharing cars with friends, family and neighbors. Since these P2P companies takes a portion of rental cost they will not suit sharing scenarios where friends and family will use each others car. In our scenario, we are not considering monetary incentive but this can be easily added as form of points or any other form to our system.

Our major motivation in this project is to develop an application that can help a friend, trusted neighbor or family member to easily request, safely use and return car as per agreed schedule. At the same time our application also aims to provide car owner with tools to learn about borrowers past driving score, giving guidelines about his car usage and report on how his car was handled by borrower. To achieve this we developed friend-to-friend car sharing application to solve technical challenges in car sharing scenarios as discussed here.

Technical challenges for friend to friend car sharing

Next, we explain technical challenges for the car sharing that we tried to solve in the AT&T hackathon based on the scenario above. We tried to solve common issues we see when we share own cars with friends, family and neighbors.

Owner’s problem

• (security) It is difficult to give car keys to borrowers if owner is not available at the time of pickup.
• (1) Actual world authentication
• If the owner is not available at the time of pickup, the owner can’t check borrower’s ID and the borrower lose a chance to share his car. To make a sharing process easy, the owner needs not only online authentication at a time of reservation but also actual world authentication that works remotely.
• (2) Electronic keys for car and garage 
• If the owner is not available at the time of pickup, the owner also can’t give keys to the borrower. The owner need a way to provide electronic keys for car and garage from remote.
• (safety) Owner wants his car to be driven without aggressive driving and return on schedule without any problems.
• (3) reinforce the borrower with good driving behavior
• Even though the borrower is reliable person, the borrower might not be safe driver. The owner might want to have a way to monitor the borrower’s driving and reinforce the borrower with good driving behavior. 

Borrower’s problem

• (usability) It may be difficult for borrowers to drive this car without knowing operation details or local driving etiquette.
• (4) Less distractive car operation details for a car
• It is common that we don’t know how to operate a car if we rent different type of car from own car. The borrower wants to operate the car with the same way as his own car.

System overview

Our solution aims to provide two main features for car sharing. First it support automatic borrower authentication through in-home unit. Our in-home unit detects proximity of borrower by bluetooth beacon technology and enables him to access car parked inside a garage. Second we provide authentication and driving related information through in-car unit.

Why we chose Kinoma Create

We planned to use Arduino and Raspberry Pi to implement the in-car unit. Although, when we joined the Kinoma Create meetup, we found out that it supports the following requirements and it is easier to implement the in-car unit. So we decided to use it for the in-car unit.

• Built in network functionality to communicate with server and mobile application
• Display and touch interface to implement interactive user interface for a driver

Our impression on Kinoma Create

Kinoma platform is helpful for us to quickly develop a simple prototype for our application due to its hardware programming capabilities, built in connectivity and sample application provided by Kinoma team. Developing application on Kinoma studio is straightforward and easy to port completed application to platform. Programmers who are familiar with Java Script may find it easy to write new applications while Hardware programmers like flexibility of programmable pins. Their examples need to be expanded for network and protocol applications but with little effort we were testing our first applications.

Conclusion

We defined and created a simple car sharing application for friends, family and neighbors. We used multiple hardware platforms. We used Kinoma Create to build our in-car unit quickly with built in network functionality and display and touch interface. Through the prototyping, we saw potential of Kinoma Create for mainly web developer to build IoT application rapidly.

Acknowledgment

Thank you Kinoma Team for providing pre-release version! Since Kinoma Create is nice product for Javascript programmers to take a step into IoT world. We believe this product will become successful in IoT industry.

Stepping into a pattern for handling asynchronous operations in Java Script called "JavaScript Promise"

I have been currently working on Parse.com and Automatic for my prototyping. I had a problem communication between Parse and Automatic web service through HTTP.

Here is my question on stackoverflow.

Parse.Cloud.httpRequest call with HTTP request header
The point is that I sent an HTTP request but any callback functions are not called.

According to the suggested solution, I found the following two mistakes in my code.

• Wrong callback function
• I did not respond to a response from a server 

Thanks to the solution, I could learn how to write a callback function for  asynchronous operations in JavaScript.

This is a pattern for handling asynchronous operations in JavaScript that you can see in JavaScript libraries like AngularJS. It is called "JavaScript Promises".

I stepped into the very beginning of asynchronous operations in JavaScript this time. If I found something interesting, I will write it on this blog.

Completed "Introduction to Databases" of Stanford Univeristy with "distinction"

I completed "Introduction to Databeses" of Stanford Univeristy with "Distinction", Yeah! This entry explains how this online course looks like.

Why did I participated in this course?

I was embedded software engineer and seldom use databases. Although, I was transferred to the Business Intelligence company as temporally trainee last October. Then, I often use several relational databases recently. So, I became to think I need to study database again. At that time, I found this course. Studying with book alone is a little bit hard, So I took this course.

Topics that this course covers

As I expected this course focuses on relational database theory, including SQL or table design, but they also cover non relational database including XML and JSON.

• Relational Database Overview
• XML Data
• JSON Data
• Relational Algebra
• SQL
• Relational Design Theory
• Querying XML
• Unified Modeling Language
• Indexes
• Transactions
• Constraints and Triggers
• Views
• Authorization
• Recursion
• Online Analytical Programming
• NoSQL Systems

How to participate in this course

In this course, every material became public from the beginning. You can start from any topic. Although, deadline of each assignment work is different. So, you should pick a topic that deadline is close first.

For each topic, you repeat the following procedure:

• Watch some short videos that explain topics
• Answer simple quizes（No score here）
• Try to basic problems
• multi-selection question（low score）
• Programming work（middle score）
• Try to challenge problems
• Advanced programming work（high score）

Certification

When you answer each question, it is automatically scored. You can see how much percentage of scores you already got. If you solve all basic problems, you get 50 % and can complete the course. If you also try challenge problems and get 75 %, you can get "State of Accomplishment with Distinction".

Oh, everything is English!

It is really hard point for Japanese people. I actually worked in the office where no one speaks in English but now my all coworkers speak in English. So, no problems with this for me now.

Although, I felt difficulty when a teacher talked really fast about a topic that I was not familiar with. At the time, I had to watch the videos again and again. As a result, I tried to understand by solving some problems. Or sometimes I research some Japanese materials related to the topics.

How hard it was?

It depends on your skill and knowledge. It seems that it takes 3 hours every week to complete the course. You need 7 to 8 hours every week to complete with distinction. I sometimes had to study all day on Saturday or Sunday. If you work full time, you cannot take more than one course at the same time.

The important point to take an online course

If every material becomes public from the beginning, you should consider your private schedule and make a concrete schedule for the course. You might be struggling with difficult problems, so you should have some time to spare.

You should not postpone challenge problems even though their deadlines are late because you might forget about details when you are back to the problems :)

--

Have fun with an online course!!

"Fluentd: Open Source Log Management" by Sadayuki Furuhashi, Treasure Data at SF Metric Meetup

Sadayuki Furuhashi talks about Fluentd: Open Source Log Management from Librato on Vimeo.

This is a participation report for the SF Meetup. This meetup is probably data analysis for server administration. A log management is really far from my current job but I have been interested in fluentd since I knew that it is good approach to make data collection easy. And this is a good chance to hear about it from the founder. So, I participated in this.

About SF Metric Meetup

In the 21st Century successful teams are data-driven. This monthly meetup provides a forum for monitoring geeks to gather and trade new ideas, data, and war stories. If you love data, this meetup is for you! (cited from meetup.com) 

About the speaker

Sadayuki Furuhashi is an architect and founder of Treasure Data. He is also a founder of some open source projects including fluentd.

About fluentd

Fluentd is extensive log management tool written by Ruby. Nowadays, the major web tech companes produce millions of lines of logs every day and their analysis sever operational status with the logs. Fluentd makes this collection/shape/analysis process easier. 

What they tried to do in this meetup

This time he came to the meetup with his coworkers of Treasure Data. I talked with some his coworkers about their business. They are trying to make his open source projects, including Fluentd, MessagePack popular through some meetups, before providing charged service. I realized that it is important to have this kind of networking opportunity and talk about own products to potential customers. I keep paying attention to their products and services.

Remote debugging for Tomcat and IntellJ on Max OS X

IntellJ supports remote debugging for Tomcat. It is really convenient for debugging. This entry shows how to adopt remote debugging with Tomcat and IntellJ.

My dev environment

• Max OS X 10.9.2
• Java 1.8.0
• Tomcat 8.0.3

Set up

Tomcat

1. Create a new file called <Tomcat install directory>/bin/setenv.sh. (This file will be automatically read by catalina.sh)
2. Add the following content to setenv.sh. 8080 is a port number for Tomcat. Please specify for your tomcat.
• CATALINA_OPTS="-agentlib:jdwp=transport=dt_socket,address=8080,server=y,suspend=n"
3. Start Tomcat normally using startup.sh.

IntellJ

1. Select "Run -> Edit Configuration" then "Run/Configuration" screen is open.
2. Specify "Name" like "Tomcat".
3. Specify "Port", which is the same number as Tomcat's port numer

Debug

1. If you want to start debugging on IntellJ, just select "Run -> debug 'Tomcat'". 
2. If IntellJ shows a message like below, you are already connected.
• Connected to the target VM, address: 'localhost:8080', transport: 'socket'
3. Now you can debug. Please refer here for more detail about debugging by IntellJ.

Apache Lucene: Then and Now, Java User Group meetup at Twitter HQ

Meetup.com - Apache Lucene: Then and Now

This time, Doug Cutting (@cutting) talked about the history of Apache Lucene and how Apache Lucene is used for the implementation of Internet search engines and local, single-site searching in major tech companies like Linkedin or Twitter. He also mentioned this project is integrated with Hadoop and still evolving.

Doug Cutting (@cutting) is the founder of numerous successful open source projects, including Lucene, Nutch, Avro, and Hadoop. Doug joined Cloudera in 2009 from Yahoo!, where he was a key member of the team that built and deployed a production Hadoop storage and analysis cluster for mission-critical business analytics. Doug holds a Bachelor’s degree from Stanford University and sits on the Board of the Apache Software Foundation. (cited from Meetup.com) 

His talk is based on the following blog entries.

• Cloudera Search: The Newest Hadoop Framework for CDH Users and Developers
• The Blur Project: Marrying Hadoop with Lucene

As you see, his roles in his company (Cloudara) is the implementation of the new open source project, Blur.

Blur is an Apache Incubator project that provides distributed search functionality on top of Apache Hadoop, Apache Lucene, Apache ZooKeeper, and Apache Thrift. When I started building Blur three years ago, there wasn’t a search solution that had a solid integration with the Hadoop ecosystem. Our initial needs were to be able to index our data using MapReduce, store indexes in HDFS, and serve those indexes from clusters of commodity servers while remaining fault tolerant. Blur was built specifically for Hadoop — taking scalability, redundancy, and performance into consideration from the very start — while leveraging all the great features that already exist in the Hadoop stack. (cited from blog.cloudera.com)

And Cloudera is providing a better way for non-programming users interact with Hadoop data.

In the context of our platform, CDH (Cloudera’s Distribution including Apache Hadoop), Cloudera Search is another framework much like MapReduce and Cloudera Impala. It’s another way for users to interact with Hadoop data and for developers to build Hadoop applications. Each framework in our platform is designed to cater to different families of applications and users (cited from blog.cloudera.com)

 See Cloudera blog for more details.

It seems that there are meetup of Java user group in SF once or twice a month. I am planning to continue joining meetup.

JSON Lint - validation tool for JSON

There are several validation tools for JSON. These tools help you to write and test JSON data or schema.

• JSONLint
• This tool validates that JSON data is valid or not.
• It does not deal with JSON schema.
• JSON Schema Lint
• This tool validates
• JSON schema is valid
• JSON data is valid
• JSON data is valid against JSON schema

INSERT VS COPY: The fastest way to do a bulk insert into PostgreSQL

It takes time to populate lots of data in database by many "INSERT" statements. The official PostgreSQL documentation mentions the topic.

PostgreSQL 9.3.2 Documentation Chapter 14. Performance Tips 14.4. Populating a Database

The chapter shows many options, but I am paying attention to first two.

Disable autocommit

PostgreSQL commits each statement automatically. That means if you run an INSERT statement, it runs like this for each statement:

1. Open a transaction
2. Insert data
3. Close the transaction

It is redundant. In this case, it becomes faster if you use autocommit like this:

BEGIN; -- the beginning of the transaction
INSERT xxxx
INSERT xxxx
END;    -- the end of the transaction

PostgreSQL does not commit the statements between BEGIN and END.

COPY

COPY is also to use populate data, but the different point is that COPY does not offer autocommit. It means that PostgreSQL commit after all data is populated on a database.
Please refer to the document about how to use COPY statement.

Auto increment primary key in PostgreSQL

If you want to increment primary key automatically in Postgres, "serial" is a good option.

• PostgreSQL 9.3.2 Documentation Chapter 8. Data Types 8.1.4. Serial Types 
• Stackoverflow - Set auto increment primary key in Postgresql

Please that you do not need to specify a primary key to insert data to your database.

How to fix an error "Unable to find a $JAVA_HOME at “/usr”, continuing with system-provided Java"

Somehow, I have got this error whenever I run Java command:

$java -version
Unable to find a $JAVA_HOME at "/usr", continuing with system-provided Java...
java version "1.7.0_25"
Java(TM) SE Runtime Environment (build 1.7.0_25-b15)
Java HotSpot(TM) 64-Bit Server VM (build 23.25-b01, mixed mode)

I found the same error as my case at the stackoverflow.

Unable to find a $JAVA_HOME at “/usr”, continuing with system-provided Java

This happens when $JAVA_HOME is not set correctly. In my case, the following works form me.

export JAVA_HOME="$(/usr/libexec/java_home)"

My environment is as follows:

• OS X 10.9
• Java 1.7.0_25 (installed by brew)

Install R with homebrew under Montain Lion (Mac OS X 10.8.X)

You need to install gfortran before R. 

$ brew install gfortran

You might find that you don't have XQuartx. Please download and install it at https://xquartz.macosforge.org.

$ brew install R
Unsatisfied dependency: XQuartz
Homebrew does not package XQuartz. Installers may be found at:
https://xquartz.macosforge.org
Error: An unsatisfied requirement failed this build.

After you install XQuartz, you can install R like this:

$ brew install R

Ant 1.8 "warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds"

I installed Ant 1.8 and build Java applications with Ant. Then, I found the following message:

warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds

You can find a good explanation of this warning with following link:

http://ant.1045680.n5.nabble.com/warning-includeantruntime-was-not-set-td2639463.html

There were incompatible change on Ant version 1.8. So, you may find the same message if you build old ant file with Ant 1.8.

Please just add includeantruntime="false" to javac task.

How I did interpretration for TDD workshop by James Grenning at #agilejapan

I did interpretation for TDD workshop by James Grenning, the author of "Test Driven Development for Embedded C" at Agile Japan 2013 on May 24th. This was first time for me to be interpreter at a conference.

Test-Driven Development for [Embedded] C by James Grenning at Agile Japan 2013 from Yohei Onishi

This time was good lesson for me. I summarized my experience so that this entry also become good lesson for some other people.

Why did I become one of interpreters for the workshop?

Let me tell you about history that I became interpreter fot the workshop.

I am an organizer of study group for "Test Driven Development for Embedded C" in Tokyo, Japan and I also was a reviewer for Japanese translation of the book. When translation was going to an end, I proposed to Akiyuki Takoshima, a supervisor for translation, that we invited James to Japan for publication party for translation. We thought Agile Japan was the best for it because it was planed to held after publication and organizers of Agile Japan usually invite keynote speaker from foreign country, mostly US. 

I expected that it would be difficult negotiation but the answer from Kenji Hiranabe, one of organizers, was very good. He said "Oh, that's nice idea. I have wanted to invite him to discuss about importance of quality in agile software development". Then Agile Japan organizer started negotiation and I and Akiyuki started preparetion for TDD workshop.

Basics of technical interpretation

Next, I want to tell you about technical interpretation.

You may think interpretation is just word exchange of different languages. I think real interpretation is more than that. In my opinion, we do interpretation as follows:

Maybe this is almost same as what non-native English speaker do when they talk in English...

1. Understand what speaker said
2. Understand its intention and digest result of step 1
• If we don't understand its intention, we have to ask speaker
3. Just exchange result of step 2 from original language to target language
4. Change result of step 3 in natural and simplified expression and talk to other people 
• We do step 3 and step 4 at a time

I think you think that step 3, automatic language exchange, is interpretation but I have to do more because we want to avoid loosing original speaker's intention and to tell it people in an easy-to-understand way.

To do this, we need both language skill and specialized knowledge for target area because we have different terminology each area. If we don't care about that, attendee don't understand what speaker say.

For example, in the case of translation, we often found that reviewers on amazon.co.jp write comment for translation of technical book that quality of translation is very bad.This is because there are problems on either step above.

• Problem on step 1: translator don't understand original text 
• Problem on step 2: translator don't have enough knowledge for target area
• Problem on step 3: translator don't improve quality of translation

Then what do interpreter do during workshop?

It depends on workshop. I basically think that interpreter there are two tasks.

• Tell trainer's explanation to attendee
• Help question and answer

No big deal. That's it.

Tell trainer's explanation to attendee

For this, interpreter see trainer carefully and waits good time to interpret. Then good time come, interpreter tells it to attendee. Trainer sometimes tell us about timing for interpretation but he/she usually want to focus on what he/she say and attendee's reaction . So interpreter should     manipulate timing.

Help question and answer

For this, it is easy. We just hear question from attendee, check its intention and tell it to trainer. When trainer answer then we do opposite.

What do interpreter should prepare? 

(1) acquisition of specialized knowledge for target area

As I wrote above, we need specialized knowledge for target area to interpret. we have to read whole workshop material. We also should read trainer's blog, interview article or movie to understand content of workshop, trainer's way of speaking/pronunciation.. If we do, we can understand what trainer say easily. 

(2) Practice of interpretation

I think shadowing is easy and best for practice of interpretation. There are good books for interpretation. If you are interesting in practice, please refer the books.

Retrospective for this time

Keep（Good point, what I want to continue）

(1) I had enought time to aquire specialized knowledge for target area

Before workshop, I held study group, wrote interview article, reviewed translation of the book and   translated workshop material. I knew everything. No worries about that :)

(2) I had meeting with trainer twice before workshop

I had meeting with trainer 1 week before and 2 hour before workshop. I could solve our question such as timing for Q&A, laptop for demo, etc. 

(3) I could do interpretation without any eye contact

In this time, interpreter manipluated timing for interpretation so that trainer can focus on what he/she say. We sometimes made mistakes but we did good job in general.

Problem

(1) We translated workshop material then trainer couldn't read it

In this time, we are not professional interpreter. So we thought we should show Japanese translation of workshop material so that attendees can understand it easily. But trainer couldn't read it and sometimes didn't know what to speak next. This is not big problem because he had original version of material and we can tell. But this was not good for trainer's cognitive load.

(2) We couldn't interpret jokes

Trainer often said jokes because he want attendees to be relaxed. But joke is difficult to interpret. Sorry James :(

Try（What to improve）

(1) Keep original text on material and add Japanese subtitle

This is good for both of trainer and attendee. Kenji Hiranabe, interpreter of keynote, did this.

(2) Interpret jokes

We need sens of humor? ummm :(

In the End

Thank you Agile Japan organizer for giving me a chance to interpret. It was very good lesson for both of TDD and interpretation. This chance must improve my interpretation skill. It was good experience for my life :)

Thank you Akiyuki-san, supervisor for translation. We did translation of material and interpretation at workshop. He gave me lots of advice. I could do it because he helped me.

Thanks you James. My interpretation was not like professional interpreter. James helped me : ) I have to say thank him for his open mind for my proposal.

If I will have next chance for interpretation, I will do better. I have confidence now :)

Want an opportunity for programming education? Join the global day of coderetreat!

I join the "Global Day of Coderetreat 2012" on December 9th 2012. I also joined one in 2011. So, this was the second time for me to join it :) In this entry, I explain followings to you:

• What is Coderetreat?
• What is the Global Day of Coderetreat?
• How to maximize your educational effectiveness

What is Coderetreat?

Coderetreat is a retreat for programmers. Most programmers need retreat because they always have to work for next delivery and they don't have enough time to learn basic skills such as programming and testing. The purpose of this event is to give them an opportunity to learn basic skills in a place where there is not delivery date. 

Characteristics of coderetreat are follows:

• We make program for Conway's game of life. It is a kind of simulation or artificial life as follows.

A single Gosper's Glider Gun creating "gliders"

• You have 6 pair programming sessions. Each session includes 45 minutes pair programming and 15 minutes retrospective. You change your partner each session.
• You have to delete your code at the end of session. You write code from the scratch every time. This gives you a chance to try different way of design. Actually you have to think design of code differently because facilitator gives you design constraints each session.
• You get a chance to learn object-oriented design, pair programming and TDD. Facilitator or other participants teach you by hands-on if you need it. 

• You can have good chance to learn new languages because your partner may know only languages that you are not familiar with. You may use it for the first time but you will make program with your partner's help. Actually I tried Ruby, Java, Scala and JavaScript despite I usually use C/C++ in my work. 
•  Free lunch and sweets are provided by sponsors. Yeah! Here are sweets at Tokyo site.

What is the Global Day of Coderetreat?

This time was not just a "Coderetreat" but the "Global Day of Coderetreat" because it held in many countries. You can see city list on the official site. 56 cities? I am not sure how many countries hold but it was global event !

We connected to India last year. We chat what was going on both site. I was very excited that we were doing same activity with global scale! 

How to maximize your educational effectiveness

You spend whole day for this event. You should make this event meaningful.

But how to do it? I think that it is better to try different way, learn from everyone and broaden your eyes. 

Here are my activity list for this time.

1. Use different language and testing framework from usual one.

I am embedded software engineer and I usually write C/C++ code. I sometimes write Visual Basic or Python code. I don't have opportunities to use other languages in my work but I am interested in other languages.

For C/C++ programmer, sometimes other languages and frameworks have good features and concepts. Learning new language cultivate your programming style. But it is little bit difficult for us to start learning by ourself because we don't know where we go at first. So ask your partner about language is better way to learn.

In my case, I had fun with Ruby and RSpec, testing framework for Ruby. You can write code with your intuition without deep understanding of language specification. That's really comfortable for beginners. And you can struct your test code by elements such as "description" and "context". Your test code will be easy to read.

Features of Ruby and RSpec are quite new to me and quite different from C/C++ world. You can't everything of other languages but you may be able to bring essence into your work. Your work will change.

2. Design your code with different way as usual

The purpose of this event is to give participants new experience. We should try little bit different way. 

For example, if you don't do modeling or object-oriented design during your work, you might design your code with array. But I recommend you to try object-oriented design. You might think:

• What kind of object do we need? 
• What responsibilities does each object have?

Thinking about these questions make your design better.

In my case, we tried different one. I used map and filter to overcome several constraints such as no if statements and no loop. With this idea, I could write shorter and smart code.

3. Try technical practice in ideal situation

Coderetreat is not real job. Even if you don't complete, no one gets angry. You can try any practice as you want.

For example, if you know TDD but you have never tried it, then try it in coderetreat. You will discover how it works.

I tried double test loop that is introduced in "Growing Object-Oriented Software, Guided by Tests". In this practice, we write end-to-end acceptance test first and write code for inside module with TDD to pass acceptance test.

I often see that many participants tend to focus on detail of components and end up with perfect components and incompletion of whole game. If you do this in your work, your project will collapse.

If you write acceptance test for one scenario and write code with TDD to pass acceptance test, you get a system that works for certain scenario early. There is another benefit. You found considering points for architecture design early. You can avoid some risks in your project.

4. Share what your were interested in during retrospective

At the end of each session, every pair do retrospective, wrote own thought about session on stickies and put them on the board to share your experience.

Facilitators usually pick up topics to share but you can ask them to share your topics. You will receive many feedback from other participants.

5. Communicate with participants in other countries

At this time, we have two tags for twitter. "#gdcr12" for global and "#gdcr12ja" for Japan. I tweeted for "#gdcr12" then @jbrains from Canada respond me quickly! It was really quicker response than I expected! It was very helpful. 

Thanks to @jbrains and @haradakiro, I found the video that introduce "Verbs instead of Nouns". Byt there isn't much information about that :( I don't understand what happens with that activity yet.

Conclusion

In this entry, I introduced coderetreat and how to maximize your educational effectiveness. This event is well structured and worth to join. If you want to get an opportunity to learn basic skills of programming and testing, this is good way for you:) Please join next time!

Book review: Test Driven Development for Embedded C

I wrote Japanese article to introduce “Test Driven Development for Embedded C” for embedded software developers in Japan last month. This entry is English version of it.

----------

As you can see with the title, this is the book about Test Driven Development for embedded software with C language. The author, James Grenning, is training, coaching and consulting agile software development or TDD over the world. He is also the inventor of agile estimation technique “planning poker” and one of the authors of agile manifesto.

Must read book for embedded software developer

Recently Japanese TDD passionates like @t_wada are holding TDD boot camp in major cities of Japan. This is an event for TDD beginners to have an exercise of TDD. People get together at one place, make pair for exercise and do TDD with specific problems.

Thanks to them, we can have more chance to exercise TDD and people are becoming to discuss practical topic about TDD on their blog or social media. TDD is not majority in the industry yet but I am feeling TDD is becoming common technique for us.

On the other hand, most of embedded software are left out of boom of TDD boot camp. They think that they have nothing is to do with TDD because every article on book or blog is written about enterprise or web system. They are things on other planets.

This is a book for you, embedded software developers :) This book deals with typical example of embedded software and C/C++ testing framework and explain how to do TDD with very tiny step of carefully. It also can be good textbook for module design of C code.

Why embedded software developer need TDD?

Embedded software developer might think why they need TDD. The author answer the question in the chapter 5 “Embedded TDD Strategy” of the book. He explain the we can solve “the target hardware bottleneck” problem, which is common for embedded software development.

The target hardware bottleneck is a problem that target hardware could be a bottleneck disturbs software testing. Generically, target hardware is different from host computer that is used to develop software. And we can have these situation:

• The problems leads to delay or increasing of working hour for software testing
• Target hardware are not ready till the end of project
• Target hardware is expensive and we can’t prepare enough number of target hardware 
• It takes long time to build and upload software to target hardware 
• The problems leads to difficulties to find bugs
• It is difficult to identify where bugs are because the target hardware also have bugs in the beginning
• Compiler for target hardware is different from one for host computer. Then we find unintentional behavior of software on the target hardware

To solve the target hardware bottleneck problem, the author recommend these approaches:

• Ensure quality of software by TDD on host computer before testing on the target hardware
• Do same tests on both of host computer and target hardware to find bug that are happend because of different between them.

And the author recommends TDD cycle for embedded software development as follows:

The book focuses on the stage 1 on the cycle. It doesn’t explain stage 2 to 5 in detail but you can find more information about step 5 on the author’s paper (PDF) “Scenario Testing with Executable Use Cases”.

I also write this cycle on my blog entry.

How is the book organized

The book consisted of three part.

In the first part “Test Driven Development”, the author introduces the process of TDD when we develop LED device driver. He also introduces why embedded software developer need TDD, test cycle including TDD and testing framework for C, Unity and CppUTest.

The second part is about technique to replace dependant module to test with C. It is very difficult to replace module with C, the only ways we can do are link substitution, preprocessor substitution and function pointer substitution. It also explains C++ mocking library “CppUMock” to test complex interaction with dependent module.

In the third part “Design and Continuous Improvement”, knowledge and technique to improve design continuously such as SOLID principle, test for legacy code and test patterns are introduced. If you are not familiar with object oriented design and programming, it is very difficult to understand. You need to read the books about object oriented design and programming first.

Please refer to the full table of contents on the official site.

How to read the book

For those who wants to know characteristics of TDD for embedded software

When I introduce the book to my coworker and he asked me “What is the difference between TDD for embedded and the others?” I think that basic points are same. For example, there is no difference between dependency to module and dependency to hardware.

I think that the most biggest difference is language constraint. Comparing to object oriented language, you need more technique to make test doubles, which is module to replace dependent module. In order to understand the technique, please refer to the chapter 7 “Introducing Test Doubles”, the chapter 8 “Spying on the Production Code” and the chapter 9 “Runtime-Bound Test Doubles”.

On the other hand, there is special bonus for embedded software developer. As I wrote before, we can solve the hardware bottleneck problem. Please refer to “Benefit for Embedded” at the chapter 1 and the chapter 5 “Embedded TDD Strategy”.

For those who wants to try TDD with example

To put TDD in practice, you need not only knowledge but technique. You must acquire technique by doing exercise. You can read the process to test and refactor your code for example of LED device driver in the chapter 3 “Starting a C Module” and the chapter 4 “Testing Your Way to Done”.  Try them.

Example in the chapter 3 and 4 is quite simple. You need test doubles in actual development. You can read about test doubles in the chapter 7 ”Introducing Test Doubles”, the chapter 8 ”Spying on the Production Code” and the chapter 9 “Runtime-Bound Test Doubles”.

For those who wants to know better C module design

TDD covers not only design and implement activity but also continuous improvement to better design by refactoring. You need to know what is better design to improve design of your software.

The chapter 11 “SOLID, Flexible and Testable Designs” introduces better design principles and implementation with C. You can understand how to design low-coupling module design with C.

There is also explanation to eliminate piles of if / switch-case statements by implementing polymorphism in the object oriented design concept with C. The code is tricky and it is very difficult to understand if you don’t understand object oriented design. In the case, you need to read object oriented software design book such as “Agile Software Development” first.

How Japanese readers say about the book

In conclusion, I introduce comments from the members of “Test Driven Development for Embedded C” studying group.

• It is very helpful to understand how TDD master think and do because every tiny step and design decisions are written at the beginning of the book.  
• I read the book and solve some exercises, then I could understand basic rhythm of TDD. 
• I was wondering how to apply object oriented design to C code before I read this book. Thanks to the book, I understood how to do it.
• I think that it is very difficult to understand the book at first if the readers understand object oriented design and test. Example of C code is not enough for beginners.
• This book is very good as index of modern software testing. It includes TDD, test for legacy code, SOLID principle and refactoring, xUnit Test Patterns. Every embedded software developer should read this!

Test Driven Development for Embedded C: Introducing runtime-bounded test doubles and the mock object

I had a meeting with reading group of "Test Driven Development for Embedded C" in Tokyo, Japan. We read the chapter 9 "Runtime-Bounded Test Doubles" and the chapter 10 "The Mock Object", then we discussed them. In this entry, I introduce the chapters and some topics from our discussion.

The chapter 9 "Runtime-Bounded Test Doubles"

When we implement product code with C, we have three ways to replace dependent module with test doubles. They are preprocessor substitution, link substitution and function pointer substitution.

Link-time substitution, which is introduced in chapter 7 "Introducing Test Doubles", is not flexible because it can't be replaced at runtime. If you want to use product code once and use test doubles at other time, function pointer substitution, which is introduced at chapter 9, is effective way.

Although function pointer substitution is very flexible, it has a side effect. Heavy usage of preprocessor or function pointer substitution reduces code readability. You should use them in a limited way.

By the way, the author doesn't explain preprocessor substitution. He may doesn't like it because of side effect.

The chapter 10 "The Mock Object"

The chapter 10 introduces the mock object, which is not so popular among embedded software engineers. The subject is a device driver for existent flash memory. The mock object is useful way to ensure quality of device driver before integrate it with hardware because:

• It is difficult to test complex interaction between software and hardware based on specific protocol with usual testing framework.
• It may be impossible to test error case in real device. 

Most part of the chapter 10 focuses on how to implement the mock object in C. You can also know "CppUMock", which is C++ mocking library, and "CMock", which is mock generation tool for C.

I found that mock object is usually complicated. If you implement mock object in C by yourself, you must to test it thoroughly before you use it. Test may be unnecessary if a mocking library assure quality of mock object.

You might think that making mock object is waste of time. But I think that it worth spending cost to make mock object and test with mock object. The more you standardize  device I/O, the more effectiveness of test grow. You can ensure quality of device driver more and more before integrate it with hardware.

At last we discussed why the mock object is popular among enterprise or web engineers but not among embedded engineers. Our conclusion is that mocking library with LL is very useful because LL has less constraint and mocking library is well maintained to use easily. On the other hand, library for C/C++ are still on the way to become useful.

This is one of difficulties for embedded engineers. To gain strength from TDD, we need to prepare way to overcome language constraint.

Boost quality of embedded software early with Dual Target Testing

I held the third monthly meeting of "Test Driven Development for Embedded C" reading group in Tokyo.

We read following two chapters then.

• The fifth chapter "Embedded TDD Strategy"
• This chapters introduces "Dual Target Testing" which is important to do TDD for embedded software development
• The sixth chapter "Year, but..."
• This chapter introduces opposing views for adopting TDD and answers for them

In this entry, I explain what is dual target testing and why we need it.


Concurrent development software and hardware makes embedded system development difficult

Concurrent development of software and hardware is common in embedded system development. As I told in the entry "Is TDD one of the solutions for difficulty of concurrent development of software and hardware?", concurrent development is one of difficulties in embedded system development. Bug whose cause are difficult to find often occurs because we move ahead while software and hardware are low quality.

I give you an example of concurrent development. I often develop embedded software as follows.

• I divide development period into several iterations.
• Each iteration has function scope.
• At the end of the iteration, we integrate software and hardware to make prototype. We evaluate it.
• I continue prototyping and create spec incrementally.

I think this kind of prototyping is common in embedded system development.

In this kind of prototyping, we have difficulties as follows.

• I don't have hardware at the beginning.
• I usually use minimum hardware to check booting OS.
• I also start from checking operation with previous product

After integration of hardware and software, QA team starts evaluation of prototype. They find lots of bugs. I recieved piles of bug reports and I start investigation of bug cause. This is very difficult task because quality of software and hardware is low then. Bugs are everywhere. Software engineers have to spend much time to prepare evidence to say software has no bugs.

This work is not productive. To realize productive embedded system development, software engineers have to spend enormous effort boosting software quality before hardware is ready.


Boost quality of embedded software early with Dual Target Testing

What can we do to boost quality of embedded software before hardware is ready ? James Grenning, the author of Test-Driven Development for Embedded C, recommend Dual Target Testing. This is a technique to make test with both of target hardware and PC possible at the beginning.

Here are key points.

• We build high quality software with exhaustive unit test and refactoring on PC from the beginning. Then we move to  unit test with eval hardware.
• This work reduce unproductive work like finding logical mistake with poor debug environment on eval hardware.
• Compiling for target processor and unit test on eval hardware and target hardware reduce bugs which causes are difference of development environment between host and target.

We can check only with unit test scope but we can start smoothly becausse we get high quality software before we move to eval hardware.

We don't need check logical mistake of module on eval hardware. Reducing unproductive work make us concentrate on spend enormous effort for advanced work.


We need to write unit tests consistently for dual target testing

We need to write many unit test code for dual target testing. This is more a time-consuming way than moving to eval hardware without tests.

It is very easy to check with eval hardware without unit test but everything needs to be done manually. Keeping quality is very difficult if everyone commit lots of code. We spend much time to find cause of bugs.

Writing unit tests for dual targeting test is very time-consuming way in the short term but it is very productive way in the long term because we can find mistake quickly. We reduce debug time and we can spend more advanced work.

We will talk which style is good for us at the next meeting.

How can we do dual target testing?

I will explain detailed techninque for dual target testing next time. Stay tuned :)

Rarejob lesson logs

December 23, 2012
And I took article discussion. Today we discussed about this article.http://online.wsj.com/article/SB10000872396390443675404578059311445951162.html

Discussion points are follows.
·What are possible problems that countries may encounter wh
en dealing with each other?
·How can people ensure harmony between nations with different cultures, values, or religions?

We usually have some problems with countries close to us. Japan has territory issues with China. Philippine has other issues with China.
I think that It is difficult to have harmony at political problem but we can do something at citizens level. Chinese people who have never been to Japan might think Japanese people are bad people but ones who live in Japan don't think like that because they knew what is reality. Reality is that most Japanese people are polite and kind. Once they visited Japan and knew reality, they will change his mind. Having chance to know each other is good way to cooperate each other.

December 23, 2012

I took TOEFL interview test for the first time. I was asked to hear following question, prepare answer in 15 minutes and answer for in 45 minutes. It is very hard.

"Some families think that having a pet is important to provide enjoyment to the members. Others think that a pet would only be a burden at home. Which opinion do you agree with and why? Give specific details and examples in your answer"

March 26, 2012
Total 13.5 hours. I used conversation material for intermediate level today.
In today's lesson, my tutor said that her mom loves the movie 'Hachiko the Akita dog' so they call her pet "Hachiko". That is nice.
I asked my tutor "What is your most favorite food?" Her answer is chocolate. Why do every girls like chocolate? I don't understand why because I don't eat it often.

March 18, 2012

Total 11.5 hours. I used conversation material for intermediate level today.

In today's lesson, we talked about my role in my family. It is to show my family outside of home. For example, I take them to overseas. I took them to Taiwan, Switzerland and Germany. Best city to visit is Zurich. I watched Italian opera with English subtitles. It was my first time to watch opera. So I was very impressed. My family is too.

March 17, 2012

Total 11.25 hours. I used conversation material for intermediate level today.

In English lesson, we talked about small family is better than big one. I don't think so. Big family is fun and you can learn social skill. You have to appeal to your parents in big family. That make you social person. In small family, you cannot get that experience.

But for my own family, I want to have small one. Raising many children costs very much in Tokyo. Two children are enough for me. My tutor has same opinion with me. She grew up in big family but she wants to have small family.

The second monthly meeting of the "Test-Driven Development for Embedded C" book club in Tokyo

The second meeting of "Test-DrivenDevelopment for Embedded C" book club in Tokyo took place on March 4th.

We read the three and fourth chapter

At this time, we read the third chapter "Starting C Module" and the fourth chapter "Testing Your Way to Done". We had first TDD session during these chapters. We wrote test code and product code for LED driver then we refactored both of them.

The third and fourth chapters are highlight of the first half of this book. If you start to read this book, I recommend you to try the example of these chapters by yourself.

There are sample codes on my github repository.

• Test code for LED driver
• LedDriverTest.c
• Product code for LED driver
• LedDriver.c
• LedDriver.h

Always make code stable

There are two very informative topics at this time. First one is to make code stable. Author describes TDD work like this:

And I saw this diagram and I found like these:

• There are two stable states of code, which are stable state that passed tests and one that don't pass.
• TDD doesn't make another state of code.

In the stable state that passes tests, there are not refactored code and refactored code. This topic is well known for TDD new comers.

The stable state that doesn't pass tests is a transient state. Compile error, link error, test failure. TDD makes this process obvious. TDD doesn't make unbalance state that passes tests but include incomplete implementation and mistakes.

You might think that you don't need to do such rigorous way. Of course, some people can implement perfect code at a time but most of us can't do it. We make mistakes. So we need way to find mistakes.

Do not care about performance too much at first but care about readability

Second topic is code readability. Here is informative description in the book.

don't let the performance factor outweigh improved design and readability unless there is proof the code is contributing to a specific performance problem.

Author doesn't mean that performance is not important. We had same opinion. When we deal with problem on performance, we have to find performance bottleneck with profiler. If you put emphasis on performance and write unreadable code while you don't know where bottleneck is, maintenance time increase and have bad effect on costs.

So here is better way:

• Considering a way to meet consider requirement at architecture design phase
• Finding performance bottleneck with profiler at the beginning of architecture implementation phase
• Refactoring code so that each module becomes more readable

We talked about own attempt

First topic is auto build and test with Eclipse. If you use this function, you can also do static analysis, code format and code coverage as follows. This is very easy.

Second topic is 'AdLint'. This is new open source static code analysis tool. I talked my plan to introduce own environment. I want to show demo next time.

Third topic is auto code formant with code formatting tool such as Artistic Style. You need to care when you use it on build process because it may make unintended code format. Manual execution may be better.

Fourth topic is code coverage tool. 'lcov' is good for gcc.

What's next?

Third meeting will take place on April. We will read the fifth chapter "Embedded TDD Strategy" and the sixth chapter "Yeah, but …".  First one is good to adopt TDD to embedded system development. Second one gives us hints to recommend TDD to other people. 

Is TDD one of the solutions for difficulty of concurrent development of software and hardware?

In my 5 year experience of embedded software engineer, I became to think concurrent development of software and hardware is the biggest risk of embedded systems. I thinks TDD is one of the solutions for the problem. So I held a book club of "Test Driven Development for Embedded C (Pragmatic Programmers)" in Tokyo. Here is my thought on how we fight with difficulty of  concurrent development of software and hardware.


Writing code for uncompleted hardware is so hard : (

General process of embedded software development is as follows:

1. Developing software on own computer
2. Debugging on an evaluation board
3. Testing on a target hardware

At the first step, we don't have the target hardware and have to write code for imaginary hardware that we have never seen. After we got the evaluation board, which is the second step, we try to debug on it to find bugs.

Most of engineers that I have met try to debugging on an evaluation board as quickly as possible, so they tend not to care about testability for unit test. As a result, we get high coupling code.

At the time, we have to advance the development with several risks as follows:

• Own computer, evaluation board and target hardware have a little bit different library and compiler.
• The evaluation board is bad in quality at the beginning of the development.
• There are many mistakes in logic of code
• Software engineer has to resolve underlying discrepancy in specification at the end of development.
• etc.

Problems occur someday. They make debugging time longer and distinguishing cause of bugs more difficult.

If we do agile software development for embedded software system, this will be a bottleneck.


Dual-Target Testing

For this problem, the author of "Test Driven Development for Embedded C (Pragmatic Programmers)" recommend hardware-independent software design and unit test on both of own computer and evaluation board from the beginning. This is called "Dual-Target Testing". If we do it from the beginning, we can eliminate bugs and make test easier.


What do we need for Dual-Target Tesing

At first, we need software design techniques to make software more testable. That is discussed in the books like "Test Driven Development: By Example" written by Kent-Beck or "Working Effectively with Legacy Code" written by Michael Feathers.

Second, we need cooperation with hardware engineers for low layer software development. For example, in the book of "Test-Driven Development for Embedded C ", there is a scene that software engineer and hardware engineer are discussing about design of interface between software and hardware so that software engineer can decide design of Test Double.


What "Test-Driven Development for Embedded C" is doing for this?

There are some code examples in my github. There is product code and test code for LED driver.

Key point of this example is a dependency injection at a constructor. Passing writing port address to constructor makes software independent with hardware. We can do unit test on both of own computer and the evaluation board.

This driver has writing port only. State of LED driver is stored at variable ledsImage once, then write to the port. This was an issue that software engineer and hardware engineer discussed in the book.

You might think that this example is quite simple and there are more difficult examples in real world. I think basic idea is the same as this example, which is interface design between software and hardware and low coupling software design against hardware and other software modules.

I will held the third monthly meeting of "Test Driven Development for Embedded C" book club in Tokyo and see this problem deeply : )