For my photography news site, I want to use Twitter as a data source for popular links. This involves getting tweets out of Twitter and into MongoDB.

The first step is to be able to read the tweets from an account using Clojure. The library that holds the key is clojure-twitter.

To try this out, I created a new lein project, and added the following to project.clj:

(defproject twitter "1.0.0-SNAPSHOT"
  :description "Sample project to read in a tweet stream"
  :dependencies [[org.clojure/clojure "1.2.1"]
                 [clojure-twitter "1.2.5"]]
  :dev-dependencies [[swank-clojure "1.3.0"]])

Then run lein deps and start working out the code.

The example is a good start, however it the various authentication keys didn’t quite make sense to me on first look.

After registering an application, you will need to copy four strings from your Twitter account:

• Consumer key - identifier for your application, found with your app details.
• Consumer secret - shown with your consumer key and almost twice as long. Keep this secret.
• Access Token (oauth_token) - identifier for your user, found under the “My Access Token” button next to your application details.
• Access Token Secret (oauth_token_secret) - shown with your Access Token

The access token took me a little while to find. Yours will be at a URL similar to:

https://dev.twitter.com/apps/123456/my_token

Where 123456 is the identifier of your application.

Armed with these four pieces of information, the code is quite simple. First include the API:

(ns twitter.core
  (:require [twitter :as twitter]
            [oauth.client :as oauth]))

Then setup the various keys and secret phrases:

;; Make a OAuth consumer
(def oauth-consumer
  (oauth/make-consumer "N0tr3a11ym1k3y" ;; consumer key
                       "R23mnasdfphsfjas08ujhasf08aslfkjasfd234asfd" ;; consumer secret
                       "https://api.twitter.com/oauth/request_token"
                       "https://api.twitter.com/oauth/access_token"
                       "https://api.twitter.com/oauth/authorize"
                       :hmac-sha1))

(def oauth-access-token
  "1234456789-asdfjklkjasdfjkllASDFASDF")
(def oauth-access-token-secret
  "CompletelyRandomStringWithNumbersAndLetters")

Obviously, include your own keys and tokens.

The Clojure API is easy to read from the source and follows closely to Twitter’s API.

I chose to use the https version of the API and also scope the authentication. To read the most recent tweets from my followers:

(defn latest-tweets [count]
  (twitter/with-https
    (twitter/with-oauth
      oauth-consumer
      oauth-access-token
      oauth-access-token-secret
      (twitter/home-timeline :count (str count)))))

I’m not sure why the limit parameter only takes a string. Now after running lein swank and connecting up Emacs, I can run:

user> (use :reload 'twitter.core)
nil
user> (first (map :text (latest-tweets 1)))
"OK, that's 20 videos this weekend alone I posted at http://t.co/Vm2sVQV Hope you like (please subscribe if you do!)"

There is quite a lot of information returned for each tweet, such as conversation threads, user info and retweet data. Now I just need to decide how much of it to store for later analysis.

Kanban is a scheduling system for work items that I’m exploring further.

I heard about it from several sources, but didn’t focus my attention until earlier this year.

I highly recommend Kanban by David J Anderson. He is someone who is a practitioner as well as an author, which really helps in his explanations.

His book is broken up into four sections:

• Introduction
• Benefits of Kanban
• Implementing Kanban
• Making improvements

Each section is filled with references to industry theory, both in software and manufacturing, and liberally sprinkled with case studies from his own companies.

One part that really resonated with me is that introducing Kanban layers over the top of an existing development process. That is, you do not need to change your current process. Instead, the focus is on scheduling the work that goes into the process.

This makes Kanban a logical fit for processes such as Agile, as well as more traditional iterative or even waterfall development projects.

From the book:

Kanban is an approach that drives change by optimizing your existing process. The essence of starting with Kanban is to change as little as possible. You must resist the temptation to change workflow, job titles, roles and responsibilities, and specific working practices. Everything from which the team members and other partners, participants, and stakeholders derive their self-esteem, professional pride, and ego should be left unchanged.

The main target of change will be the quantity of WIP and the interface to and interaction with upstream and downstream parts of your business. So you must work with your team to map the value stream as it exists. Try not to change it or invent it in an idealistic fashion

As with any new area of knowledge, I feel like I’ve only scratched the surface. The interesting part is the range of topics to explore further, such as Deming’s writing.

It is also helpful to be working with teams that have already implemented many of the techniques from Kanban process, and to hear first hand stories of their implementation.

Now that I’ve been using Emacs for a while, I’m finding that my reflexes are adjusting to the Emacs way. On the bright side, I can still jump back into Vim and be productive.

A recent blog post highlighted the areas that I still need to explore.

I’m finding the deeper I dive into Emacs use, the more there is to find. The idea of a 30 day challenge to use nothing but Emacs is going a bit far, however I do find myself using it for more and more things.

Originally my focus was on Clojure, due to the great Lisp integration. I even have the keyboard shortcuts for Paredit mode as the background image on my screen. Now, I also use it for note taking and for organising ideas before preparing presentations. Rails coding is also emacs friendly.

A pleasant surprise is just how many controls and tools support emacs key bindings. Almost every textbox on Mac OS X for example. Typically, I make use of Ctrl-A, Ctrl-E to jump to the beginning and ends of lines, and Ctrl-D to forward delete a character.

The one area I still tend to fall back to Vim with is managing servers. I think I need some more practice with keyboard combos on terminals. I am also still to really learn emacs’ lisp.

Overall, I’m very happy with the switch.

I’ve been looking at build tools for JVM projects. These are the notes I took while reviewing Maven.

• mvn install
• mvn site - generate a summary of the project
• mvn dependency:analyze - determine missing or unused dependencies
• mvn dependency:tree - tree view of dependencies
• mvn help:effective-pom - useful for debugging

Refactoring Maven Projects

For a given Maven project, these are some strategies to optimise the process. Mostly this is the DRY principle - Don’t Repeat Yourself.

• Pull up common dependencies to “dependencyManagement”, then only refer to groupId:artifactId in the child project
• Use built-ins ${project.version} and ${project.groupId} when referring to a sibling project
• Use properties to re-factor version numbers across multiple dependencies from the same group

Build profiles

Use build profiles to setup different environments for dev/test/staging/production

• Specify profiles as the last things in a POM
• Call using -P e.g. mvn clean install -Pproduction -X
• Activation parameters automatically select a profile based on a set of selectors. (ie. a JDK6 profile if JDK6 is used)
• Profiles can be stored in a separate “profiles.xml” that lives next to “pom.xml”
• mvn help:active-profiles
• You can set a default profile in your ~/.m2/settings.xml and put passwords into settings files as well

Finding bundles

There is an online repository at http://mvnrepository.com/

Maven Properties

• project.* – Maven Project Object Model, that is stuff in your pom.xml. See: http://maven.apache.org/ref/2.2.1/maven-model/maven.html
• settings.* – Things from your ~/.m2/settings.xml file. See: online docs
• env.* – Environment variables, such as PATH, HOME, JAVA_HOME and M2_HOME. Note: prefer ${user.home} to ${env.HOME}
• System properties – things from System.getProperty(), such as java.version, java.home, os.name, os.arch, user.dir, user.home, etc.
• User defined – arbitrary properties within your pom.xml

Note: you can use maven properties in resource files, such as a .properties or .xml file under src/main/resources.

You need to enable this in the pom.xml:

<build>
     <resources>
          <resource>src/main/resources</resource>
          <filtering>true</filtering>
     </resources>
</build>

Assigning custom properties per profile can help with deployment into multiple environments.

Maven & Eclipse

Eclipse has very good integration for Maven. Later versions of Eclipse include this in their package. It can also be added from http://m2eclipse.codehaus.org/

Maven Archetypes

Archetypes exist for a range of different project types.

mvn archetype:generate

Custom archetypes can be developed if you have a new type of project or generated from an existing project.

Maven sites

Maven can generate a site for your project showing information about the project and various reports.

mvn archetype:create -DgroupId=com.example -DartifactId=sample-project
cd sample-project
mvn site:run
mvn clean site
mvn site:run
mvn clean site-deploy

Repository Manager

Hosting a local repository for a team provides a number of benefits, including a local cache to avoid excessive network usage. Additionally, work can be shared both internally and externally.

A repository can be as simple as a file system with the appropriate layout, or a full repository manager. Nexus is an option for hosting a repository. It can be downloaded from: http://nexus.sonatype.org/download-nexus.html

Other Resources

• Cheat sheet – comprehensive list of useful commands.
• Maven - The Definitive Guide
• Various Books
• Using Maven with Clojure

Note: I’m continuing with lein for my clojure projects. To integrate with Maven, you can generate a pom.xml file for a with “lein pom”.

I’ve been thinking about building a web application that behaves more like a desktop application. Think MobileMe, GMail, Google Docs, etc.

This post collects together some of the research that I’ve been doing so I can refer back to it.

SproutCore

• Homepage
• SproutCore on Wikipedia
• Getting Started

Cappuccino

• Homepage
• Cappuccino on Wikipedia
• Tutorials

Raphaël

• Homepage
• Raphaël on Wikipedia
• Drag & Drop support

There is a few places that have a comparison between some of these, including:

• SproutCore vs Cappuccino (stackoverflow)
• SproutCore and Cappuccino (stackoverflow) - if you use Cocoa, pick Cappuccino, else use SproutCore.
• To SproutCore, or not to SproutCore
• Building a Project Planning App (ruby or rails oceania)

From my current reading, my inclination is to dive deeper into SproutCore. Apple and Yehuda seem to be investing a lot into it. That said, Cappuccino looks quite interesting.

Update: This article provides a good summary of large scale development using Javascript.