$1 Gesture Recogniser by Wobbrock, J. O. et. al. adaptation in C++

The original codes can be found at http://depts.washington.edu/aimgroup/proj/dollar/

I made a C++ adaptation of the original C# version to use with OpenCV during my internship.
I've uploaded a copy here for academics to use in their C++ projects with as little code change as possible.

For simplicity, the code follows a singleton pattern.

http://www.comp.nus.edu.sg/~situyi/DollarRecogniser/

// Include headers at the top of your code
#include "dollarUtils.h"
#include "dollarRecogniser.h"

......

// Somewhere within the body of the code
Recogniser* gestureRecogniser = Recogniser::Instance();

vector<point> gesturePts;
// Populate gesture pts here
Result res = gestureRecogniser->Recognise(gesturePts);


if(res.GetScore() <= 0.6) {
strGesture = "0";
cout << "Unrecognised gesture." << endl;
}
else {
strGesture = res.GetName();
cout << "Best gesture match is " << res.GetName() << " (" << res.GetScore() << ")" << endl;
}

gesturePts.clear();

New Lightdraw Video

Video: Lightdraw, Alpha 2

TIME.com Interview with G.R.L

Graffiti Meets the Digital Age - Interview with James Powderly and Evan Roth of Graffiti Research Lab

http://www.time.com/time/video/?bcpid=1214055407&bctid=1483830664

Week 14 - The Teapot from 1975, Utah.

0407

I've stumbled upon a few videos related to the OpenTouch project that I've mentioned last Friday, 0404.

Introduction to MacLibre and OpenTouch
http://video.google.com/videoplay?docid=-3884258697819652860
Topics covered:

• MacLibre
• Multitouch introduction
  
• OpenTouch concept
• OpenTouch modules description
• OpenTouch gestures
  

Compiling OpenTouch on Linux and Mac OSX
oscpack

1. Copy /oscpack/MakefileLinux over /oscpack/Makefile
2. Change "INCLUDES = -I./" into "INCLUDES = -I./ -I./ip/" !!!important!!!
   

Some related files posted on GSoC file repo
http://groups.google.com/group/google-summer-of-code-discuss/files


TUIO: A Protocol for Tangible User Interfaces
http://tuio.lfsaw.de/



0408

Today I experimented with Quartz Composer mainly because it has the potential for lots of eye candy and also because it is the easiest way for me to create a splashTop with widgets. I was a little reluctant at first, but this beats having to do the aesthetics in OpenGL where it could potentially eat into my development time.

So I did a little exploring and within minutes I was able to get a cloud to follow my mouse pointer. Then Kevin took this a step further and threw in a Utah teapot that rotates with respect to the cursor location. However, there was a latency between the laser point and mouse cursor, mainly because my Lightdraw application was calling X11 API to move the pointer. This would be slow since X11 is not the native windowing system on Leopard OS X, and also because Quartz Composer was polling the mouse pointer location on every loop.

I would think that the simplest way to resolve these issues is to send the laser coordinates from Lightdraw to Quartz Composer directly, via OpenSoundControl (OSC), much like what OpenTouch uses.



0409

Finally hooked up Lightdraw and Quartz Composer to communicate via OSC protocols. I shall not have to endure choppy X11 on Mac anymore!

To do:

1. Standardize usage of OSC protocol communication.
2. Learn more about Quartz Composer. Find out if persistent variables are possible. (DONE, use Javascript)
   

0410 - Thanks Ben!

Ben from Apple Singapore came over to visit the Lightdraw crew. He kindly assisted us by cleverly introducing a smoothing function to our Quartz Composer Utah teapot demo, and that disguised the visual disruptions caused by unstable motions from a user's trembling hands. I suspect that I could implement the smoothing functions in Lightdraw to correct such visual disruptions too. After which Kevin and I prepared a few demos for tomorrow's video recording and also for future presentations.




0411

Recorded our first Lightdraw video with an entire length dedicated to lazer applications. That's right, its lazer with a "z". The demos recorded involves:

• Rotating an enironment mapped Utah teapot
  
• Moving a rendered image of cloud
  
• Navigating Mozilla Firefox
  
• Interacting with Google Earth
  

These demos are more interesting now because of flashy graphics they show how we can easily extend the laser interactions to applications with practical uses.

Here's the link to our previous video. Pardon the crappy music loop. I suggest that we may try using loops from fffff.at, with thanks and credit where credit is due of course.


Outstanding tasks:

• Multiuser splashtop
• OSC multiuser protocol
• $1 gesture recognition module
• Profiling
• Auto calibration

Week 13 - Back to my blogging routine

0331

Here I am back to my blogging routine. I've just finished my interim report and handed it to Kevin for review. Meanwhile, I'm reading up on calibration references, primarily homography which should correct keystone distortion. There are so many other kinds of intrinsic and extrinsic calibration to be done programmatically (e.g radial lens distortion, projector roll etc) but I do not think I'll have the time for these minor calibration now.



0401

Spent the day revising my linear algebra.
I also came across an interesting gesture article called the "$1 Gesture Recogniser". I've tested the applet and its fairly accurate for such a small piece of code. It will be great if I can add it to Lightdraw package as a gesture recognition module.



0402

Finally I've implemented the calibration using homography matrices, cvSolve and cvFindHomography. At first I screwed up but then I've recalled that I've got to normalize my matrices, since one of my assumptions is that I work with z-component = 1.

To calibrate, all I need to do now is to identify four predetermined non-collinear vertices by selecting their location on the camera image. In future, this method can be automated by projecting recognizable patterns for the camera to pick up.

Now I've got really accurate laser to screen pixel mapping. Yeah!



0403

Following up from yesterday's work, I find that I can place the camera no further from where the projector is. The DVCam should be able to see an area bigger than the projected screen if I positioned the camera at a side, such that the camera would observe a trapezoid projection of the screen. However, the downside of this is that the laser point would lose intensity when observed an angle away from the screen perpendicular. This can be resolved by using a fairly brighter laser, as tested with my 50 mW green laser.



0404

Very soon I've got to develop a multiuser capable application since I am not able to find one that is both multiuser and aesthetically pleasing and yet serves a practical purpose.

In summary, I would have to deliver, sometime in May:

• A multiuser framework, such as OpenTouch (it is currently still under development). This is everything! I cannot use the current operating systems' single user interface, so I'll have to design my own. The design worries my abit as it is quite challenging.
  
• A single application, probably like a splashtop, with many widgets inside. Each widget is controlled by unique user, and multiple users can come together and control many widgets. These widgets will have to communicate with each other of course.
  
• Aesthetically pleasing application with nice transition effects.
  
• Has to have very responsive interface for it to be usable.
  

QtRuby may be my solution to my GUI design woes. It doesn't look pretty though. I'm not so sure about those transitional effects, but the framework design (multiuser support) should be my priority.

Finally, I've submitted my interim report today.

Week 09 - Beginning of my Interim Report

0303

We had a Lightdraw presentation to SIGGRAPH Asia 2008 correspondents, and I was surprised by the appearance of my Computer Graphics professor, Dr. Huang Zhiyong. Turns out that he's a member of SIGGRAPH Singapore.

First meeting with open source committee today. We plan to have a custom IHPC OpenSource webpage for Lightdraw instead of the usual SourceForge-ish layout.

For the month of March I'll be spending time on my interim project report and blogging less.



0305

Went with Kevin to scout for a High Definition (HD) camcorder that can function as a webcam. Unfortunately, HD camcorders cannot be used as webcams, only non-HD camcorders can have this feature.

Whackapeng can now be played with variable number of players and avatars.



0306

Translated laser tracking vertices to mouse coordinates. I am able to move the mouse pointer with my laser via X11 API calls.

Debugged a bug in Whackapeng which counts the number of circled avatars incorrectly. This is caused when multiple threads access a common critical region. In this particular case, the bug happens like this:

Thread A counts number of encircled avatars, and increments a global counter each time an avatar is encircled. It then checks this counter against number of avatars to determine if the player has won.
Thread B is the countdown timer, clears all avatar encircled state.

Two out of three avatars are already encircled. Thread A examines an avatar that is encircled but already flagged.
Thread B interrupts, clears the flag of all avatars. And just before Thread B resets the global counter representing the number of encircled avatars...
... thread A interrupts! It sees that the current avatar's flag is not set, and increments the global counter. This incorrectly triggers a winning condition.

Proposed solution is to use a semaphore to protect the critical region where the game is reset at the end of every countdown.


Note to self: X11 path on mac is /usr/X11R6/include

Week 08 - "Leap Week"

0225

The demo program "moveresize" is able to work with laser. Thresholding of image affects the detection of laser blobs. Lower thresholds makes the laser dot highly visible but introduces more false positives. With higher thresholds, the laser "drops" once in awhile, so I've added loss tolerance for two frames after my program doesnt pick up the laser.

Kevin simplified Lightdraw into three particular areas the team has to look into.
1. Full screen program, probably with transparency overlay. With graffiti, and gestures.
2. Window interaction.
3. Contextual interaction.



0226

Quickly put together an extremely simply gesture recognition for moveresize. Though it is simple, it is extremely reliable as it does corner detection using Douglas-Peucker algorithm with high rate of success.

Increased the effective area of the binarized pointer size to allow a margin of human error when user wants to initiate a pause (It is difficult to keep your hands still and focus the laser on a point).



0227

Went for this Apple talk in the morning that was about role of Apple computing in Science. We were introduced to specialised software available by Apple that is free, such as "stitching" your displays together or desktop sharing.



0228

Read up on Apple window manager programming. I think X11 windowing system would be a good choice as my SuSe development platform uses it natively and the Mac Pro has X11 as well.
I would want to achieve transparent window overlays and control event signals using X11.



0229

Taken OpenCV motion template apart in order to assess how well it can detect a (anti)clockwise motion detection. Even after removing the direction weights on motion history two or more frames ago, I am not able to get a reliable reading from the motion template. Humans cannot draw a perfect circle, and so the direction tangents do not change uniformly. This makes it difficult to determine when the circular motion starts or ends.