Week 04 - Triggering Events

0128

Went to a local precision optics company to sample the ~532nm bandpass and ~650 longpass filters. The red longpass worked really well and the green laser point was not visible at all despite its high intensity. The laser dots appear as the brightest pixels when the webcamera were fitted with the respective filters.

Back at the Cove, we resumed our UML discussion and added methods to class diagram, mainly to the Lightdraw controller and DisplayManager singletons.



0129

Implemented a feature to detect when the user has the laser point hovering over an area (triggers a hold event). This is done using a separate motion history buffer with a shorter timeout duration.

With this, a user can control when he wants to draw graffiti by triggering the hold event to begin drawing. Drawing ceases when laser point goes out of screen, or when laser is switched off.

Note: Further work can be done to make the detection more accurate. Take the region of pixels the laser point is sitting on, which will have timestamp equal to 0 or some value. Search for the set of immediate pixels with timestamp equal to this value, and empty the complement of this set (i.e setting the timestamp of the rest of the canvas to 0). This will guarantee that only one laser point can trigger the hold event.



0130

Because the UML diagrams have grown so much, Kevin suggested we have the diagrams on the projection screen, and seemed like a much better idea. I had a go with Mac's version of UML diagram software, called OmniGraffle. I felt that it was easy to use, and had all the basic functionalities that I needed. However, I felt that the feature that allows a user to tag notes is quite inflexible. What I wanted was to tag individual methods in a class but it only allowed me to tag the entire class due to some restrictions.

Rebuilt one of the USB infrared pens to a battery operated one instead. The end result works but did not look pretty at all.



0131

Today's UML discussion was mainly on how to handle events. Kevin suggested a signal-slot mechanism, and I got really confused while trying to understand what "slots" were.

So I did some reading up, and these are two of the more informative links I came across.

http://doc.trolltech.com/3.3/signalsandslots.html

http://sigslot.sourceforge.net/sigslot.pdf

The paper has a simple and effective example (using lights and switches) to show how two tightly coupled classes can inherit signal-slot classes to become loosely coupled classes, yet maintain type safety between them. Neat!



0201

Contacted my academic supervisor today to setup a meeting on Tuesday the week after next.

Merging of the programs moveresize and laserMotion still in progress.

Week 03 - Design, design, design!

0121

The Lightdraw team held a meeting to this morning as planned, and will continue to do so every morning this week.

For once, I found myself in a place where design and planning is not brushed away as if it were an unnecessary and time-wasting process. As this blog is not a platform for me to talk about the kind of development horrors or design architectures that will inevitably-collapse-unto-itself that I've previously encountered in past working experiences, I would just like to say that I am so darn glad that things are different here. Hope, is that you?

With help from everyone, the use case diagram was done without too much difficulty. I felt that it was a good reality check for two reasons. Firstly, it was a good chance for the team to clarify doubts, so everyone can be sure to have a common understanding of Lightdraw. Secondly, this forces me to revise my software engineering concepts.

Originally we had planned to use the afternoon to go Sim Lim today to get our supplies. However, the trip got postponed to Tuesday. I carried on with my debugging and managed to get my program to clear the canvas properly. Did so by splitting update_mhi into two parts: one for marking out motion, the other for indexing each pixel of movement with timestamp in a separate image buffer.

I contacted a local company that deals with precision optics, and they claimed to have the optic filters that I am looking for. However, I wasn't able to get a quotation from the lady since the sales department was busy.



0122

Did domain modeling in the morning. Its like a simplified class diagram, so that its easier to separate the different aspects of our program, and group components with similar functionality together.

We went to Sim Lim after lunch. I was able to find the stuff needed for making an IR Wii pen. I bought:

• 4x rechargeable AAA batteries
• 2x IR LED (2-3V)
• 2x 39 Ohm resistor (0.25 Watts)
• 2x Usb connectors
  
• Crocodile clips

The camera shops did not sell the optical filters that we were looking for.



0123

Spent the morning assembling the two IR pens. How do I know it works? Webcams can see infrared light. Hand phone cameras will work too. With my webcam I was able to tell that the infrared LED can be switched on.

Overloaded the minus '-' operator in imageWrapper.h, so that I can subtract pixel values from an image with another. i.e two image buffers A and B

dst[x,y] =
0, if A[x,y] - B[x,y] is less than 0,
A[x,y] - B[x,y] otherwise.

I needed this to kill the dopplegänger effect in motion detection (it renders the motion twice, with delay). This is caused by the way update_mhi (originally) behaves. A motion is defined as a change in light. And light is given a numerical representation using image buffers. update_mhi finds the absolute difference between two frames, so it defines motion as the pixel difference between the old and new image. So it draws the moving object twice: first from the object's previous location (found in the previous frame), second the object's new position.

By Wednesday, we've got a design pattern that's beginning to look like Models View and Controllers (MVC ). We've also agreed on a nifty menu that we can use to interface with applications. Kevin found two more use cases we missed. Given time, all design diagrams will grow into big ugly monsters, and I've got two (unrelated ones) right here:




Flowchart of my school's software engineering project. Chart is so big it had to be cropped into two pictures. I still find these diagrams hilarious.



0124

We've got a good-looking but incomplete class diagram today. CT was able to write a pseudo graffiti program with ease. We noticed that the developer writing the program had to pass a function pointer though. There's still multithreading that we've yet to take into account.

Added a window that is a cross between canny and motion detection, so I get a visual feedback of where my pointer is when I'm drawing an image.

Got the Wii setup up, works well when wiimote is 35degrees from the LCD plane. Calibration was not as easy as it seemed, mainly because the wiimote was not able to see the infrared LED at certain angles.



0125

Experimented using some kind of grey scale and pattern printed on a transparency as a webcam filter as I thought that it may help detect the laser light on a bright background. No dice, I should use a tinted filter to help to reduce intensity instead.

Colour detection much more successful with my new webcam, Z-Star Micro zc0301p.



















Next would be flood fill, pause detection, calibration and some kind of demo.

We're going to visit the precision optics company next Monday.

Week 02 - Laser Detection I

0114

The week began with a question: How do I differentiate the laser dot from its background? The bright green dot could easily be a bullet on a presentation slide, or the desktop background, or the mouse pointer, or ....

I would like to think that I am able to differentiate a laser point from the background via:

1. Color separation
2. Motion tracking
3. Canny edge detection

I decided to use OpenCV libraries to implement these methods, with the aim of finding the capabilities and limitations of what I can do for each of the three. Hopefully, one method would be able to make up for the shortcomings of the others. Then, a combination of methods will give me a better approximate for the location of the laser point.

I began working on the HSV colour model. I know that I will be looking for a green hue of high saturation and brightness. The green hue would be in a extremely small range of approximately 180 degrees. I tried to use OpenCV to convert the image from a RGB to HSV colour model, before extracting the individual H,S and V channels from the resulting image. Getting the full channels is no problem at all, but I am only interested in a certain hue. Unfortunately for me, OpenCV uses a very different scale for HSV.

Usually, H spans from 0 to 360 degrees, and S, V would take on values from either 0 to 1, or 0 to 100. OpenCV did not state explicitly what range of values HSV would take on, but a quick look at the sample code (camshiftdemo) reveals that in OpenCV, H spans from 0 to 180 degrees, and values of S,V spans from 0 to 255. Confusion! I would have to try and error to understand the scale used in OpenCV.

Took a little time off after work to help debug Peng's program and caught two nasty bugs.



0115

By Tuesday, I was still unable to extract the laser hue from the webcam feed. I found out that my webcam sees the red and green laser dots as white instead of their respective colours, mainly because of their intensity. I am able to get the webcam to pick up their colours only on two occassions; when the laser dot is in motion, and when the camera is ridiculously out of focus. A comparison between my webcam and the iSight shows that the Apple iSight is able to pick up the colour of the laser dots better than my Logitech Quickcam IM

As progress was slow with HSV, I had to put a hold on this until I can grab hold of a better webcam or think of a solution. I spent the rest of the day working on motion tracking, mainly with the use of the template included in OpenCV.



0116

Motion tracking is far more successful at detecting the laser dot. By using a relaxed version of the problem, I am able to "draw" on a black background by making the motion persistent. The setup involves a smooth non-glossy black wall approximately 2m away from the quickcam.

Motion tracking is also able to give a rough figure of how intense the dot is. The red laser, being less intense than its green counterpart, gives a much smaller dot. Also, gaps appear when trying to draw a line quickly using red laser. This did not occur with the green laser. Noise can be observed around the trail left by the green laser but the noise is not present the red one.



0117

Added removeNoise, which works by downsampling the image followed by an upsampling, using Gaussian pyramid decomposition. The noise caused by green laser is gone for good!

I proceeded to get the setup to work with a projector screen after having success working with a relaxed environment. I noticed that "Whiteouts" always occur at the beginning (and when I clear the image buffer by pressing 'c'), because the image buffer is initially empty, and the next and also first incoming frame is not, so the resulting difference between the two is picked up as motion.

Note: clock() behaves differently on light and my laptop.



0118

Had our weekly update today. Next week, mornings will be spent on design pattern as the project will be shifting to OOP design. I will have to get the draft UML diagrams ready. We plan to buy parts to make the IR pen, and a 4.5m long firewire cable. Parts of IR pen include: 40-70 Ohm resistor for usb powered, usb power adapter, momentary switch, 1.6v infrared LED. I might want to check out light blocking filters too. (I am looking for a 532nm pass-band filter for the green YAG-medium laser, or 650 longpass filter for the common red Krypton-medium laser.
Ref: http://www.seattlerobotics.org/encoder/200110/vision.htm)

The team tried with various setups of the iSight. Having the camera at the behind the screen is not feasible as there is more interference there. [Edit: see 0211, camera @ back is better] So we ended up placing the iSight in front of projector screen to test how well the iSight is able to see the laser dot. We cheated a little by overlaying terminal transparency on the desktop to improve detection of dot with iSight.

Week 01 - SAB Week

0107

My Unneeded Prescence

2nd SAB Meeting will be held in the Cove over the next two days (8^th, 9^th Jan), so everyone was taking turns using the Cove for preparations. The increased signs of activity along the corridor and in the Cove is an obvious indication that the SAB meeting must be a fairly important event. Amidst the temporal chaos, it was decided that it would be best for JL and I to be out of action the next two days. Peng is needed for the demo though.

Also, I have discovered the cause of the unresponsiveness in the Light Draw programs on my laptop. The culprit:

cvWaitKey(n)

After a few trials, I found the uncanny behaviour that the frame will not be rendered if the parameter n passed to cvWaitKey() is <= 1. Weird huh. I would attribute the root of the problem to my laptop (IBM T43 Thinkpad) not having less than 2 millisecond resolution timer, but I do not know this for a fact. The uber machine Light (where we ran our codes on) did not have this problem. Then again, I'm comparing my miserable Thinkpad with a 10,000 monster. (This also enforces my dev hypothesis that if I'm able to get things running smoothly on my laptop, then logically Light should not have any problems running the same program too.)

So, apparently cvWaitKey(1) was (and still is) used in the codes because cvWaitKey(30) was causing too much lag between rendering loops. This reminds me again of the problem that too much processing is done in one loop; rendering, fading, detection, trailing, memory operations etc. are done in one loop. Peng and JL are already looking into multithreading, but my previous experience with multithreading was generally negative. I was not able to control which thread to wake up next (in Java), and I have no control over what the scheduler does anyway. Could be due to my ignorance with multithreading, and I knew it would come back and bite me someday.


0108

My Fix: Artefacts appearing in lower/bottom right half of the screen. Config: ATI, SuSe 10.3

Setting

Option "XAANoOffscreenPixmaps" "true"

under

Section "Device"

in /etc/X11/xorg.conf should solve the problem. !!! Remember to backup xorg.conf first !!!

I've got my laptop running a few hours now, and the problem seems to be resolved. The occurrences are random, so I am not able to tell for sure.

Ref: http://wiki.sabayonlinux.org/index.php?title=Black_borders_around_windows_fix_(ATI)

0111

Friday the team had a morning meeting, we discussed about Light Draw's progress and also planned the next phase of development. Light Draw involves two aspects in general, that is HCI, and recognition. The latter should be our main focus for now, and we will have to explore methods to detect irregular shapes.

Got Peng to show me the student's quarters on the second level.

I tried to find out why Light Draw isn't as sensitive to Red channel. Tried messing with the individual colour channels, and with Peng's help I realised that it isn't enough to work in RGB color representation only. I was looking CT also mentioned that the problem is non-trivial. Will probably need get their help on this again.

Also, I had a go with the Wiimote Whiteboard program mentioned on post 0103, 0104. The setup is fuss free, except for the pairing between the Wiimote and the BlueSoliel program. Instead of holding Wiimote buttons 1 & 2 down while pairing, press the red button found in the battery compartment of the Wiimote. I tested the setup on my laptop and used a remote control instead of a infra-red pen. This gave me problems during calibration.

My first treat from Kevin! Kevin bought dinner today. Woohoo!


Misc stuff:

Draft N for my laptop! Got DWA140 USB dongle to work in Linux. The installation disc that came with the dongle supports only M$. Pffft.

Google tells me that the DWA140 uses the RT2870 chipset. Glad I don't have to open it up and void the warranty.

Download the RT2870 chipset driver at http://www.ralinktech.com/ralink/Home/Support/Linux.html

Some tweaks to the default installation required:

• Edit Makefile, replace /tftpboot with another path. e.g /opt/dlink/
  Default installation places tftpboot in root directory, which I did not like
• Follow instructions found in README_STA, would be a good idea to copy it from extracted folder to /opt/dlink/
• Modify os/linux/config.mk, set
  HAS_WPA_SUPPLICANT = y
  and
  HAS_NATIVE_WPA_SUPPLICANT_SUPPORT = y
• Copy the folders /os/linux to /opt/dlink/

Week 00 - Because that's how programmers count.

0102

My First Day at Work

A Wednesday. I reported to work punctual despite the new year mood. (Hey I'm sober) I was introduced to Janice from HR, who assisted me in filling up a couple of lengthy documents (I've noticed that IHPC has more forms than usual and they are particularly long). There is one document which lists the schedule of shuttle bus services, and I bet it will come in handy. I was also given a student pass which has yet to be activated.

My Worker's Quarters

"The Cove" is what we call Lightdraw's HQ. And The Cove was used for meeting in the morning, so I chilled in the Library (which looks more like an aquarium), deleting Laptop's bookmarks, history, stored passwords temp files, system restore files, cookies etc. as I was supposed to give it to IT department for a scan.


My First Assignment

I received my first assignment from Kevin, Lightdraw's research officer. The Lightdraw team is to dismantle the Christmas tree. We rewarded ourselves by eating the Christmas Tree decorations (A choco gingerbreadman and a white pastry-slash-biscuit xmas tree shaped thingy). They looked somewhat edible. I am not in IHPC long enough to know whether they reuse their Christmas tree decorations or not. So Peng, my colleague from Temasek Polytechnic attachment, bravely took the first bite. Monkey see, monkey do. The ornaments didn't taste too bad actually.

Then minutes later, Kevin opened his pack of choco-chip cookies given by Lester to share. D-oh! Kevin also bought us T-shirts from Mauritius. He flew back to Singapore only yesterday.


My Leaking Air-con

Peng and JL (surprisingly he has the same surname as me), also another intern from Temasek Poly, taught me how to fix the faulty air-con which, Peng says, "fails only after holidays (and holidays only), don't know why."

Here are the steps taught to me by Peng and JL:

1. Take the make-shift funnel made out of a cut mineral bottle out of the first drawer of the cabinet the air-con is sitting on.
2. Get ready a pail, stunned (army slang) from the pantry.
3. Open a trapdoor located at the side of the air-con, and drain the water into the pail.
4. Close the trapdoor.
5. Pour the water into the beans compartment of the coffee machine which Kevin drinks from everyday.
6. Pour the water away and return the pail to rightful owner.
   

On to serious matters. Peng and JL gave a demo and showcased their progress. Kevin, having just travelled back from Mauritius yesterday, suffered slightly from jetlag, but that did not stop him from giving us a briefing to put the team on track. We (including Cheng Teng aka CT) were given a brief update of Lightdraw's progress. Here are my current tasks, in general:

• Laser pointer (depth) detection. (Read paper provided by Kevin.)
• Stretching, possibly detect two pointers and stretch a generic primitive.
  
• Mainly HCI, look for ways to generate events to OS.
  

SAB's coming over, so Kevin did a few trial runs to prep him self for the demo.

Harold joined us for lunch. Spent the rest of the day setting up Suse, Emacs, OpenCV, SVN and Lightdraw.


Note: The access-point in the Cove is Light. External IP of SVN repository noted in bible.



0103

My Excursion

Today the Lightdraw team went to the Visualization Centre (VC) in Temasek Polytechnic. Kevin hooked us up with Mr Tan, who was kind enough to explain the technology behind each of the setups in the VC. The first setup is more related to what we are designing. It projects image onto a mirror which reflects the image onto a piece of vertical glass that has a "special refractive holographic film which scatters light" (Ref: TouchLight: An Imaging Touch Screen and Disply for Gesture-Based Interaction). The image shows up clearer on the viewer's side of the glass. There are two infrared cameras to pick up infrared reflection from the user's hand, and therefore this enables multitouch interaction, like what you see from Minority Report. I've seen something similar before on Youtube, be sure to check out the series of videos from Johnny Chung Lee (HCII, Carnegie Mellon University) at http://www.cs.cmu.edu/~johnny/projects/wii/
The first video is analogous to the technology of VC's first setup. We have Wii back in the Cove, so Kevin suggested that we try out his setup as it looks pretty darn fun and interesting.

There is also a flat plasma display that supports 3D without headgears (stereoscopic) of any kind. The plasma is slightly fatter and the degree of vision for 3D is very limited, possibly 20 degrees by my estimate. Also, the viewer has to stand approx 1.5 metres away for the 3D effect to be noticeable, otherwise the image will look blurry, and that might irritate the viewer.

Next setup uses stereoscoping enabled by polarising glasses and polarising filters on projectors for three rear projections (Two projectors for each of three panels, both projectors use different polarisers). On screen is a simulation of a virtual MRT station. It is similar to setup seen in IHPC, except that it has a nifty handheld controller for X-Z plane panning, and a headgear mounted with ultrasonic transmitters to communicate with 10 (?) strips of ultrasonic receivers mounted on the ceiling (My secondary school ultrasonic project paid off). Disadvantages are obvious: Headgear is heavy and messes up my do, fast movements cannot be detected, significant lag, prolonged use may cause dizziness, requires low hanging ceiling for ultrasonic receivers to pick up signal accurately, headgear comes along with heavy battery pack, setup is huge and requires lots of space.


Last setup is for teleconferencing, It has a normal glass tilted at an angle to reflect images off a LCD to a viewer, as such the image can only be seen from the viewer and not behind the glass. This is important as there is a motorised camera behind the glass to capture the user behind the see-through glass. Our first teleconference? With a cleaner at an office in Aeon. This setup is more suited for single user only as there is only one keyboard, mouse. I would think quality of conference directly depends on how well camera operates and network latency. Camera looks very expensive as it as motors, wide angle lens and ability to zoom at the same time.

I heard the total setup costs 4.2M. You have to be very credible to get that kind of funding.

Once again, thank you Mr Tan! Learnt alot during this trip, and I got to experience advanced tecnology and equipments you dont see everyday.

Wise man says: Five makes bad number for long journeys.

We took bus service 10 all the way back to the Capricorn. The miserable journey is an hour plus. Slept most of it away. Makes one wonder how Peng takes this bus service to work everyday. I Kowtow to you man!


My OpenCV Installation on SuSe 10.3

Back at the Cove, I resumed setting up OpenCV. Finally got it up at the end of the day with CT's help.

Here's what needs to be done. For Suse 10.3.

Google "ffmpeg opencv". First search result:
http://www.comp.leeds.ac.uk/vision/opencv/install-lin-ffmpeg.html

Follow the instructions there with a few tweaks. Instead of installing to /home/ffmpeg/ as the guide says, install it to /usr/local/. Change the paths respectively in ./configure commands.

RTFM! Read all INSTALL and README included in the packages, and the ./configure output as well. Also make sure you have the devels, image libs and GTK+ 2.x which does not come with SuSe by default. For GTK+ 2.x, it is called "gtk2" in Yast, so search for "gtk2" instead.


This is my config from OpenCV ./configure which you will execute later on.
Now may be a good time to download the devels of the packages marked "yes".

Windowing system --------------
Use Carbon / Mac OS X: no
Use gtk+ 2.x: yes
Use gthread: yes

Image I/O ---------------------
Use libjpeg: yes
Use zlib: yes
Use libpng: yes
Use libtiff: yes
Use libjasper: yes
Use libIlmImf: no

Video I/O ---------------------
Use QuickTime / Mac OS X: no
Use xine: no
Use ffmpeg: no
Use dc1394 & raw1394: yes
Use v4l: yes
Use v4l2: yes



Also, only the "make install" commands requires root privileges. You do not need root access for ./configure despite what the guide says.

CT pointed out an essential fix to ffmpeg avcodec.h (Thanks CT!): Patch avcodec.h in /usr/local/include/libavcodec and other places it may be found or used:
Replace:

#define AV_NOPTS_VALUE INT64_C (0x8 0000 0000 0000 000)

with

#define AV_NOPTS_VALUE 0x8 0000 0000 0000 000LL

The line is found at first page of the code.


Afterwhich, opencv may still refuse to compile with ffmpeg. It didnt compile for me. Solution? Dr Voicu from CS3212 would say, "accept it!". That is one of the more common solution to problems in CS. So, take out ffmpeg from opencv ./configure, and mine looked something like this:

./configure --enable-apps --enable-shared --without-ffmpeg --with-gnu-ld --with-x --without-quicktime CXXFLAGS=-fno-strict-aliasing

That should be all there is to compiling opencv. Next, getting my ancient Logitech Quickcam IM to work. Perhaps I should invest in a Logitech Quickcam Sphere? Sponsors? :)



0104

My Webcam

I managed to setup my webcam the night before. Found good sources for Logitech linux drivers:
http://qce-ga.sourceforge.net/
http://www.quickcamteam.net/hcl/linux/logitech-webcams (go here for Quickcam IM)

Note: Quickcam IM uses spca drivers, not uvcvideo.

For the quickcam drivers, you will also need xawtv, v4l, SDL etc.
I got the SDL off SuSe repository through Yast, wasnt able to compile either of the downloaded rpm and tar file.

spcaview works like magic.

Brought my camera to office, managed to test it with samples included in openCV with Kevin's help. Change access to opencv/samples/build_all.sh to executable, run it to compile the samples.

"motempl" is a good sample to use to test your webcam. "edge" too.

Next, I managed to compile lightdraw codes.

For some reason I've yet to uncover, only "alpha", "edgeBlending" and "trail_v2" is responsive enough. Refer to bible for complete list. Think I'll debug this over the weekend. Also, the codes are half written in C++, so I will rewrite a version in CPP anyway.

Todo:

• Find out why some programs are unresponsive on my laptop, probably use my desktop as control.
• Find out why I cannot connect to external IP of Lightdraw repository
  
• Read paper
  
• Read openCV wiki and manuals.
  http://www.cs.iit.edu/~agam/cs512/lect-notes/opencv-intro/opencv-intro.html looks good.
• Get laser point detection up.
  
• Try to borrow a SATA/External DVD reader off colleagues.
• Try out Johnny Lee's Wii codes
• Contact my supervisor from SoC, Dr TEO Yong Meng