blog.sketchit.de Glaubt mir ich hab den Weihnachtsmann mit eigenen Augen gesehen…

ToneMatrix – Webcam Object Tracking

During our subject Interface Culture we (Matthias Schmidt, Bettina Steger and my humble self) were working on a project that uses hand-drawn objects on a white paper for making music.

Note: You have to allow the webcam!

Hold a white paper with black (or dark) objects that you painted on it in front of the camera. The red rectangle indicates the tracked area, the green rectangles indicates the tracked objects. Then hit the space-bar (Note: be sure that the flash movie gets the focus)

Check it out!

The flash-app uses the webcam for accessing three different videos. The normal video that we convert into two other videos: A threshold video and a palette video.

The palette video is used for tracking the brightest area in the video and reducing the tracking-area to this area (e.g. the white paper is usual the brightest color in the video, so objects that are in the background aren't tracked any longer). The threshold video is used for tracking the darkest objects within the brightest area.

How does this work? In the treshold video it is supposed that the red pixels are the objects, and coherent pixels belongs to one object. I've implemented a flood fill algorithm that counts the pixel of each object and returns the x,y position and width and height of an object. With this information we can nearly perfectly track the object and use for example the pixelamount for the volume of the sound and the x,y/width,height information for the timeline.

When all objects are tracked and you press the Space-Bar the tracked area is converted into a matrix of 16x16 fields. Each field represents a sound, fields that are lower plays a lower sound, fields that are higher a higher one. The timeline goes from left to right.

The source code is available here: ToneMatrixSource