The jamTable is a tangible system that was developed with the goal of lowering the entry bar for novice users who want to perform and learn with more experienced musicians. This is enabled by allowing two users to collaborate in real-time over an interactive tabletop: while one plays a musical instrument, the other controls a simple tangible musical sequencer (in the likes of the Audio d-touch)

The interface for this sequencer relies on a grid of 24 music tiles, seven control tiles (one for each of the music tile columns), and a record tile. The interaction relies solely on tangible actions, with users having two sets of tokens to manage: music and control tokens. Users can start or stop recording an instrument’s output by placing or removing a music token on the recording tile. Likewise, users can play recorded sounds by simply adding the corresponding music tokens to any vacant music tiles. Furthermore, by using a control token, users can change the volume and pitch, or apply the popular drive effect to any set of music tokens playing in parallel above the control tile just occupied.

The jamTable interface

The jamTable was implemented using conventional technologies, such as the Processing programming language, the reacTIVision tracking software, and the TUIO messaging protocol. The surface has an area of 120 by 70 centimeters, and each token is a cuboid of 8 by 8 by 1 centimeters. Twenty tokens were deployed: 15 music tokens that enable users to record and play sound clips; and five control tokens that can alter the playback through higher and lower pitch and volume, or through a drive effect. Finally, recording is enabled through a small directional microphone capable of recording most musical instruments. Its sound is played through four small speakers and a subwoofer.

Two participants interacting through the jamTable

A study to explore how a tangible representation impacts user performance in a problem-solving task resulted in the development of an augmented version of the popular game of Four-in-a-row. In this game two players take turns dropping colored disks in a 7×6 vertical grid. The objective is to be the first player to connect four disks of the same color in either a horizontal, vertical, or diagonal line.

The augmented version of the game included a hovering feedback, that simply took the form a highlight in response to exploratory gestures with the game pieces. Essentially, if participants positioned a game token at the top of one of the game board columns, they were presented with appropriately colored visual feedback indicating the position the disk would reach if dropped. This was intended to provide information on the board’s possible future states prior to making an actual move in the game.

The game featured a 7×6 grid of holes with a total visible size of 26×24 cm, and game diska of 3 cm in diameter and 0.5 cm in thickness. The hover feedback was realized via two vertically stacked photo interrupters mounted on top of each of the columns (14 sensors in total). Placing a physical token in between the top emitter and sensor triggered the hovering feature, while an interruption of the bottom sensor was used to indicate a disk drop. Each of the bottom sensors was located 0.5 cm above the game board, with the top sensors located at 1.5 cm.

Graphical feedback for the hover event was enabled by placing a diffuser screen (Rosco Grey) and seven strips of digitally addressable RGB LEDs behind the board (so that there was one LED for game-board hole). All electronics were connected to an Arduino Mega microprocessor that monitored input and displayed the feedback.

Participant playing Four-in-a-row in each of the three interfaces

Esteves, A. & Oakley I. 2011. Eco Planner: A Tabletop System for Scheduling Sustainable Routines. ACM Tangible and Embedded Interaction, Funchal, Portugal, January 23 – 26, 2011, pp. 139-144. [download]

In Eco Planner users position iconic tokens on a grid to create routines representing their activities in the home. Its focus is on creating and visualizing activities that involve energy consumption in order to calculate and understand the environmental impact of particular behavior patterns and consumption choices.

It is composed of a set of physical tokens that are set up on an interactive tabletop surface. Via iconic labels, each token represents a specific activity (e.g., watching TV, doing the laundry), and users create their household routines by configuring these objects on the tabletop’s surface. The application provides real time feedback and recommendations relating to the objects arranged on its surface, including an overall score representing resources consuming by the entire schedule. Users can to set the system to display these cues as either ecological or financial messages (e.g. by displaying the CO2 footprint or expenses accumulated).

The application relies on the the ReacTIVision tracking software, using the TUIO message protocol. The tokens used in the system were 7 by 7 by 2.5 centimeters wooden cuboids with iconic labels affixed to their uppermost surface and ReacTIVision markers on their base. 27 tokens were deployed, representing 13 unique activities.

Eco Planner

Specifications

- The interactive surface was 78 by 57 centimeters in size and composed of a diffusing layer (5mm thick Evonik ACRYLITE 7D006) placed on top of a sensing layer (10mm thick Evonik Endlighten);

- The sensing layer was wrapped with a string of 850nm IR diodes and a near throw projector and IR camera positioned underneath the plastics such that both could address the full area of the interactive surface;

- Infrared Flat Camera for Multitouch Screens (850 nm bandpass filter) with a vari-focal IR Lens, 2.6-6 mm, m12 (Board) Mount;

- Hitachi CP-AW100N projector.

The ITS

Participants interacting with the ITS

In collaboration with students at KAIST, a short project resulted in the decoupling of the MoleBot’s controls. This allowed for it to be controlled remotely over an UDP connection, enabling the quick deployment of new input devices, such as Android smartphones.

Materials

- Dimmable Infra Red LED Light Bar (11.8 inches = 300 mm) (Infra Red 850 nm) (2x)
- Infrared Flat Camera for Multitouch Screens (850 nm bandpass filter)
- Vari-Focal IR Lens, 2.6-6 mm, m12 (Board) Mount
- MP160 Projector from 3M (plus included tripod)
- Thick diffusor
- Plexiglass (optional if the diffusor is thick enough to support interaction)
- LEGO
- Fabric and velcro (optional, use what’s available)

Instructions

- Create a LEGO structure with 47x37x94cm
- Cover it with the desired material (e.g. attach fabric using velcro)
- Hang the LED light bars on the sides of the structure, so that they stay perpendicular to the floor

Multitouch table

Thesis

The thesis was presented on the 23rd of July of 2010, to a jury composed of Professor Ian Oakley, Professor Jos van Leeuwen, and Professor Leonel Nóbrega. It was classified with an 18 out of 20, and was the winner of the Fraunhofer Portugal Challenge 2010.

Abstract

Tangibles User Interfaces (TUIs) have matured to a point where they can be considered as an established area of Human-Computer Interaction research. Despite this, most TUIs are built to address very particular tasks found normally in research labs, kindergartens, or specific work environments. This dissertation contributes with the design and implementation of two Tangible Interfaces that tackle common problems of typical users – Mementos, a TUI for tourists and travelers, and Eco Planner, a TUI that focuses on sustainability. Finally, this dissertation also reviews the theories on Embodied Cognition, opening the discussion on the possibility of using them as core to new frameworks or guidelines to aid TUI developers in creating systems that feel more like real world tools.

This work was the winner of the Fraunhofer Portugal Challenge 2010.

Independent travelers, in particular, are typically in unfamiliar surroundings, often grappling with unknown languages and dealing with unusual climates whilst they perform complex tasks such as navigation and the collaborative planning of spatial-temporal itineraries based on significant quantities of complex textual information from guidebooks and timetables.

The Mementos system was designed to address these concerns and bring the benefits of tangible interfaces – the physicality, the seamless integration with the environment, the support for memory, epistemic action and collaboration – to the domain of tourism and travel. By doing so, Mementos not only offers a novel vision of how digital technology can support travel, but also provides a practical exploration of the main focus on this paper: how tangible interfaces, and token-based systems in particular, can be designed to support complex real world application domains involving multiple contexts and use scenarios.

As most people do not travel alone, and in order to support the group tasks facing travelers (e.g. planning, navigating, and sharing media from their vacations) the Mementos system is composed of three separate parts: a set of tokens, a kiosk interface and a home interface.

Tokens

The tokens are small physical objects intended to be held in the hand or stored in pockets and key chains. Two classes were used. One set of concrete tokens represented and visually resembled specific tourist sites. For example, such a token might be linked to the Eiffel tower and take the form of a model of this monument. The other class of abstract tokens represented more general tourist infrastructure such as a set of cafes and transportation points and appeared as neutral coin-like objects identified with graphical logos. This way, sets of such tokens can be distributed for particular locations or cities in much the same way as guidebooks are currently.

The tokens are designed to provide information related to the object they represent through non-intrusive feedback. Most significantly, this is delivered via vibrotactile cues mediated by location awareness – the tokens vibrate when approaching the location (or set of locations) they represent. Furthermore, in the case of the concrete tokens, they also respond to the proximity of transportation links leading to their location. This feedback can be silenced by touching or picking up the token.

The kiosk interface

The kiosk interface was designed as a public display showing an interactive map and capable of recognizing and responding to the tokens. Kiosks are intended to be distributed around a city and at key tourist sites. Interaction is highly constrained and based on three spatially ordered sensing zones – running from left to right in front of the display. Users can place one concrete token and one abstract token on each to visualize and communicate their plans.

Kiosk's interface

The goal of this interface is to create simple queries relating to the kiosk location and other areas or resources of interest and transportation options between these sites. For instance, placing a concrete token on the leftmost sensing zone caused transportation information between the kiosk and concrete site to be presented (e.g. estimates for travelling time and cost by taxi, bus and foot). Adding an abstract café token to the same sensing zone causes dining options to be displayed in the proximity of the concrete site. In a similar manner, the three zones could be used to create multi-leg travel plans.

Preliminary study

The home interface

The home system allows users to quickly access photos and videos taken during trips on their home PCs. Placing one of the concrete tokens used while travelling on a sensing zone attached to a computer shows the media recorded in the associated site. In this way, the tokens take on a role not only of souvenirs, but also as true keepsake objects, holders of stories and memories. Users will also be able to distribute their tokens to friends and family as a personalized way of sharing mementos of their trips.

The home interface

Implementation

The Mementos system was functionally prototyped using a range of commercially available technologies. The tokens were based on the Bluetooth based SHAKE sensor platform. Among other functions, this matchbox-sized device incorporates a vibrotactile actuator and two surface mounted capacitive sensors.

The kiosk and home interfaces were reliant on RFID technology to identify tokens; three RFID readers were used to produce the kiosk, while the home interface was based on a single reader. In both cases, low-cost touchatag readers, which use coin-sized stickers as tags, were used to develop the system. The kiosk and home interfaces were implemented in Processing and used the touchatag-processing library.

Spotter is a project developed during the Interface and Interaction Design class (Location-aware tool for supporting the tourism experience) with the goal of allowing athlete tourists who enjoy Windsurf to connect to the “insider” community of the place they are visiting. To achieve this, the system would allow people to exchange rides and equipment, and, since the perfect conditions for Windsurf are always changing, the system should also enable the users to share just-in-time information about the conditions in a spot.