video mockups for physical prototypes

PhysProto is a prototype of a tool for demonstrating physical prototypes. Yes, you read that right, a prototoype of a prototyping tool! PhysProto takes small video snippets of individual actions on a physical device, and then allows others to play with the device virtually.

PhysProto was created in response to one of the outcomes of a series of workshops on the use of video in HCI education. One of the participants voiced a difficulty they were finding during Covid, as students were not able to effectively share physical prototypes with one another.

The system combines short video clips of the physical prototype being manipulated with a state machine representing the various settings of the device, a physigram. The viewer can select actions, such as turning the small wheel on the device, or opening the flap and the relevant video sequence is played. In addition, the actions of the device can be linked to videos showing the effect of the device on the environment.

See an example at: https://hcibook.net/gen/PhysProto/v0/rockall

The example is based on a fictitious brief where the newly independent state of Rockall wishes to deploy remotely-controlled fishery-protection ships. The PhysProto mockup shows a prototype controller for the remote controlled ship. The mockup control device is constructed out of corrugated cardboard and map pins.

The first two tabs in the prototype, ‘Brief‘ and ‘Unplugged‘, provide fixed videos of the project brief and a demonstration of the control device capabilities respectively.

The ‘Device only‘ tab is where users can experiment controlling the device themselves.

On a physical device not all actions are available at any time due to physical constraints. Some of these constraints relate to individual controls or physical elements of the device, for example, if you have turned the wheel fully to the right, you can only move it further to the right, only back towards the centre. More complex constraints may involve several elements, for example, there is a red button underneath the hatch in the controller.

The prototype ‘knows’ about these constraints and only offers available actions based on the current device physical state. In the two screenshots below, the first shows the state where the hatch is closed (also the flag at stop and wheel in the centre). If the user selects “open hatch”, a short video is played of the hatch opening. The second screenshot is after this video has finished and the state is now hatch: open, flag: stop and wheel: centre. Note that the image shows the hatch open and the available actions now include “press red button”.

  

The ‘Device+System‘ tab is similar, but in addition shows the state of the actual system being controlled (the remote controlled ship). The ship also has state variables (rudder direction, engine state) and PhysProto has a model of how actions and changes on the device translate into effects on the controlled system.

When the user performs an action, the device action video is played as before, but in additional short video of the effect on the ship’s state is also shown. Not having a real ship to hand to create the videos a wooden model boat was used … one that my dad made for me out of scrap wood when I was tiny.

ProtoPhys bases the above on a model of the device and system states. This behaviour is specified using the following Google Doc spreadsheet:

The sheet consists of a number of parts:

• Videos – a list of raw videos stored on Vimeo
• Clips – named portions of the videos. For example, video-3 includes lots of actions on the ship’s wheel. Clips wheel-1, wheel-2, wheel-3, and wheel-4 break this up into individual actions and states of the controller.
• Prelims – content for the ‘Brief’ and ‘Unplugged’ tabs.
• Device State – a list of facets of the device, which determine the state variables
• Device State Display – names for all possible states of the device … not actually used!
• System States – a list of named states of the underlying system (the remote-controlled ship)
• Device Actions – list of potential device actions. Each actions has a pre-condition when it is available, specified as a list of the values of various state facets, a blank facet means “don’t care”. For example, ‘turn wheel right’ is only available for device states where the facet ‘wheel’ is ‘centre’. Each action also has a post-condition, the new values of various state variable after the action has taken place. For example, if the user performs ‘release wheel’ when the ‘wheel’ facet is ‘left’, the new value of ‘wheel’ is ‘centre’.
• Device Transitions – This says which video clip to play for a given action given the previous state.
• Device-State Mapping – This says how device states translate into system states. For example, if the small flag on the device is set to the ‘full steam’ and the wheel is in the centre position, the ship’s state is ‘ahead’.

xxx