This post begins an essay about software instruments, which will be serialized on Tuesday evenings for the next five weeks or so.

Introduction

We describe software instruments as a family of interactive computational mediums (i.e. game-like artifacts) spanning both toys and tools, which all possess a stateful representation of an artifact, and a set of verb-like affordances to manipulate that artifact. Instruments may be used to discover or to characterize (‘analysis’) both artifacts themselves, and phenomena arising in artifact space which affect the translation of user intent through operational logics into (paths leading to) artifacts.

We identify ‘discovery’ with practices in artifact space, which may be improvised or rehearsed, but always traces a path from the ‘blank canvas’ (or another starting point) to the user’s goal (e.g. a finished essay). Every performance is mediated through an instrument, which translates user intents into navigation of the artifact space, and user actions into manipulation (i.e. making, or finding) of an artifact.

We identify ‘characterization’ with research distillation, which has to do with the effort of representing artifacts of a given kind (e.g. all taxation policies, or all lighting equations) as points in a navigable artifact space. Models of behavior are typical artifacts in both science and industry, whether they are predictive data-driven models (e.g. of climate change, or consumer preference), or specifications of algorithms (e.g. sharding across Amazon Web Services instances).

Skilled users engage with their instruments in a practice. A painter may apply acrylic paint to canvas stretched on wooden boards, or they may press a stylus against a tablet to manipulate a pixel grid shown on an LCD screen. An ecologist may measure the growth of algae by shining a light through their test tube and calculating its absorbance, or they may build a survey grid on a field site and calculate the density of food from the perspective of a squirrel.

Users typically modify their instruments to alter their expressive ranges, and to explore phenomena related to ease and sophistication of expression, and thus become designers of instruments. They may develop instruments that are useful to navigate complex sociotechnical domains. They may use these instruments to negotiate ontologies, theories, and other characterizations of real-world phenomena.

The next two sections of this essay draw on existing design practices, thoughts on the use of, and conversations with software instruments. We will rely upon developments in complex systems theory since 1980, when Seymour Papert’s book titled Mindstorms, - a seminal work in pedagogy using software instruments, - was published. In the future, we will pursue the application of software instruments to the navigation of extremely high-dimensional artifact spaces, especially the design of interactive artifacts, including software instruments.

We will explore data visualization as a form of interaction design, and hypertext theory as the design of tools for thinking. The future of computational literacy in our technocratic society hinges upon citizen understanding of the models with which we are concerned. The future of work in science also relies on developing expressive forms of reproducibility and proof using software instruments.

Affordance and emergence

From the design standpoint, a software instrument can be said to implement a domain-specific language of interaction, through any combination of keybindings, draggable interface elements, conversational agents, file upload, structured text input, and so on. These inputs may be reframed by the model (as when a Sim paints an image from your files and hangs it in their house), or interpreted by it (as when you direct a Sim to hang up the painting, but they are interrupted by the phone ringing). They may replace other data in the model (e.g. applying a certain file), or act to perturb it (e.g. applying a certain keystroke).

Certain instruments are open-ended, meaning that their artifact space is not just difficult to characterize (e.g. all configurations of a Rubik’s cube), but perhaps impossible (e.g. all songs that can be played on a guitar). From the user standpoint, these different affordances may be combined in ways the designer did not anticipate. The results of an analysis are the skillful result of playing with the instruments of analysis. Therefore, performance - whether from a score, or by improvisation, - is vital to how software instruments are made productive. Written scores and recipes let us understand and exercise the capabilities of a software instrument, in a way that recordings of results alone cannot. We call this capacity that of ‘emergence’ or ‘agency’ in a software instrument.

In the world of procedural generation, it is typical for the user and the designer to be the same person, at least at first. In the world of scientific research, this may initially be true, but no longer after research dissemination or cross-disciplinary collaboration. Thus an issue of reproducibility arises, in the same way that having the same guitar and amplifier settings as a famous guitarist does not, by itself, help you to play like them.

Unlike physical instruments, which can probe the world either directly or through simulacra of it (e.g. the use of paint to capture light), software instruments must substitute data structures and modelling constraints for their referent artifact or phenomenon. There is one kind of exception - when a neural network, a game with purely rational agents, or another kind of artificial life is investigated. Yet our software instruments are also called upon to describe other kinds of object, ranging from portraits and other avatars of people, to ocean-exploring robots, to global financial and climate systems.

Painting does not demand a one-to-one mapping from, say, specks of paint to grains of sand on a beach. Neither does the characterization of a complex system benefit from extreme detail, e.g. literal representation of every physical molecule in that system. Through an instrument and its simplified representation, the intentional and skillful choice of actions on the representation (e.g. brushstrokes on a canvas) is itself a source of insight. The player of the instrument cooperates with it to discover its ‘expressive range’ in artifact space.

Any expert using computers may find in this essay a discussion of the qualities in an software instrument which facilitate high-quality creative output, especially the suitability of its affordances and its representations to an open-ended set of domain-specific analyses.

Any artist using software mediums may find herein an attempt to describe the ‘conversation’ with a digital ‘canvas’, which we claim as vital not only to producing compelling artifacts, but also to engaging on shared ground with a community of fellow practitioners.

Contents

The first part of the background section of this essay will address the specific problem of ‘research debt’, coined in analogy to technical debt by Olah and Carter [1]. Along the way, we’ll critique mathematics pedagogy, learn to tune a platformer game in real time, and discover a better way to code interactive artifacts.

The second part of the background section concerns a case study in shared experiences of navigating (generative) design space, namely the Annals of the Parrigues, written by Emily Short [2]. We will read its appendix, which recounts designing a medium of expression whilst navigating the space it describes, against similar accounts by Vi Hart and Iannis Xenakis.

In the case studies section, we will explore an avatar creation tool from a videogame, and a web-based library for binding data to visual representations.

In the final, theoretical section of this essay, we will introduce terminology for practitioners who intend to develop novel or existing software instruments, and to communicate their results through landmarks, scores, and other methods of orientation in design space using software instruments.

[1] https://distill.pub/2017/research-debt/
[2] https://emshort.blog/2015/12/07/procjam-entries-nanogenmo-and-my-generated-generation-guidebook/