Back in Time
In 2019 I was working on a writing assistance project. I discontinued that project but I thought I’d share what I was working on and what I imagined the future might look like at the time based on some internal documentation.
T-3
Just 3 years before ChatGPT launched, the ‘large’ in Large Language Model meant a 1.5 billion parameters model (GPT-2). A pretty small model by today’s standards at around 100x smaller than GPT 3.5 (ChatGPT). GPT-2 was a ‘pure’ language model in the sense that it just… modelled language. It wasn’t trained to follow instructions, take part in dialogues or perform long-form ‘reasoning’ like newer models.
Alongside GPT-2 (a continue the sentence model), another model called Bert was all the rage. Bert was a masked language model (a fill in the missing words/tokens model). It wasn’t primarily used for generation, but for making nice representations of text that were useful for all sorts of natural language processing tasks. However, I felt that Bert could also be useful for writing assistance and I started a project on just that.
Poiesis Studio
The name of the project was Poiesis Studio. I took a Bert model, fine tuned it on lyrics, added some additional features and created a nice interface. This is what I call a system’s first approach to research and it has been the hallmark of my work at Sony CSL. That is, imagine the system you want to exist, build it out as far as you can and then work out the details later. It’s a way to reach out into the future in a similar way to design fiction, but grounded in engineering.
With Poiesis Studio you could regenerate arbitrary parts of text rather than just performing sentence continuation. You could also choose to change multiple tokens at once of a certain type (e.g. replace all nouns) and indicate restrictions on the tokens that would be generated such as them being adjectives or nouns.
Here is some masked text and possible generations in the interface. The generations would be shown to the right of the interface. The user could refine the masking to generate different sentences.
Poiesis Studio came with a particular creative workflow in mind. The idea was that users were in charge of Aesthetics (evaluating ideas) and Poiesis Studio aided in… Poiesis (generating ideas). This workflow consisting of four steps:
- Perceiving what was already there
- Intuiting how to expand/alter the document
- Composing/generating new content
- Integrating this into the existing document
The plan was to add additional kinds of conditioning such as syllable and rhyming constraints as well as more abstract constraints like emotional constraints. These would be visible annotations explicitly indicating the intention of the user.
It was used by some of the artists collaborating with the Music Team at Sony CSL. Notably, by artists like Nous Sommes Whim Therapy in the AI Song Contest 2021 for their song “Let it go”. Even after ChatGPT launched, I received feedback that it felt more creative to use as a tool than ChatGPT (for lyrics anyway).
Here is a little clip of Nous Sommes Whim Therapy that mentions Poiesis studio on BFM TV.
You’ll recognise the interface.
Design Considerations
There were several design considerations and insights behind this prototype.
Paint Your Mind
We write from the mind’s eye. Normally this latent experience is hidden, but a collaborative writing system would need to ‘know’ our mind’s eye. Hence the explicit annotations provided to the system.
Broad > Fine
We don’t always know precisely what we want to express, but we often have some vague intuitions. The system should be able to work with partial specifications.
Generation as a Query
After we have painted our mind’s eye, hitting “generate” is like asking the question — “what could go here?” We might need to make many querys, refining what we want based on what we find.
An Interface to the Uknown
One can view a musical instrument as an interface to the music that it creates. Discovering how the instrument sounds is an act of play with the instrument as an interface. In a similar way, providing a clear interface for text generation allows a user to work in a text space they couldn’t easily have imagined a priori. It narrows down exploration to a few simple parametres.
A Sense of Agency and Ownership by taking Cognitive Steps
As part of the creative cycle, we are given a sence of agency and feeling of creative participation by taking cognitive steps. It can be hard to define exactly what a single cognitive step is and it may well be different between different individuals based on their experience. For example, a chess player might see many moves ahead as a single cognitive step while a beginner might only think of the movement of individual pieces. In art a cognitive step might be ‘connect these lines’ or ‘fill this shape’.
To feel like we own what we create, we should be able to recognise the cognitive steps we are taking. Otherwise, we end up somewhere that might be interesting, but we might not know how or why we got there. It’s like we ‘checked out’ for a moment and lost our agency. Clearly, one could use ChatGPT to generate long texts that would require a human to perform many cognitive steps. But do you really have a feeling of creative ownership while doing that?
With regards to cognitive steps, here are some images from my slides I dug up exploring the idea.
In Part II
In the next article I’ll write about what I considered the future of writing to be and sketch what I imagined that system might look like.