In part II of this series on poiesthetic writing assistance, we introduced Poiesis Studio, our testbed for exploring AI-driven writing assistance. Describing the main technology behind and showcasing the user interface. In part III, we are going to go ahead and demonstrate interactive text generation before sharing the reactions of our users.
We’ve gone to exhaustive lengths to explain how our AI tool works and how to interact with it. Now that you’ve seen how we can specify what the AI model should generate, the magic part start! Let’s begin with a simple example, to familiarise you with generating text with Poiesis Studio. We start with ‘This is some text.” and we generate replacements for ‘some’ and ‘text’ we are shown on the right hand side of the screen with the latest generation at the bottom.
Now let’s choose one of the generations to incorporate back into the text. In this case, we choose ‘This is a joke’ by double-clicking on it. We can see that it replaces ‘This is some text.’ in the editor. We then add a new mask between ‘a’ and ‘joke’ to try and generate an adjective to describe the joke. By clicking on the generate mode button multiple times, we get: “practical”, “running”, “cruel”, “bad”, “sick” and “total”.
Now let’s try to force the adjective constraint in the system and also try and see what happens when we force a verb constraint. We do this by first marking the masked word with ‘ADJ’ and then with ‘VERB’ as shown below.
There are two things to note about the outputs of the system. Firstly, the same words occur for the ADJ constraint as the unconstrained example. That is because the underlying mechanism to stop generating duplicates is associated with a specific masking specification. As these adjectives were already very plausible, it just regenerates them a second time. Secondly, though the VERB constraint is met (according to the part of speech model) in some generations, with verbs such as “talking” and “laughing”, some generations do not meet the constraints (again, according to the part of speech model) and the generated word is outlined to show this. The lesson to learn from this is that, though users can try to force the system to generate text according to constraints, if these constraints don’t match the expected structure in language, it might not be able to do so.
Finally, let’s give the system more flexibility in what it can produce before the word ‘joke’ by adding additional masks.
We get much more interesting variations such as “a classic American practical joke” and “a goddamned sick joke”. Unfortunately, we also see a not as pleasing generation with “cruel and cruel joke”. This sort of repetition can be problematic and we address that repetition later in this article.
Before we get to composing some text with Poiesis Studio it’s worth considering how we can try to prompt the system into generating approximately what we want. If we really try to restrict which words are generated based on very specific context, we will see that the possible generations are also equally restricted. In the following example, I am prompting the system in two ways:
In the first case, I left two empty masks for the AI to generate. In every generation we observed, those two masks were filled by a single place-like sounding word. In the second case, there are two masks which can be used to complete the idea of a village with a river and something that is running in some way. From this we get: “streams running nearby”, “no running water” and “a running stream”.
Streams and water are both things that can have the state ‘running’ and in the context of villages, where people live, running water is a reasonable context. However, the ‘no running water’ generation refers back to the village, while ‘streams running nearby’ refers back to the river. The ‘no running water’ example might be surprising if you felt that you were prompting the system for completions related to the river. However, it’s important to realise that the system is using all the information available in generation, and it doesn’t know specifically what the intention of the writer is. We will discuss the notion of intentions further in part 2 of this article.
In some ways, we could feel that there is some form of semantic and syntactic independence between the first generation problem and the second. In other words, we could generate one without caring about what is generated in the other. After all, in what way does the arbitrary name of a village affect the state of river? However, as with the above example, we are limited by our own imagination. Language models have learnt all kinds of relationships at all sorts of levels of abstraction about text. A single person couldn’t predict all the associations such a model would make in the same way a single person couldn’t imagine all the ways other people would fill in these masks.
The number of masks, the restrictions, and where they are placed, matters a lot. If these are placed in awkward places, where the underlying language model doesn’t have ‘experience’, it might not be able to create reasonable outputs. This is one of the primary reasons why the model might generate malformed generations. However, these models are also not perfect and still an area of active research.
Alright, so we’ve given you an idea of how Poiesis Studio works, what happens under the hood, and how to prompt our masked language model to generate text. Let’s put the intellectual aspects of this aside and write… dammit!
The random nature of generating text with Poiesis Studio always makes demoing the system a lot of fun. We never know how the interplay between the system and the writer will go exactly. As you can see from the video, we started with a very obvious fantasy trope — a dragon. We then set off in an interesting direction that was very different than where we started with the new mythological character — DRAGONMAN.
Poiesis Studio is great at this sort of writing context, where you need some initial ideas to get started. Also, it’s not just the generations that are directly selected by the writer that influences the composition, it’s exposure to different ideas along the way and the journey the writer is going through when actively searching through possibilities.
The story we just started in the previous demo was using the generic ‘WRITE’ plugin. In this next demo we are going to use a plugin for generating lyrics.
It’s not a masterpiece, but we started with nothing, and we ended up with a direction for a song to get started.
There are lots of ways we can improve Poeisis Studio for writing lyrics: directly incorporating a melody, syllables constraints, and other musical elements. These are all fundamental parts of the craft of writing a song. However, in terms of poiesis, Poiesis Studio has sparked ideas that a songwriter can work on. Musicians are already craftsman, and in general, I would argue that it’s much easier to fix something that is slightly off than to fabricate something from scratch.
We’ve had overwhelmingly positive feedback from all people that have used our tool. Typically, the moment that generated text appears on the screen, users immediately understand the utility of the tool. Applying different masks patterns and generating different patterns just cements the idea in the head of the users.
There is a qualitative difference between generating a single word vs multiple words and also the diversity of the generations that are suggested by the system. When only generating a single word, people just think of the auto-completion function one would see on their phone. The completions are typically quite predictable and not particularly interesting. When multiple words are generated at once, it feels like a complete new idea emerges from a single click, and not just one idea. As we generate multiple ideas, it is like our brain is shifting between different possible contexts that we might be inspired by and add to our composition. Perhaps when we construct ideas internally, this process it is closer to the typical quantity of information we might consider when we expand on an idea ourselves? Too little additional context feels obvious, too much feels alien and outside of our control.
The generations aren’t perfect and this affects the perception of what is possible with the system. I think we can broadly group these into three different categories and these often have different reactions from users:
Malformed generations could be as minor as not capitalising words correctly such as “i like france”. It could be incorrect conjugation such as ‘I will ran’. In other cases, it might just be garbage ””’|||||123”. The first two forms are often easily forgivable by users, as long as they are the exception and not the rule. The last form is usually not acceptable. Luckily, the sorts of errors that are made of the third kind are typically easily identifiable by simple rules and can be filtered out during text generation by our AI.
Some generations are ‘funny’. Such as, ‘the car bounced back to heaven’. These frankly make Poiesis Studio a joy to work with. I’ve never had so many questionable laugh-out-loud moments in my office when testing to tool. Humour is of course subjective though, sometimes one person’s joke is another person’s cringe-inducing suffering (just ask my colleagues). So perhaps not everybody would agree that ‘funny’ generations are valuable and enjoyable.
I feel that these sorts of enjoyable generations have something of the pleasure of listening to children reason and explain aspects of the world. As with a child, the system has quite naive associations between words and none of the stigmatised or conventionalised ways of combining them. This might be seen as a failure on the system’s part, but boy is it a hoot!
What I’m calling none-sequitur generations, are generation that feel like that don’t follow from their surrounding context. This often feels like a failure of the system to understand what is already present. These generations can actually still provide useful ideas, but it feels as though flipping through a random book might provide a similar level of help.
One aspect of the system that can be disappointing at times, is when the examples that are generated lack in diversity. If we already generated “the red car drove quickly”, we don’t also need to generate “the green car drove quickly”. Simple variations in colour aren’t particularly interesting. Perhaps, because such variations are unsurprising to writers. I suspect that, if you asked writers to come up with ‘different’ ideas surrounding a theme, they would probably already filter out these kinds of variations intuitively. What we really need, is the variations that would pass a human diversity threshold with respect to previously generated examples.
In part III of our series on poiesthetic writing assistance with Poiesis Studio, we explained a bit about the brains of our system, before showcasing Poeisis Studio and sharing user reaction to it. In part IV, we will dig deeper into the insights we gained from building our system. We will also point out what is missing in our current approach and point at a possible future.
Our take-home messages are as follows:
We are excited to make progress on improving Poiesis Studio, both technically and conceptually by better understanding and supporting poiesthetic writing experiences. Feel free to contact me at firstname.lastname@example.org if you want to get in touch. We are very happy to collaborate with writers, language researchers and techies!