[Editor's note: While "Observatory Night Shift" sadly could not be completed for LD51 due to unrelated setbacks, I've decided to publish this blog post anyway in the hope that it will be helpful or interesting to people.]
Hi everyone!
I'd like to take a little time to lay out some of the processes we've been using to create AI-generated art for our LD51 Jam game, "Observatory Night Shift". Hopefully this will give people a clearer idea of what's involved in these techniques, beyond the very simple idea of "click button, get art"!
Goal
Our goal today will be to create an exterior image of the observatory building the game takes place in, suitable for use as a screen in an adventure game. We expect this to be one of the three main screens the player sees during the game, the first screen a player sees during the game, and most likely part of the key art we will use for the game's cover images and banners. As a result, we're willing to spend a bit more time on it.
We start with a mental image of the kind of picture we want. As anyone who has commissioned art will know, accurately translating an image in your head into a written description is very challenging, but hopefully over the course of this post you'll come to get a clear idea of what our image of the observatory building looks like.
Prompting
Of course right away, our first task is to try to describe our desired image for the AI in text form; this is called a "prompt". We generally won't figure out a prompt that accurately represents our image first time, so it's okay to start simple and build up to it. In this case we know that we want an observatory building drawn in a realistic (but not photographic) style. It must be shown at night somewhere fairly remote - perhaps on a hillside - and we want some visual interest in the sky; perhaps some aurora borealis?
Finally, we address the problem that the AI doesn't actually know that it's supposed to be making good images; to it a nice image and an ugly image are interchangeable. By asking it for an image that is "trending" we are not actually expecting it to look at any specific trending images; rather, we're asking it to use trends as a guide for what kinds of images look good or bad, according to the aesthetics of social media users.
Taking all of this together, we have our starting prompt: observatory building on a hill at night, aurora borealis, realistic, trending. We give this prompt to our generation tool - for this example we are using MidJourney - and we ask it to create a few 16:9 images based on it.
The results are... borderline passable? If you were in a hurry and didn't care too much about this asset, you could use one of these images as-is -- much like finding a vaguely suitable image on a stock image site and using it without modification. But these pictures just aren't representative of the image we envisioned, so we're going to have to do a lot more work to get them there.
Making It Better
First, we identify what we don't like about our initial batch of images:
- The buildings are too small in the frame
- The observatory's iconic dome should be more clearly a telescope, not just a dome
- The aurora borealis is a bit overpowering
- The "realism" art style doesn't show well with the nighttime environment, and is a little line-heavy.
We attempt to address the architectural and framing issues by adding a couple of new requirements: observatory building closeup, telescope. We also adjust the relative "weights" of parts of the prompt, telling the AI that observatory building closeup should be considered twice as important as usual, but telescope and aurora borealis should be considered half as important as usual, since those elements shouldn't dominate too much.
Better! We try a slightly different art style, replacing realistic with illustration, avant garde:
Eh... maybe not? These are getting little too painterly, so we abandon the Avant Garde motif. Finally we remove the aurora and adjust the style again; our new full prompt is: observatory building closeup, telescope, observatory building on a hill at night, starry sky, illustration, realism, trending. Early impressions of these results are decent, so it's time to start hammering that randomise button!
With 60 images to look over, we're starting to get a good sense of the AI's understanding of this prompt. At this point it's interesting to compare the source images that the AI used to learn about observatories; we can see how the AI's concept of what an observatory looks like has clearly been influenced by these images, but it isn't ever directly reproducing them.
We pick out three promising looking images and ask the AI to perform further variations on them, essentially running the algorithm longer to try to improve the details of the image in different ways that might be interesting. This is a technique we'll be using a lot more later.
But unfortunately it's becoming clear that this current prompt just isn't going to get us what we want. While the architecture is sometimes decent (even if it can't figure out telescopes at all), the style lacks the fun and vibrancy that we want for game art. So it's back to the prompt drawing board!
What follows is an iterative process of trying various different styles in different combinations; 50s decor, neo-baroque, art deco, neo-Rococo, modern, and photobash. We also experiment with terms like 'intricate' and 'highly detailed', and explore some variations along the way. This takes a while:
A long while.
Phew! While some of the 50s styles were very nice, towards the end of this process we settled into small observatory building, nighttime, neo-rococo, modern, photobash. At this point we've stopped asking the AI to produce images it thinks could "trend" on social media; the style is now defined precisely enough that we're happy for the AI to take more risks with the aesthetics, and we'll take responsibility for figuring out what works and what doesn't. Looking over the many images produced, we pick out a few of the most promising examples:
We put these images back into the AI, now giving it much longer to work on them using a more powerful model, a process that MidJourney calls remastering. We use this process to produce a number of versions of each image, then further refine the best of those, producing new variations of whichever images we like the most.
Results
Ultimately, two candidate images stand out, and we upscale both of them to game-ready resolutions:
Here, we pause; at this point we want to move to designing the observatory's interior, so we can make sure it's consistent with these designs before we make final selections.
In normal development, at this point I would consider the process roughly half-finished. There are a lot of improvements still to be made, from working with the AI to replace specific details of the image (like making the main doors more prominent and clearly interactable), to post-processing the final images in traditional art packages like photoshop (to tweak the colour balance and curves, or perhaps add some vignetting).
But within jam time constraints, we're probably done; anything more can be considered polish. The process has taken a little under 2 hours; looking back we can see the path that has led us, through many dead ends and iterations and more than 250 generated images, to the final images that ultimately reflect our original vision pretty well.
Final Thoughts
What do you think? Has this explanation been useful in helping you understand some of the processes and techniques that are involved in generating art with an AI? If you create AI art yourself, are your own processes very different from these? I'd love to hear from everyone!
Thanks for reading.
[Editor's note 2: Some details of the prompts and processes involved have been omitted or simplified to avoid complicating this already overwhelming article too much; if you'd like more detail about anything please feel free to ask!]
