Using Machine Learning to Make Art
Markov chains are probably best known for powering autocomplete on search engines. Here’s what happened when I applied them to art.
Markov chains are machine-learning algorithms that seem to permeate our everyday digital lives, powering everything from predictive texting to search engines to Twitter bots. Fundamentally, they are about predicting new possible outcomes given a linear sequence, much in the way other machine learning algorithms function. But their ability to perform reliably even with limited data sets creates some interesting use cases.
Conversations around machine learning and how it can help clients are part of nearly every project at Huge, and the predictive capabilities of various algorithms offer insights into patterns that would often be missed if being reviewed by human eyes. But machine learning can be about much more than driving traffic or returns. There are also countless opportunities to use it for play and creativity, such as generative art.
Understanding Markov chains
Before diving into how machines can be trained to create art using Markov chains, it’s helpful to first understand how Markov chains work.
Markov chains fall into the category of computer science of machine learning, which revolves more or less around the idea of predicting the unknown when given a substantial amount of known data. While other machine learning concepts like neural nets and decision trees may deal with identifying what something is (categorization), Markov chains deal with guessing (based on probability) what the next element in a sequence could be.
Let’s take a look at some sample data:
Today it is hot and sunny
Today it is cold and cloudy
Today it is cold and snowing
While these phrases form a very small sampling of data, they are enough to demonstrate the working of a Markov chain. When building out the trained sequence (hereafter referred to as “the model”), the most critical decision for the problem lays in picking what “order” chain we want to use.
The order of a Markov chain means how many items (in this case, words) are significant to the next choice. So if the order is `1` then it will only look back one word to see what options there are for the next word in the chain. But if the order is, say, `5` then the model will check the previous 5 words and use their order to help determine the next word.
So from the sample above if we choose a second-order Markov chain (2 words), we can quickly see that after the phrase “Today it” every possible option is the word “is,” so there is a 100% chance that is the next word chosen. At the next step, however, it gets interesting because after the phrase “it is” there is a ⅓ chance that the word will be “hot” and a ⅔ chance that the word will be “cold.” If we randomly flip a theoretical three-sided coin with two heads and one tail, where heads is “cold” and tails is “hot,” let’s say we get heads. Our next word is “cold”. Now that we have gone down this path, we keep going. The next word is 100% going to be “and.” Finally, we get a 50/50 chance or either “cloudy” or “snowing” for the last word of our sentence.
This becomes powerful, because just from this very limited data set at only a second order chain, our very specific chain can predict (based on these sentences only) that if it’s cold it is probably either cloudy or snowing.
An algorithm’s lack of insight
While it may seem that there is some awareness to the algorithm/machine, once you understand how a Markov chain works, it can also be seen that it’s just about building out probability for each step. So while there may not be as structured a pattern as we see in language, the below data is also something we could train a Markov chain on:
◼▲★•••★
◼★••▲▲
▲▲▲▲▲
From these three lines of data, a similar idea of word structure/grammar is built. When using the English language and a large enough set of training data, the chain will begin to make sentences that resemble human speech patterns (like predictive typing on your smartphone). But those words are just abstract tokens.
So even when using images, patterns are learned. If we look at the three lines above using a Markov chain of order 1, then we can see after a triangle, there is almost always another triangle, but sometimes a star. So if we start a “sentence” with a triangle, we will have a high chance that all our “words” are triangles until we are lucky enough to get a star. And a star is never followed by a triangle, from the examples above.
The possibilities that this opens up, then, is that even more abstract training sets can be used.
Images as sentences
Let’s take a look at the three 200x200 pixel images below.
Each of these images can be treated as a “sentence,” to stick with our previous metaphor, with each pixel being a “word.” The concept of order stays the same, in that we say that the color of the next pixel in the image (the next “word” in the “sentence”) is determined by however many pixels came before that.
So after we use all these pixels to create our model of possibilities (here saying that the last 200 pixels of data are all important to what the next pixel color is; aka a 200-order chain), we pick a random sample of 200 sequential pixels to start our new image, and then flip theoretical coins to get the next pixel in the sequence until our image has a total of 40000 pixels, and (just like in the English language), we are going to wrap pixels to the next line if they don’t fit on their current line.
Generating a number of images based on this data, the below are just a few of the possibilities:
The “learning” in machine learning becomes fun through delight in the unexpected. While I expected to see just squares or possibly rectangles, in fact there are far more diverse formations from their combinations and placements than existed in the three images we trained the algorithm on.
Part of this is because the “order” that we selected to train the model at was so small for a sequence of this size, and our dataset was small-ish as well. Nonetheless, given a runway of 200 pixels and the simple images above, we get some pretty cool results.
Glitchy Success
The model I built was robust enough to preserve color from the images it learns from. So what happens when we train it on a much more complex image like the one below?
While still simple and abstract, there are a lot of breaks away from from the square geometry we saw before, and in terms of sequencing, a lot more options for what pixel may come next. Running this through the algorithm at an order of 200 and creating three random images, we get the following:
This is a bit disappointing as ultimately it’s very familiar shapes that are offset and either squished or stretched. Clearly, though, we can see that the Markov chain learned some patterns; they’re just very specific. So what happens if we turn down the order and say that only the last 100 pixels should be used to predict the next pixel?
While this may look like a failure, it’s actually discernable that patterns are being learned. We can see alternating lines of color between dark blue, light blue, purple, and pink, as well as purple and dark blue’s tendency to follow themselves in order, creating those horizontal lines. And pink is quite rare compared to the other colors from the original image.
Real Art from the Bauhaus
What happens when we train a Markov chain on something slightly less curated? Training with an order of 200 generated some pretty cool results; as the images become more diverse, this limits the possibilities of them having alternative paths. When we look at fewer pixels, it can go in more directions. What would happen if we only looked at the past 1 pixel in the previous examples to determine the next?
To limit the current colors and options, I Googled “Bauhaus” and came up with a set of paintings and designs based in the Bauhaus movement in graphic design and cropped them to 200x200 pixels, reducing the color palette significantly.
By providing a much larger sampling as training data, we can have the chain generate art based on a random starting point.
The images above are a random sampling of generated images from the Bauhaus training set at an order of 120. Here we can see that while patterns emerge and repeat, creating newer and more “interesting” design, each work is mostly “inspired” by a single original image. The runway of 120 pixels is so significant given the complexity of the images that it’s difficult for the algorithm to transition.
In this set of images however, the order is scaled down to 90, and while there is still notable repetition, it can be seen that colors and lines from multiple works are present in each image. Still, a significant portion of the images are more or less visual noise.
Taking it to the extreme, the above images are at an order of 1. What’s cool here is that even though we lose geometry, we can see patterns in terms of lines of consistent color. And despite how far departed these images are from the original samples (as with the “grammar” metaphor, the algorithm is guessing which pixel comes next just based on whatever the last pixel was), they are quite visually pleasing. So much so, I decided to upload the images and have them made into this Bauhaus-inspired, Markov-chain created T-shirt:
Check out the code and train your own!
Exploring the visual properties of Markov chains was tremendous fun, and you can see the source code on my Github—though a lot was written quick and rough, so please have mercy.
