Deep STYLE VIDEO synthesis.
In a previous post I spoke about deep style image synthesis. As I started into playing with the images and their style image counterparts I thought, "Why not do this with video and see what happens?" Video is, after all, a series of images playing one after another to create the illusion of movement. The more images you pack into a second of linear time, the smoother the motion image appears to our brains.
Before going too far into the deep-style weeds, let's take a second to pay homage to Edward Muybridge, who is considered to be the inventor of motion pictures.
Muybridge was tasked with a photographic study to find out whether or not all four feet of a horse were completely off the ground at the same time while at a galloping pace. He did a number of complex (given the era and technology available in the late 1800's) tests and eventually found that horses indeed get all four off the ground.
The resulting piece was called, "The Horse in Motion," which eventually led to the first piece of proto-cinema seen here:
Flash forward (and skipping several college courses on the birth of film) and the current film and TV "framerate" standards in the United States are roughly 24 frames(images)/second –commonly referred to as FPS– for film, with 30 frames/second being the standard for modern broadcast TV in the United States.
The rest of the world uses a standard different to the one we use in the US.
Understanding 24 FPS and what that means to this process
SOURCE FRAMES (IMAGES)
Consider the video at the top of this page. I wanted to do something different with some groovy surf footage I acquired in Costa Rica. In order to do this I had to extract all the individual images from the video footage to run through deep style processing software I use. This process is daunting and time consuming on the best of days.
The frame rate standard I used was 24fps (a number expanded upon thanks to Muybridge's works) and the piece of footage is just about 10 seconds long.
IMAGE EXTRACTION & PREPARATION
I extracted a total of 284 images to run through the software. Each image can take anywhere from 20-120 seconds to process. The software I have is underdeveloped and crashes frequently so there's no walking away to come back to revel in the magic.
Not including the time required to acquire the footage and process it for normal usage, the 284 images took over 12 hours to extract the frames, treat them all, and prepare them for the final video segment.
The stlye image I used for the piece is a famous Van Gogh painting, the identity of which I'll take to the grave, so good luck...
RENDERED IMAGE SEQUENCE
The final output (or rendered) video combines the texture to the original frames and did some wild things but what I loved the most was how the style image dealt with the egdes of the water and surfers, creating loose and evolving geometric shapes with the motion from the ocean...
Hurry up and wait
When you consider that the footage (in the top video) of the just the divers was approx. 1000 images. The total process to create that segment of video took over 24 hours to complete - a figure that does not include the time needed for SCSI dive certification and learning how to use a proper underwater camera rig (not GoPro). It also does not include the time needed to acquire the footage and process it for this particular case.
Was it worth it?
I enjoyed the process and the resulting footage is interesting to watch. The most interesting thing is seeing what AI software does with images when you tell a computer to make one thing from two things.
You can try your hand at making single deep stlye images at deepdreamgenerator.com. It's free and the process is simple and the software is has a clean design that is not off-putting.
Enjoi!
RP