A complete walkthrough of the AI music video generator
There is a video located at the bottom of this page that provides a detailed explanation of all the steps discussed in this blog post.
Hello, friends! In this post, we'll delve into creating captivating music videos using the AI platform, Neural Frames. This revolutionary tool creates videos from text prompts, making it a perfect companion for the generation of AI music videos. Today, we'll create a dynamic video in under ten minutes, and trust us, it's easier than you think.
Choosing the Right AI Model
First, we have to decide on the AI model for our project. Neural Frames provides six standard models, each tailored for specific use cases. You can choose from three 'all-rounder' models that cater to broad requirements or opt for the three specialized models designed for realistic vision, analog photography, or comics and mangas.
The platform also offers the opportunity to train custom models. With this option, you can train an AI model based on yourself or any other object. But for today, let's work with a standard model. For our tutorial, we'll use the 'Dreamshaper' model.
Setting the Starting Frame
Now, we need to set the first frame for our video. You can either upload an image as the starting frame or create the first image within the platform. In this case, we'll create our own.
In the first frame editor, type in a text prompt describing what you want to see. For our music video, we'll aim to depict the 'evolution of humankind,' starting with a prehistoric cave with fire. To make our prompt more effective, we'll use the 'Pimp My Prompt' button, which employs AI to enhance our prompt for the AI model.
After setting the image format (we'll use 16:9), click on 'Render.' This process gives us four image options for our starting frame, and we choose the one we prefer.
Navigating the Video Editor
Now, we find ourselves in the video editor, which consists of three elements: the timeline, the preview window, and the settings.
The timeline comprises three parts for prompt inputs, modulation, and music. At this point, we'll add a song by double-clicking on the audio timeline. Neural Frames then extracts the stems of the song, providing the individual elements like the snare and kick drum.
For our project, we could render the prompt with some settings, adjust the 'trippiness' and movement settings, or delve into pro mode to manipulate individual settings.
Modulation and Parameters
But let's take things up a notch and add modulation based on an element of the song. For instance, modulating the snare creates an impactful effect.
Two important parameters here are 'Strength' and 'Smooth'. Strength determines how much the new image will differ from the old one; a higher strength forms a very different image, while a lower strength sticks closer to the original image. Smooth, on the other hand, determines how much we interpolate between two neural network outputs. A higher smooth value introduces more images between two outputs, making the transition smoother. You can create super trippy visuals here.
If you're using modulation based on rhythm elements, use a low smooth value. Otherwise, the modulation effect might get lost. As a rule, try not to change the smooth value mid-video as it can lead to strange visual effects.
Adding Prompts and Adjusting Settings
Now, let's add more prompts to our video. After each prompt, use 'Pimp My Prompt' for the best results. As the video progresses, you can also introduce some movement. For instance, a slight zoom-in effect can add interest.
Once you're satisfied with the settings, click on 'Render' and watch your creation. With Neural Frames, you can view your video at any point and rerender from any segment if you're not satisfied. But chances are, you'll love what you see!
Finalizing the Video
The last step is to continue adding prompts to cover the entire duration of your song. Once done, render your project and marvel at your AI-generated music video!
In essence, Neural Frames makes ai music video generation a breeze, empowering you to create engaging visual narratives effortlessly. The power of AI is at your fingertips - it's time to unleash your creativity!
A full walkthrough how to create a music video from text with neural frames
No VC money, just a physicist turned indiehacker in love with text-to-video. Contact our team here: help@neuralframes.com. This website is greatly inspired by Deforum, but doesn't actually use it. For inspiration on prompts, I recommend Civitai.