These projects are usu­al­ly found on plat­forms like GitHub. Devel­op­ers and researchers put their work out there for oth­ers to use and build upon. This is dif­fer­ent from the big, closed-source mod­els from large tech com­pa­nies where you can only use their tool through a web inter­face. With open-source, you have more con­trol, but it also means you’ll like­ly need a decent com­put­er, often with a pow­er­ful graph­ics card (GPU), and you'll have to be com­fort­able with some tech­ni­cal set­up.

Let’s look at a few of the main projects peo­ple are using right now.

Stable Video Diffusion

This is one of the more well-known names, and it comes from a com­pa­ny called Sta­bil­i­ty AI. They are the same peo­ple behind Sta­ble Dif­fu­sion, a pop­u­lar open-source image gen­er­a­tion mod­el. Sta­ble Video Dif­fu­sion is essen­tial­ly an exten­sion of that, designed to cre­ate short video clips. It comes in two main ver­sions: one that can gen­er­ate 14 frames and anoth­er that can do 25 frames. You can also adjust the frame rate, any­where from 3 to 30 frames per sec­ond.

It pri­mar­i­ly works as an image-to-video mod­el. You give it a start­ing image, and it ani­mates it, cre­at­ing a short video clip. This is use­ful for adding motion to sta­t­ic images. For instance, you could take a pic­ture of a calm lake and make the water rip­ple. The code is avail­able on GitHub, and the mod­el weights are on a plat­form called Hug­ging Face, which is a com­mon place for devel­op­ers to share AI mod­els.

Get­ting it run­ning involves a few steps. You'll need to have Python installed on your com­put­er, along with some spe­cif­ic libraries that the mod­el depends on. You then down­load the mod­el files and run a script to gen­er­ate the video. While it's intend­ed for research, peo­ple have found cre­ative ways to use it. It’s a sol­id start­ing point if you’re new to this because it's well-doc­u­­men­t­ed. But, it does have lim­i­ta­tions. The videos it cre­ates are short, usu­al­ly just a few sec­onds, and it can some­times strug­gle with cre­at­ing pho­to­re­al­is­tic results.

ModelScope Text-to-Video

Mod­elScope is anoth­er sig­nif­i­cant open-source project, devel­oped by Alibaba's DAMO Acad­e­my. Unlike Sta­ble Video Diffusion’s pri­ma­ry image-to-video func­tion, Mod­elScope focus­es on text-to-video syn­the­sis. You give it a writ­ten descrip­tion, and it gen­er­ates a video based on that text. This is pos­si­ble because the mod­el has about 1.7 bil­lion para­me­ters, which are the vari­ables the mod­el learns from data to per­form its task.

The archi­tec­ture of Mod­elScope is bro­ken down into three parts: a text fea­ture extrac­tor to under­stand the prompt, a dif­fu­sion mod­el that works in the 'latent space' to trans­late text fea­tures into a video rep­re­sen­ta­tion, and anoth­er com­po­nent to turn that rep­re­sen­ta­tion into the final video you see. This whole process starts with ran­dom noise and grad­u­al­ly refines it until it match­es the text descrip­tion you pro­vid­ed.

Like Sta­ble Video Dif­fu­sion, you can find the code and mod­el for Mod­elScope online and run it your­self. There are even tuto­ri­als and Colab note­books avail­able, which are a way to run the code in a web brows­er with­out hav­ing to set every­thing up on your own machine. How­ev­er, it’s worth not­ing that the mod­el was trained main­ly on Eng­lish text and pub­lic datasets, so its out­put might reflect the bias­es present in that data. It also has trou­ble gen­er­at­ing clear text with­in the video and isn't per­fect for cre­at­ing long, high-qual­i­­ty cin­e­mat­ic pieces.

A relat­ed project, Zero­Scope, is an improved ver­sion of the Mod­elscope mod­el. It’s been specif­i­cal­ly trained to pro­duce videos with a 16:9 aspect ratio and with­out the Shut­ter­stock water­mark that some­times appeared in the orig­i­nal. Zero­Scope comes in two ver­sions: one for faster cre­ation at a low­er res­o­lu­tion and an XL ver­sion that upscales videos to a high­er res­o­lu­tion.

Other Notable Projects

The open-source AI video world is mov­ing fast, and new projects pop up reg­u­lar­ly. Here are a few oth­ers to be aware of:

• Lat­te: This project uses a dif­fer­ent archi­tec­ture called a Latent Dif­fu­sion Trans­former. It works by break­ing down a video into a sequence of tokens in a latent space and then uses a Trans­former (a type of neur­al net­work) to mod­el how these tokens relate to each oth­er to gen­er­ate a video. It has shown strong per­for­mance on sev­er­al stan­dard video gen­er­a­tion bench­marks. The code and papers are avail­able if you want to dig into the tech­ni­cal details.

• Open-Sora: This is an ini­tia­tive that aims to repli­cate the results of OpenAI's impres­sive, but closed-source, Sora mod­el. The goal is to make high-qual­i­­ty video pro­duc­tion acces­si­ble to every­one by being ful­ly open-source. They pro­vide the code, mod­el check­points, and train­ing details. This project is ambi­tious and is built on the work of many oth­er open-source mod­els for han­dling images, text, and video.

• Hun­yuan­Video: Devel­oped by Ten­cent, this is a large mod­el with over 13 bil­lion para­me­ters. It's known for gen­er­at­ing high-qual­i­­ty, cin­e­mat­ic videos and has good align­ment between the text prompt and the result­ing video.

• Mochi 1: Cre­at­ed by Gen­mo AI, Mochi 1 is a 10-bil­lion-para­me­ter mod­el built on an Asym­met­ric Dif­fu­sion Trans­former archi­tec­ture. It's rec­og­nized for its cre­ative out­put and strong adher­ence to prompts for both text-to-video and image-to-video tasks.

How to Get Started: A General Guide

If you want to try run­ning one of these mod­els your­self, the process gen­er­al­ly looks like this. Let's use Com­fyUI as an exam­ple, as it's a pop­u­lar and flex­i­ble tool for run­ning these kinds of mod­els local­ly.

1. Get Com­fyUI: First, you need to down­load and install Com­fyUI. It’s a node-based inter­face, which might look intim­i­dat­ing at first, but it gives you a lot of con­trol over the video gen­er­a­tion process. You essen­tial­ly con­nect dif­fer­ent blocks (nodes) to build a work­flow.

2. Down­load the Mod­els: You'll need to down­load the spe­cif­ic AI mod­el you want to use. For exam­ple, if you're using a mod­el like Wan 2.2, you'd down­load its mod­el files. These files are often large, sev­er­al giga­bytes each. You'll also need to down­load sup­port­ing mod­els, like a VAE (Vari­a­tion­al Autoen­coder) and text encoders, which help the main mod­el func­tion. These files need to be placed in spe­cif­ic fold­ers with­in your Com­fyUI instal­la­tion direc­to­ry.

3. Load a Work­flow: Many projects pro­vide pre-made work­flow files, often in a JSON for­mat. You can drag and drop this file direct­ly onto the Com­fyUI inter­face, and it will auto­mat­i­cal­ly set up all the nec­es­sary nodes for you. This saves you from hav­ing to build the work­flow from scratch.

4. Con­fig­ure the Nodes: Once the work­flow is loaded, you'll need to tell each node which mod­el file to use. You’ll typ­i­cal­ly see drop­down menus on nodes for the main mod­el, the VAE, and the text encoder (often called a CLIP mod­el). You just select the files you down­loaded ear­li­er.

5. Enter Your Prompt and Gen­er­ate: With every­thing set up, you can now write your text prompt in the appro­pri­ate node, adjust set­tings like video dimen­sions and length, and then click the but­ton to gen­er­ate the video. Your com­put­er will then start work­ing, and depend­ing on its speed, you'll have a video in a few min­utes.

Run­ning these mod­els does require a good amount of com­put­er mem­o­ry and a capa­ble GPU. Some mod­els have ver­sions that are opti­mized to run on sys­tems with less VRAM, but a pow­er­ful machine will always give you a bet­ter expe­ri­ence.

The open-source AI video space is active and con­stant­ly chang­ing. New mod­els and tech­niques are released fre­quent­ly. By get­ting involved, even just by run­ning the soft­ware, you can get a real sense of what this tech­nol­o­gy can and can't do right now.