JoyAI-Echo
🎬 Pushing the Frontier of Long Video Generation
Official model weights for minute-level multi-shot audio-video generation with a distilled DMD generator, paired cross-modal memory, and story-level consistency.
For academic research and non-commercial use only.
📄 Paper |
🌐 Project Page |
💻 Inference Code |
🧬 Model |
🚀 Usage |
📊 Results |
📝 Citation
## Model Summary
**JoyAI-Echo** is a long-form, multi-shot, audio-video generation framework that breaks the barriers of error accumulation, weak temporal coherence, and prohibitive latency in long video generation. A cross-modal audio-visual memory bank preserves character appearance and voice timbre consistently over **five-minute** videos, while a post-training pipeline combining memory-based reinforcement learning with distribution matching distillation (DMD) delivers a **7.5× inference speedup** without sacrificing quality.
JoyAI-Echo decisively outperforms *HappyOyster* (directing mode) on long-form generation and even surpasses the short-video specialist *Wan 2.6* on human-centric tasks.
This repository hosts the **released checkpoint**. Inference code is released separately — see the [Usage](#usage) section.
## Model Details
- **Developed by:** Echo Team @ Joy Future Academy, JD
- **Model type:** Text-to-(Audio+Video) diffusion transformer, DMD 8-step
- **Modality:** Text → synchronized video + audio
- **Backbone:** Built on top of [LTX-Video](https://github.com/Lightricks/LTX-Video)
- **Text encoder:** [`google/gemma-3-12b-it`](https://huggingface.co/google/gemma-3-12b-it) (downloaded separately)
- **Resolution / length (by default):** 1280 × 736, 241 frames @ 25 fps per shot
- **Max story length:** up to 5 minutes (multi-shot)
- **License:** LTX-2 Community License Agreement
## Highlights
- 🎞️ **Minute-level multi-shot stories**: generate a sequence of coherent shots from one prompt JSON.
- ⚡ **DMD-distilled few-step inference**: ~7.5× faster than the original pipeline.
- 🔊 **Joint audio-video generation**: one pipeline produces synchronized video and audio.
- 🧠 **Paired cross-modal memory bank**: conditions each new shot on prior visual identity and voice context for story-level consistency.
## Demo Gallery
Explore long-form and short-form JoyAI-Echo cases on the [Project Page](https://echo-team-joy-future-academy-jd.github.io/Echo-LongVideo-Page/). 🍿
## Usage
Inference is run with the standalone **JoyAI-Echo** inference repository.
### 1. Download the checkpoint
```bash
huggingface-cli download jdopensource/JoyAI-Echo \
--local-dir checkpoints
```
Also download the Gemma text encoder:
```bash
huggingface-cli download google/gemma-3-12b-it \
--local-dir checkpoints/gemma-3-12b
```
Expected layout:
```text
checkpoints/
├── echo-longvideo-release.safetensors
└── gemma-3-12b/
```
### 2. Get the inference code
```bash
git clone https://github.com/jd-opensource/JoyAI-Echo.git
cd JoyAI-Echo
```
Environment: **Python 3.11 + PyTorch 2.8 + CUDA 12.8** (see the inference repo's `environment.yml` / `requirements.txt`).
### 3. Write a story prompt
**Enhance your prompt first.** We provide prompt enhancers — system prompts that expand a short story or idea into well-formed shot prompts: **`prompts/long_story_writer_system_prompt.md`** for long, multi-shot video, and **`prompts/short_story_writer_system_prompt.md`** for single-shot short video. We **strongly recommend** running your input through the matching enhancer before inference; un-enhanced prompts tend to produce noticeably weaker results.
Create a JSON file under `prompts/`. Each file is a single object with a `prompts` list, where **every string is one complete shot**. A single string produces one shot; multiple strings produce a multi-shot story, with each new shot conditioned on the previous ones through the paired audio-video memory bank.
Inside each string, write these parts in order:
| Part | What to describe |
| --- | --- |
| **Roles & Subjects** | Describe the appearance of all visible people, including age, build, hair, face, wardrobe, and speaking voice timbre when applicable. |
| **Action & Dialogue** | What the subject does and speaks. |
| **Style** | The overall visual and emotional aesthetic — e.g. realistic motorsport film language, cool daylight, restrained cinematic tension. |
| **Camera Movement** | The shot type and framing or movement — e.g. a stable close-up on the face, or a medium shot from the waist up. |
| **Background** | The setting and scene details behind the subject. |
| **Sound Effects & BGM** | The sounds in the scene and the background music — e.g. room tone, wind, footsteps and fabric, with a soft low music bed under the dialogue or no background music. |
### 4. Run
```bash
python inference.py
```
Outputs land in `inference_result/outputs/