OpenSong Self-hosted music generation

YuE2, a 3.6-billion-parameter open music model, has a quick start that asks for a 24 GB card. On a 4:35 song it peaked at 21,337 MiB, mostly in the acoustic stage (the NAR), which attended over the whole song at once. Upstream’s NAR can chunk its attention, but the pipeline never passes the setting. ComfyUI, another runtime, already bounds this stage; OpenSong does it on upstream’s own pipeline, with a shim that chunks attention by query. Full quality’s peak fell to 8,804 MiB at the tightest chunk size; a card with more free memory now gets bigger chunks and a bigger peak, on purpose (Case 2).

Before writing any quantization, I compared a third-party int8 build with the full model by ear on matched score, seed, plan mode and length, FLAC against FLAC, and it sounded nearly as good. Only that runtime could load the file, so Compact quantizes upstream’s own weights to int8 on the CPU, before they reach the card. Under an emulated 8 GB budget the one-shot prompt prefill then missed the 6 GiB ceiling by 389 MiB, so a second shim feeds the prompt through in 512-token pieces. Models that won’t fit the live card are greyed out, with the reason shown.

To measure what the bound really costs, I built an A/B rig. It starts a second engine on the same card with one variable pinned, the NAR chunk size, waits for that engine’s health check to confirm the setting, replays the same request, times each phase from the job’s progress events and reads the peak from the memory log. Crashed runs are recorded as results. Chunk 4,096 matched unbounded speed for about a third less memory, so Full quality now sizes the chunk to the free memory at load, with no slider for the user.

Once, the card was found holding 22,850 of its 24,576 MiB with the engine idle. Later full-length runs released memory cleanly, and three theories reasoned from the source had all been wrong: this was a state to catch, not a rule to read. So the engine got a flight recorder: one greppable line per load, job start, job end and cache release, giving what the process holds and what the card reports. The gap is labelled “others”, because the transcriber is a separate process on the same card. Any job that keeps more than 512 MiB is flagged, and every exit path writes its line, failures included.

A music model, a transcriber and a model reload must never share the card. So one worker thread owns the pipeline, and generating, transcribing and switching models all go through its single queue. A queued job cancels at once; a running one sets a flag the pipeline polls. The transcriber has its own Python environment and runs as a subprocess, killed on cancel.

Progress shows a percentage only where a real total exists. Before the acoustic and decoder steps reported theirs, the bar sat frozen for 166 s of a 427 s song; planning shows a count and transcription shows elapsed time, because neither has a knowable end. A deploy ends with a restart that won’t report success until the health endpoint answers with the version just shipped.

Upstream’s quick start targets Linux. On Windows, the PyTorch wheels register the flash-attention operator without building it, and upstream’s auto-detection only checks registration, so generation crashed. Eager mode worked but was slow. The fix is a shim that makes “auto” mean cuDNN for attention and keeps CUDA graphs.

Networking had a similar trap. The server’s LAN hostname resolves to IPv6 first and a plain 0.0.0.0 bind is IPv4-only, so an IPv4 curl got a 200 while browsers reached nothing. The server now builds its own dual-stack socket.