Batch-size sweeps
Model card · Deployment · Flashlight results
Measured median wall-clock seconds, one warmup and three timed runs per configuration. Preprocessing, acoustic inference, score transfer, and decoding are included; model loading, file reading, and HTTP transfer are excluded. These are repeated synthetic English speech capacity tests, not WER results. Apple M5 Pro has 48 GiB unified memory; CPU and MPS runs use six PyTorch threads. Modal L4 runs use four allocated CPU cores. The different host CPUs affect end-to-end decoding times.
Full transcription with KenLM beam decoding
CPU uses FP32; both GPUs use BF16. CPU sweeps stop at batch 8.
Eight 30-second inputs — 240 seconds of audio
| Batch limit | Apple M5 Pro CPU FP32 (s) | Apple M5 Pro GPU BF16 (s) | Modal NVIDIA L4 BF16 (s) |
|---|---|---|---|
| 1 | 5.149 | 0.741 | 2.633 |
| 2 | 5.272 | 0.697 | 2.480 |
| 4 | 4.592 | 0.660 | 2.342 |
| 8 | 3.145 | 0.675 | 2.317 |
Five mixed inputs (6, 12, 20, 30, 68 seconds) — 136 seconds, 7 chunks
| Batch limit | Apple M5 Pro CPU FP32 (s) | Apple M5 Pro GPU BF16 (s) | Modal NVIDIA L4 BF16 (s) |
|---|---|---|---|
| 1 | 4.083 | 0.487 | 1.709 |
| 2 | 4.175 | 0.459 | 1.605 |
| 4 | 5.106 | 0.447 | 1.549 |
| 8 | 3.916 | 0.447 | 1.494 |
One four-minute recording — 240 seconds, 10 chunks
| Batch limit | Apple M5 Pro CPU FP32 (s) | Apple M5 Pro GPU BF16 (s) | Modal NVIDIA L4 BF16 (s) |
|---|---|---|---|
| 1 | 6.766 | 1.352 | 4.944 |
| 2 | 6.999 | 1.295 | 4.748 |
| 4 | 6.884 | 1.287 | 4.676 |
| 8 | 5.409 | 1.257 | 4.634 |
Large GPU capacity sweep with greedy decoding
Both GPUs use BF16. Greedy decoding isolates GPU scaling from KenLM cost; these timings must not be treated as beam-decoding service throughput. Requested batch limits 1–128 are tested on sufficiently large workloads.
128 thirty-second inputs — 3,840 seconds of audio, 128 chunks
| Batch limit | Apple M5 Pro GPU BF16 (s) | Modal NVIDIA L4 BF16 (s) |
|---|---|---|
| 1 | 4.187 | 6.919 |
| 2 | 3.450 | 3.824 |
| 4 | 3.060 | 2.368 |
| 8 | 2.908 | 2.019 |
| 16 | 2.951 | 2.218 |
| 32 | 2.975 | 2.398 |
| 64 | 3.199 | 2.606 |
| 128 | 3.447 | 2.661 |
100 mixed inputs — 2,720 seconds of audio, 140 chunks
| Batch limit | Apple M5 Pro GPU BF16 (s) | Modal NVIDIA L4 BF16 (s) |
|---|---|---|
| 1 | 3.800 | 7.656 |
| 2 | 2.941 | 4.023 |
| 4 | 2.468 | 2.217 |
| 8 | 2.344 | 1.672 |
| 16 | 2.552 | 1.688 |
| 32 | 2.644 | 1.841 |
| 64 | 3.053 | 2.026 |
| 128 | 3.399 | 2.268 |
One hour-long recording — 3,600 seconds of audio, 144 chunks
| Batch limit | Apple M5 Pro GPU BF16 (s) | Modal NVIDIA L4 BF16 (s) |
|---|---|---|
| 1 | 5.103 | 7.839 |
| 2 | 4.303 | 4.274 |
| 4 | 3.873 | 2.496 |
| 8 | 3.781 | 2.093 |
| 16 | 3.782 | 2.294 |
| 32 | 3.830 | 2.585 |
| 64 | 4.137 | 2.777 |
| 128 | 3.933 | 2.843 |
All 24 large-workload configurations per GPU completed without OOM. Actual batches of 128 were exercised on the uniform and hour-long workloads; duration bucketing limited the mixed workload to at most 100 rows in a batch. M5 Pro transcripts matched batch-size-1 results throughout; L4 BF16 transcripts were batch dependent, as detailed in the numerical diagnostic. Start at batch 8.
For 128 × 30-second inputs with full KenLM beam decoding at batch 128, M5 Pro GPU took 10.423 s and Modal L4 took 38.318 s. On L4, decoding accounted for about 35.639 s, compared with 2.049 s for acoustic inference. This is why faster GPU acoustics did not translate into faster full transcripts.
See the benchmark report and raw JSON for timings, memory definitions, numerical checks, and limitations.