mlboydaisuke/qwen3-vl-8b-stateful-coreml

Use it from Swift

Add the package

Package.swift:

.package(url: "https://github.com/john-rocky/CoreML-LLM", branch: "main"),

// In your target:
.product(name: "CoreMLLLM", package: "CoreML-LLM"),

Platforms: iOS 18+ / macOS 15+.

Download + chat (one call)

import CoreMLLLM

let llm = try await CoreMLLLM.load(repo: "mlboydaisuke/qwen3-vl-8b-stateful-coreml")

let stream = try await llm.generate(
    [CoreMLLLM.Message(role: .user, content: "Hello!")],
    maxTokens: 256
)
for await chunk in stream { print(chunk, terminator: "") }

With an image

import CoreGraphics

let cgImage: CGImage = ...   // your CGImage

let stream = try await llm.generate(
    [CoreMLLLM.Message(role: .user,
                       content: "What's in this image?")],
    image: cgImage,
    maxTokens: 256
)
for await chunk in stream { print(chunk, terminator: "") }

Qwen3-VL 8B — Core ML stateful

Core ML port of Qwen/Qwen3-VL-8B-Instruct for iPhone / iPad / Mac Apple Neural Engine. Text + vision, 6.19 GB on disk.

The KV cache lives inside the ANE via MLState, so it does not spill to GPU memory as the context grows. Same layout as qwen3-vl-2b-stateful-coreml, scaled to 8B: 6 body chunks instead of 4.

Files

qwen3_vl_8b_stateful_chunks/
├── chunk_0.mlpackage … chunk_5.mlpackage   ← body, multifunction: infer (T=1) + prefill_b8 (T=8)
├── chunk_0_vision.mlpackage                ← chunk_0 + DeepStack injection
├── chunk_head.mlpackage                    ← final_norm + lm_head + in-graph argmax
└── embed_weight.bin                        ← raw fp16 embed table, Swift mmaps it

qwen3_vl_8b_vision/
└── vision.mlpackage                        ← image encoder + DeepStack taps

Component	Size
6 body chunks + vision chunk	4.05 GB
head	311 MB
embed table	1.25 GB
vision	576 MB
total	6.19 GB

Each body chunk carries two Core ML functions sharing one state. Swift creates the MLState once from the prefill instance and reuses it across both — Core ML binds state by name and shape, not by MLModel instance.

What this repo does NOT ship

• No model_config.json — Core ML packs shapes into each .mlpackage, and coremltools reads them directly.
• No tokenizer / processor — pull them from the upstream model:

from transformers import AutoTokenizer, AutoProcessor
tok  = AutoTokenizer.from_pretrained("Qwen/Qwen3-VL-8B-Instruct")
proc = AutoProcessor.from_pretrained("Qwen/Qwen3-VL-8B-Instruct")

Standalone usage (Python / Mac)

import coremltools as ct, numpy as np
from huggingface_hub import snapshot_download

local = snapshot_download("mlboydaisuke/qwen3-vl-8b-stateful-coreml")
root  = f"{local}/qwen3_vl_8b_stateful_chunks"

prefill_chunks = [ct.models.MLModel(
    f"{root}/chunk_{i}.mlpackage", function_name="prefill_b8"
) for i in range(6)]
decode_chunks  = [ct.models.MLModel(
    f"{root}/chunk_{i}.mlpackage", function_name="infer"
) for i in range(6)]
head = ct.models.MLModel(f"{root}/chunk_head.mlpackage")

# State is created once and shared across infer + prefill_b8.
state = prefill_chunks[0].make_state()

Conversion

github.com/john-rocky/CoreML-LLM

License

Apache 2.0 (inherits from the base model).

---

More models in this format: Core ML Model Zoo — 46 models, each with the recipe that produced it.

Want a different model on-device? Open a request — free, open weights only; the export and its measured numbers get published publicly.