StereoMaker/stereospace-v1-0-coreml

StereoSpace v1.0 — Core ML (FP16 / 6-bit)

Core ML conversion of prs-eth/stereospace-v1-0 for Apple silicon Macs. Given a single image, it generates the right-eye view of a stereo pair (depth-free, diffusion-based).

This is an unofficial conversion. It is not affiliated with or endorsed by the StereoSpace authors or Stability AI. Please see the original paper and code.

What was changed

• The PyTorch checkpoints were split into five Core ML programs (below) and converted with coremltools (ML Program, macOS 15+).
• Weights are stored in FP16 (fp16/), or additionally post-training 6-bit palettized (6bit/: k-means, per-grouped-channel, group size 16). The two VAEs are kept FP16 in both sets.
• No retraining or fine-tuning. Numerical outputs differ slightly from the PyTorch model because of FP16 / palettization.
• To make the graphs convertible, a few ops were rewritten without changing the maths: the 5-D nearest upsample and 5-D GroupNorm of the 3D UNet were expressed in 4-D, and the reference-attention "bank" is passed as explicit inputs/outputs (see below).

Files

Model	Role	FP16	6-bit
SS_CLIPImage	CLIP ViT-H/14 vision tower → image embedding	1.2 GB	0.46 GB
SS_VAEEncoder	source image → latent	65 MB	(same FP16)
SS_ReferenceUNet	source view → 16 attention features ("bank")	1.6 GB	0.62 GB
SS_DenoisingUNet	one denoising step of the target view	1.6 GB	0.62 GB
SS_VAEDecoder	latent → target image	95 MB	(same FP16)
Total		≈ 4.6 GB	≈ 1.9 GB

Layout: fp16/*.mlpackage (all five) and 6bit/*.mlpackage (CLIP and both UNets; use the VAEs from fp16/).

Model I/O

Fixed resolution: 768 × 768 image, 96 × 96 latent.

Model	Inputs	Outputs
SS_CLIPImage	pixel_values (1,3,224,224), CLIP-normalized	image_embeds (1,1024)
SS_VAEEncoder	image (1,3,768,768), range [-1, 1]	latent (1,4,96,96), already × 0.18215
SS_ReferenceUNet	ref_latent (1,4,96,96), plucker_src (1,6,96,96), clip_embed (1,1,1024)	bank0 … bank15
SS_DenoisingUNet	latent (1,4,96,96), plucker_tgt (1,6,96,96), t (1,), clip_embed (1,1,1024), bank_bias (1,), bank0 … bank15	noise_pred (1,4,96,96), v-prediction
SS_VAEDecoder	latent (1,4,96,96), scaled	image (1,3,768,768), range [0, 1]

Inference outline

1. Resize the source to 768 × 768 and encode with SS_VAEEncoder; compute the CLIP embedding (224 × 224 input) with SS_CLIPImage.

2. Run SS_ReferenceUNet once (plucker_src, source latent, CLIP embedding) to get bank0…15.

3. Start from Gaussian noise (96 × 96 × 4) and run a DDIM loop (η = 0, v-prediction, zero-terminal-SNR betas, trailing timestep spacing; beta_start=0.00085, beta_end=0.012, scaled-linear, 1000 training steps). The original default is 50 steps, guidance scale 1.5.

4. In each step call SS_DenoisingUNet twice (classifier-free guidance):

   • conditional: clip_embed = CLIP embedding, bank_bias = 0
   • unconditional: clip_embed = zeros, bank_bias = -30000 (disables the reference bank)

   then v = v_uncond + guidance * (v_cond - v_uncond) and take a DDIM step.

5. Decode the final latent with SS_VAEDecoder. The result is the right-eye image; the (resized) source is the left-eye image.

Plücker maps (default camera, 54° FOV, square image) at 96 × 96: for pixel (x, y), ray d = normalize(((x+0.5-48)/f, (y+0.5-48)/f, 1)) with f = 0.5/tan(27°)·96. plucker_src = [d, 0, 0, 0], plucker_tgt = [d, 0, -b·d_z, b·d_y], where b is the baseline (default 0.15).

Performance

Informal measurement on one Apple silicon Mac (Mac16,10), compute units CPU+GPU, FP16: about 1.7 s per denoising step (two UNet calls), i.e. roughly 35 s for 20 steps. 6-bit mainly reduces size and memory; it was not measured to be faster on the GPU. No systematic benchmark was done.

Limitations

• Not numerically validated against PyTorch. Only visual checks were done (10-step runs on a few images). A side-by-side comparison of FP16 and 6-bit showed small differences with no clear winner, but there is no quantitative evaluation.
• Square 768 × 768 input only. Non-square images must be resized (aspect ratio is not preserved by the reference app) and the output resized back.
• Output is a parallel-view pair: infinity has zero disparity. A horizontal shift of the right image can move the zero-parallax plane toward the viewer (cropping the unmatched edges).
• Inherits all limitations of StereoSpace and Stable Diffusion 2 (disocclusions are hallucinated, reflections / transparent surfaces can be unreliable, etc.).
• The CLIP vision tower comes from a LAION-trained model whose authors consider deployed use cases out of scope and recommend research use. Keep this in mind for your use.

License

This repository contains derivatives of Stable Diffusion 2 via StereoSpace and is released under the CreativeML Open RAIL++-M License (creativeml-openrail-m, dated November 24, 2022). The use-based restrictions of that license apply to this model and to any further derivatives. The full license text is included in this repository as LICENSE-MODEL. It was copied from the community mirror sd2-community/stable-diffusion-2, because the original stabilityai/stable-diffusion-2 repository was not accessible when this card was prepared.

Third-party components:

• StereoSpace (original model and code by ETH Zürich et al.): the code is under the MIT License, "Copyright (c) 2025 ETH Zürich". The license text is included as LICENSE-StereoSpace-MIT. Source: https://github.com/prs-eth/stereospace
• CLIP ViT-H/14 (LAION-2B): its model card declares the MIT License (license: mit), see https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K. The upstream repository does not include a license file or name a copyright holder, so none is reproduced here; please refer to the upstream page. The authors recommend research use only.

Citation

Please cite the original work:

@misc{behrens2025stereospace,
  title  = {StereoSpace: Depth-Free Synthesis of Stereo Geometry via End-to-End Diffusion in a Canonical Space},
  author = {Behrens, Tjark and Obukhov, Anton and Ke, Bingxin and Tosi, Fabio and Poggi, Matteo and Schindler, Konrad},
  year   = {2025},
  eprint = {2512.10959},
  archivePrefix = {arXiv}
}