The Auto-HF exporter class that takes care of automatically instantiating to the correct HfExporter given the ExportConfig.

exporter = AutoHfExporter.from_pretrained("org/model-name")
program = exporter.export(model, inputs)

The Auto-HF export config class that takes care of automatically dispatching to the correct export config given an export config stored in a dictionary.

Abstract base class for all Transformers exporters.

Subclass and implement ~HfExporter.export to add a new export backend.

Exporter that converts a PreTrainedModel to an ExportedProgram.

>>> from transformers.exporters.exporter_dynamo import DynamoExporter, DynamoConfig

>>> exporter = DynamoExporter()
>>> exported = exporter.export(model, inputs, config=DynamoConfig(dynamic=True))
>>> outputs = exported.module()(**inputs)

Exporter that converts a PreTrainedModel to an ONNX ONNXProgram.

>>> from transformers.exporters.exporter_onnx import OnnxExporter, OnnxConfig

>>> exporter = OnnxExporter()
>>> onnx_program = exporter.export(model, inputs, config=OnnxConfig(dynamic=True))
>>> outputs = onnx_program(**inputs)  # run in-memory
>>> exporter.export(model, inputs, config=OnnxConfig(output_path="model.onnx"))  # save to disk

Exporter that converts a PreTrainedModel to an ExecuTorch ExecutorchProgramManager.

>>> from transformers.exporters.exporter_executorch import ExecutorchExporter, ExecutorchConfig

>>> exporter = ExecutorchExporter()
>>> et_program = exporter.export(model, inputs, config=ExecutorchConfig(backend="xnnpack"))
>>> et_program.write_to_file("model.pte")

Configuration class for exporting models via torch.export.

Configuration class for exporting models to ONNX via torch.onnx.export.

Inherits all fields from DynamoConfig (dynamic, strict, dynamic_shapes, prefer_deferred_runtime_asserts_over_guards).

Configuration class for exporting models to ExecuTorch format.

Inherits all fields from DynamoConfig (dynamic, strict, dynamic_shapes, prefer_deferred_runtime_asserts_over_guards).

Configuration class for exporting models to OpenVINO IR via openvino.convert_model.

Inherits all fields from [DynamoConfig] (dynamic, strict, dynamic_shapes, prefer_deferred_runtime_asserts_over_guards).

Base class for post-training quantizers applied during export.

A quantizer declares the export formats it supports and the packages it needs (with the versions it is tested on, like the exporters) and whether it quantizes the torch.export FX graph, before any conversion (so every backend takes the result), or the model a backend converts it into (onnx_ir.Model, openvino.Model). The exporter validates it against the export up front, then calls ~ExportQuantizer.quantize with that model and the export’s sample inputs. Subclass it and implement quantize to add a new quantization method; a CalibrationSet maps the calibration samples to the model’s own inputs.

Re-iterable view of calibration samples as model inputs.

The samples are the quantizer’s calibration_dataset, or the export’s own sample_inputs (with a warning, once they are read) when it has none. They can be any mapping, such as a tokenizer’s BatchEncoding. transform maps each forward-kwarg sample to what the quantized model takes (traced kwargs, ONNX feeds, OpenVINO port feeds), lazily, one sample at a time, on every pass.

Quantize the FX graph with PyTorch 2 Export (PT2E) quantization: prepare_pt2e inserts observers, the calibration set drives them, and convert_pt2e replaces them with quantize/dequantize ops. Works with every export backend.

>>> from torchao.quantization.pt2e.quantizer.x86_inductor_quantizer import (
...     X86InductorQuantizer,
...     get_default_x86_inductor_quantization_config,
... )
>>> from transformers.exporters import DynamoConfig, PT2EQuantizer

>>> x86 = X86InductorQuantizer().set_global(get_default_x86_inductor_quantization_config())
>>> DynamoConfig(quantizer=PT2EQuantizer(x86, calibration_dataset=samples))

Quantize the converted ONNX model with ONNX Runtime’s own tools (onnxruntime.quantization): quantize_static with an OnnxRuntimeStaticConfig (the default), or quantize_dynamic with an OnnxRuntimeDynamicConfig.

>>> from transformers.exporters import (
...     OnnxConfig,
...     OnnxRuntimeDynamicConfig,
...     OnnxRuntimeQuantizer,
...     OnnxRuntimeStaticConfig,
... )

>>> static = OnnxRuntimeStaticConfig(per_channel=True)
>>> OnnxConfig(quantizer=OnnxRuntimeQuantizer(static, calibration_dataset=samples))
>>> OnnxConfig(quantizer=OnnxRuntimeQuantizer(OnnxRuntimeDynamicConfig()))

Arguments of onnxruntime.quantization.quantize_static: int8 activations and weights (QDQ by default), calibrated on the quantizer’s calibration_dataset. A field left to None keeps ONNX Runtime’s default. See quantize_static for each one.

Arguments of onnxruntime.quantization.quantize_dynamic: int8 weights, activations quantized at runtime, no calibration. A field left to None keeps ONNX Runtime’s default. See quantize_dynamic for each one.

Quantize the converted OpenVINO model with NNCF: nncf.quantize with a NNCFQuantizationConfig (the default), or nncf.compress_weights with a NNCFWeightCompressionConfig.

>>> import nncf
>>> from transformers.exporters import NNCFOpenVINOQuantizer, NNCFWeightCompressionConfig, OpenVINOConfig

>>> OpenVINOConfig(quantizer=NNCFOpenVINOQuantizer(calibration_dataset=samples))
>>> OpenVINOConfig(
...     quantizer=NNCFOpenVINOQuantizer(NNCFWeightCompressionConfig(mode=nncf.CompressWeightsMode.INT4_SYM))
... )

Quantize the converted ONNX model with NNCF through its ONNX backend: nncf.quantize with a NNCFQuantizationConfig (the default, int8 QDQ), or nncf.compress_weights with a NNCFWeightCompressionConfig (int4 as ONNX Runtime’s MatMulNBits).

>>> import nncf
>>> from transformers.exporters import NNCFOnnxQuantizer, NNCFWeightCompressionConfig, OnnxConfig

>>> OnnxConfig(quantizer=NNCFOnnxQuantizer(calibration_dataset=samples))
>>> OnnxConfig(quantizer=NNCFOnnxQuantizer(NNCFWeightCompressionConfig(mode=nncf.CompressWeightsMode.INT4_SYM)))

Quantize the torch.export FX graph with NNCF through its TorchFX backend: nncf.quantize with a NNCFQuantizationConfig (the default, quantize/dequantize ops), or nncf.compress_weights with a NNCFWeightCompressionConfig. Like PT2EQuantizer, it runs before any conversion, so every backend takes its result. ExecuTorch lowers only the weight-only result: XNNPACK doesn’t delegate the quantize/dequantize ops, and ExecuTorch has no kernels for them.

>>> from transformers.exporters import DynamoConfig, NNCFTorchFXQuantizer

>>> DynamoConfig(quantizer=NNCFTorchFXQuantizer(calibration_dataset=samples))

Arguments of nncf.quantize: int8 activations and weights, calibrated on the quantizer’s calibration_dataset. A field left to None keeps NNCF’s default, except model_type, which defaults to nncf.ModelType.TRANSFORMER. See nncf.quantize for each one.

Arguments of nncf.compress_weights: weight-only int8 or int4, with data-aware methods (AWQ, scale estimation, GPTQ) for the int4 modes. The int8 modes are data-free; the others calibrate on the quantizer’s calibration_dataset. gptq and lora_correction are OpenVINO-only. A field left to None keeps NNCF’s default. See nncf.compress_weights for each one.

Recursively retrieve all leaf tensors from a potentially nested structure.

Configure model and inputs for export. Mutates both model and inputs in place, returning (model, inputs, output_flags) where output_flags holds the values popped from inputs for use_cache, return_dict, etc. (to be applied reversibly onto model.config by patch_model_config during the trace).

• Strips label inputs (labels, future_values) — loss computation is unsupported.
• Pops output flags (use_cache, return_dict, …) from inputs so they don’t appear as traced kwargs; the values are returned for the trace block to apply onto model.config.
• Pre-computes data-dependent vision/audio kwargs registered via @register_export_input_preparer and writes them into inputs.
• Casts input tensors to match the model’s dtype / device.

Run model.generate() and capture prefill and decode inputs.

Reuses the full generation machinery so every architecture (decoder-only, SSM, encoder-decoder, multi-modal, …) gets correct inputs without reimplementing the loop.

generation_config is forwarded to generate() (defaulting to the model’s own), so the captured inputs use whatever cache generate() would build. Pass one with cache_implementation="static" and max_cache_len=N to capture a statically sized cache in the decode inputs — the basis for a static-cache export. max_cache_len sizes the cache independently of the capture, so the exported decode takes a fixed [..., N, ...] cache rather than a growing one.

When multi_token_decode, the decode component is captured as a multi-token decode — two consecutive decode steps merged (see _merge_decode_calls) so its query-sequence axis stays symbolic (a single-token decode would specialize that axis to 1). It then handles both one token (ordinary decoding) and many (continuation-from-past, or a plain prefill when the cache is empty). Otherwise decode is the classic single-token decode.

Capture inputs to each multi-modal submodule via a single forward pass.

Detects all known multi-modal submodules by attribute name (vision tower, projector, language model, lm_head, …) and captures their forward kwargs during one model(**inputs) call.

Each submodule is returned as a separate name: (module, inputs) entry for independent export. The token-merge step (e.g. masked_scatter for multi-modal models) is intentionally left outside the exported graphs — it is the caller’s responsibility to assemble inputs_embeds from the encoder outputs before running the decoder.

Decompose a generative model into independently exportable (model, forward_inputs) pairs.

Runs decompose_prefill_decode to capture prefill and decode forward kwargs from a real model.generate(**inputs, max_new_tokens=2). If the prefill is multi-modal (per is_multimodal), further splits it into one entry per submodule (vision/audio encoder, projector, language model, lm_head) via decompose_multimodal.

Returns True if the model is multi-modal with modal encoders and a language model.

A non-PreTrainedModel (e.g. a bare nn.Module) has no canonical get_encoder/get_decoder accessors and is trivially not multi-modal, so it short-circuits to False.