MTL-PepPred — multi-task peptide bioactivity predictor (21 tasks)

MTL-PepPred predicts 21 peptide bioactivities from amino acid sequence in a single forward pass, using a frozen ESM-2 (650M) backbone with a learnable base embedding, a parallel Transformer / 1D-CNN feature extractor, and one small classification head per task. 104.7 M of the 755.7 M parameters are trainable (13.9%).

Code, datasets and the scripts that produce every reported number: https://github.com/minhquoc95/MTL-PepPred

These weights accompany a manuscript under revision at Food Bioscience. All performance figures below are the mean ± standard deviation over five independent training runs (seeds 42, 1, 2, 3 and 4), which differ only in random initialisation and data shuffling. They supersede the single-run values published in an earlier version of this model card (89.4% ACC / 94.0% AUC / 78.6% MCC); those came from one run and should not be cited.

Held-out test performance, five runs

Metric	Mean ± SD
Accuracy	87.43 ± 0.23%
AUC	92.84 ± 0.28%
PR-AUC	92.19 ± 0.43%
MCC	73.73 ± 0.46%

Averaged over the 21 tasks. PR-AUC is the average precision throughout this card. Variability is concentrated in the two smallest datasets: quorum sensing (40 test sequences) and blood-brain barrier penetration (38) show MCC standard deviations of 7.1 and 6.8 percentage points. All other results in this card come from the same five runs.

Per-task test performance, five runs

Task	ACC (%)	AUC (%)	PR-AUC (%)	MCC (%)
Signal peptide	99.30 ± 0.12	99.91 ± 0.07	99.86 ± 0.12	98.60 ± 0.25
Anti-MRSA	99.09 ± 0.23	98.70 ± 0.87	97.97 ± 0.70	96.46 ± 0.88
Antimalarial (alternative)	97.92 ± 0.33	99.69 ± 0.18	98.76 ± 0.57	92.57 ± 1.22
Antimalarial (main)	97.36 ± 0.27	91.62 ± 1.65	80.50 ± 1.45	74.98 ± 2.84
Antimicrobial activity	97.03 ± 0.31	98.12 ± 0.27	95.28 ± 1.91	92.83 ± 0.69
Antibacterial activity	94.52 ± 0.30	97.29 ± 0.18	97.24 ± 0.40	89.06 ± 0.61
Antifungal activity	94.37 ± 0.45	98.91 ± 0.22	99.04 ± 0.15	88.78 ± 0.92
Anticancer (alternative)	92.89 ± 0.70	96.81 ± 0.66	96.93 ± 0.97	85.82 ± 1.40
Toxicity	92.48 ± 0.29	96.14 ± 0.47	95.44 ± 0.77	84.92 ± 0.60
Quorum sensing	90.00 ± 3.54	98.30 ± 0.91	98.42 ± 0.82	80.16 ± 7.07
Neuropeptide	88.72 ± 0.51	94.94 ± 0.33	95.22 ± 0.35	77.59 ± 0.86
Bitter taste	87.97 ± 1.31	95.96 ± 0.19	96.23 ± 0.28	76.39 ± 2.84
DPP-IV inhibition	85.86 ± 1.32	94.64 ± 0.55	94.31 ± 1.12	71.95 ± 2.91
Umami taste	85.17 ± 1.46	92.73 ± 0.89	84.88 ± 2.34	65.22 ± 3.05
Antiviral activity	84.00 ± 0.87	91.55 ± 0.46	92.00 ± 0.77	68.10 ± 1.83
Blood-brain barrier penetration	80.53 ± 3.99	92.52 ± 2.93	93.53 ± 2.05	63.75 ± 6.84
ACE inhibition	78.39 ± 1.12	86.00 ± 1.56	85.86 ± 1.75	57.02 ± 2.26
Antioxidant activity	75.09 ± 1.31	82.97 ± 0.53	81.60 ± 0.59	50.83 ± 2.24
Tumour T-cell antigens	74.01 ± 2.31	79.57 ± 1.94	86.03 ± 1.54	43.37 ± 4.91
Anticancer (main)	73.60 ± 1.26	81.49 ± 1.24	82.12 ± 1.85	47.66 ± 2.53
Antiparasitic activity	67.83 ± 3.13	81.79 ± 6.32	84.71 ± 4.02	42.24 ± 6.16

What this model does and does not do

• Multi-task, not multi-label. Each task is a separate binary classifier reading a shared representation. A peptide receives 21 independent probabilities.
• Joint training does not help every task. Against single-task models of identical architecture, 12 of 21 tasks improve by AUC and 9 decline; by accuracy 9 improve, 11 decline and 1 is unchanged. Ten tasks change the sign of the difference between AUC and MCC. The largest loss is quorum sensing (−15.1 points MCC). Per-task differences are in the manuscript's Table S4.
• Anti-MRSA and signal peptide are at ceiling (single-task AUC already above 0.99), so neither should be cited as evidence of positive transfer.
• The signal-peptide task is easier in-benchmark than in the wild. Its negatives are peptides with other validated activities, not confirmed non-secretory sequences. On the independent SignalP-6.0 benchmark the same head reaches 0.768 AUC / 0.303 PR-AUC and 66.4% accuracy at the natural 1:6 class ratio, against 99.3% accuracy in-benchmark. Treat the in-benchmark number as an upper bound.
• Sequence only. No conformation, post-translational modification, gastrointestinal stability or bioavailability. Outputs are for prioritising candidates, not for concluding that a peptide is active.
• Thresholds. All reported values use the default 0.5 threshold. Calibrating the threshold on validation data helps where the operating point is poorly placed (antiparasitic activity: accuracy 69.6% → 76.1%, MCC 45.9% → 56.7% at threshold 0.22) and barely matters where it is not (bitter taste at 0.55: accuracy unchanged, MCC 75.2% → 75.3%).

Extending to a new activity

A 22nd activity can be added by attaching one head and training only that head, with the backbone, the dual-branch extractor and the 21 existing heads frozen. Demonstrated with haemolytic activity (HemoPI2, filtered to 708 train / 162 test sequences after removing sequences already present in the 21-task pool): the new head holds 688,770 parameters, 0.09% of the model, and reaches 80.3% ACC / 86.1% AUC / 60.8% MCC, against 79.6% / 86.9% / 60.2% for a dedicated single-task model needing 90.9 M trainable parameters. Re-running inference on all 21 original test sets through the extended model reproduced the original predictions to within 5 × 10⁻⁵.

Files

File	Contents
shared_backbone.pt	base embedding, shared Transformer, CNN branch, LayerNorm
heads.pt	the 21 task-specific classification heads
ablation_config.json	architecture configuration
task_config.json	task name → head index mapping
test_results.json	held-out test metrics
mtl_peptide_classifier.py	model definition

The frozen ESM-2 encoder is not stored here; it is loaded from facebook/esm2_t33_650M_UR50D at run time.

Usage

The model is a torch.nn.Module, not a transformers model: it has no from_pretrained, and forward takes (input_ids, attention_mask, task_name) and returns the logits of that one task. The weights are split into shared_backbone.pt and heads.pt, so they are loaded into an instance built from task_config.json. mtl_peptide_classifier.py must be importable (it is in this repository and in the GitHub repository).

import json, os
import torch
from huggingface_hub import hf_hub_download
from transformers import EsmTokenizer

from mtl_peptide_classifier import MTLPeptideClassifier

REPO = "minhquoc95/MTL-PepPred"
ckpt = "MTL-PepPred-weights"
os.makedirs(ckpt, exist_ok=True)
for f in ["heads.pt", "shared_backbone.pt", "ablation_config.json", "task_config.json"]:
    hf_hub_download(repo_id=REPO, filename=f, local_dir=ckpt)

device = "cuda" if torch.cuda.is_available() else "cpu"
tokenizer = EsmTokenizer.from_pretrained("facebook/esm2_t33_650M_UR50D")

with open(f"{ckpt}/task_config.json") as fh:
    task_configs = json.load(fh)          # the canonical 21 tasks; no dataset folder needed

model = MTLPeptideClassifier(
    task_configs=task_configs,
    hidden_dim=1280,
    esm_ratio=0.9,
    num_transformer_layers=4,
    dropout=0.3,
    use_transformer=True,
    use_cnn=True,
    unfreeze_esm=False,
)

backbone = torch.load(f"{ckpt}/shared_backbone.pt", map_location=device)
heads    = torch.load(f"{ckpt}/heads.pt", map_location=device)
model.base_embed.load_state_dict(backbone["base_embed"])
if "transformer" in backbone:
    model.transformer.load_state_dict(backbone["transformer"])
if "cnn" in backbone:
    model.cnn.load_state_dict(backbone["cnn"])
    model.layer_norm.load_state_dict(backbone["layer_norm"])
for name, head in model.heads.items():
    if name in heads:
        head.load_state_dict(heads[name])
model = model.to(device).eval()

sequence = "KWKLFKKIEKVGQNIRDGIIKAGPAVAVVGQATQIAK"
inputs = tokenizer(" ".join(sequence), return_tensors="pt",
                   max_length=128, padding="max_length", truncation=True)
input_ids      = inputs["input_ids"].to(device)
attention_mask = inputs["attention_mask"].to(device)

# one task
with torch.no_grad():
    logits = model(input_ids, attention_mask, task_name="Antimicrobial")
    p_active = torch.softmax(logits, dim=-1)[0, 1].item()
print(f"Antimicrobial: {p_active:.3f}")

# all 21 tasks from a single encoder pass is what the architecture is for; the reference
# implementation re-runs forward per head, which is still one backbone pass per call:
with torch.no_grad():
    probs = {
        task: torch.softmax(model(input_ids, attention_mask, task_name=task), dim=-1)[0, 1].item()
        for task in model.heads
    }
for task, p in sorted(probs.items(), key=lambda kv: -kv[1]):
    print(f"{task:20} {p:.3f}")

Task names are the keys of task_config.json: ACE_inhibitory, DPPIV_inhibitory, Bitter, Umami, Antimicrobial, Antimalarial, Antimalarial_alt, Quorum_sensing, Anticancer, Anticancer_alt, AntiMRSA, TTCA, BBP, Anti_parasitic, NeuroPred, Antibacterial, Antifungal, Antiviral, Toxicity, Signal_peptide, Antioxidant.

Sequences are tokenised with a space between residues and padded or truncated to 128 residues (the 90th percentile of the training sequences is 121; about 6% are truncated). Index 1 of the output is the active class.

Training

Frozen ESM-2 (esm2_t33_650M_UR50D, layer-33 hidden states) blended with a learnable embedding at α = 0.9; 4-layer Transformer (d_model 1280, d_ff 5120, 8 heads, dropout 0.3) and a Conv1d branch (kernel 7); masked average pooling; two-hidden-layer MLP heads (2560→256→128→2). Task-uncertainty multi-task loss (Kendall et al., 2018) with cross-entropy and label smoothing ε = 0.1. Task-homogeneous batches with the task drawn uniformly at each step, so every task gets the same number of updates regardless of dataset size. AdamW, lr 1e-4, weight decay 1e-5, gradient clipping 1.0, batch 16, 50 epochs, cosine annealing with 5 warm-up epochs, FP16. Checkpoint selected on the best mean F1 over a 20% validation split (seed 42).

Class imbalance inside a task is not corrected: the published class ratio of every source dataset is preserved so that results stay comparable with the original benchmarks. No focal loss, class re-weighting or synthetic oversampling.

Citation

A citation will be added once the manuscript is published. Until then please cite this repository and the GitHub repository above.

Licence

MIT.