abdurrahman-gulmez/Qwen2.5-Coder-1.5B-Instruct-Deep-Finetuning

Qwen2.5-Coder-1.5B - DEEP - Fine-tune

This model is a LoRA fine-tuned version of Qwen/Qwen2.5-Coder-1.5B-Instruct trained on the Naholav/CodeGen-Deep-5K dataset. It was trained to improve Python code generation capabilities using the Unsloth library.

Model Details

• Base Model: Qwen/Qwen2.5-Coder-1.5B-Instruct
• Dataset: [CodeGen-Deep-5K OR CodeGen-Diverse-5K]
• Training Method: LoRA (Low-Rank Adaptation) via Unsloth
• Objective: Solution-Only Code Generation

Hyperparameters

The following hyperparameters were used during training:

• Learning Rate: 2e-4
• Batch Size: 2 (Effective Batch Size: 16)
• LoRA Rank (r): 32
• LoRA Alpha: 64
• Dropout: 0.05
• Optimizer: AdamW (8-bit)
• LR Scheduler: Cosine
• Max Sequence Length: 1024
• Epochs: 3

Usage

You can use this model with unsloth or peft.

from unsloth import FastLanguageModel

# Load the fine-tuned model
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "YOUR_USERNAME/YOUR_REPO_NAME",
    max_seq_length = 1024,
    dtype = None,
    load_in_4bit = False,
)
FastLanguageModel.for_inference(model)

messages = [
    {"role": "system", "content": "You are an expert Python programmer. Please read the problem carefully before writing any Python code."},
    {"role": "user", "content": "Write a Python function to check if a number is prime."}
]

inputs = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt").to("cuda")

outputs = model.generate(input_ids=inputs, max_new_tokens=1024, temperature=0.1)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))