I recently worked on a LoRA that improves tool use in LLM. Thought the approach might interest folks here.
text
I recently worked on a LoRA that improves tool use in LLM. Thought the approach might interest folks here.
new_line
new_line
The issue I have had when trying to use some of the local LLMs with coding agents is this:
text
The issue I have had when trying to use some of the local LLMs with coding agents is this:
new_line
new_line
Me: "Find all API endpoints with authentication in this codebase"
text
Me: "Find all API endpoints with authentication in this codebase"
new_line
LLM: "You should look for 
text
LLM: "You should look for 
@app
mention
app
.route decorators and check if they have auth middleware..."
text
.route decorators and check if they have auth middleware..."
new_line
new_line
But I often want it to search the files and show me but the LLM doesn't trigger a tool use call.
text
But I often want it to search the files and show me but the LLM doesn't trigger a tool use call.
new_line
new_line
To fine-tune it for tool use I combined two data sources:
text
To fine-tune it for tool use I combined two data sources:
new_line
new_line
1. Magpie scenarios - 5000+ diverse tasks (bug hunting, refactoring, security audits)
text
1. Magpie scenarios - 5000+ diverse tasks (bug hunting, refactoring, security audits)
new_line
2. Real execution - Ran these on actual repos (FastAPI, Django, React) to get authentic tool responses
text
2. Real execution - Ran these on actual repos (FastAPI, Django, React) to get authentic tool responses
new_line
new_line
This ensures the model learns both breadth (many scenarios) and depth (real tool behavior).
text
This ensures the model learns both breadth (many scenarios) and depth (real tool behavior).
new_line
new_line
Tools We Taught:
text
Tools We Taught:
new_line
- 
text
- 
read_file
`read_file`
inline_code
 - Actually read file contents
text
 - Actually read file contents
new_line
- 
text
- 
search_files
`search_files`
inline_code
 - Regex/pattern search across codebases
text
 - Regex/pattern search across codebases
new_line
- 
text
- 
find_definition
`find_definition`
inline_code
 - Locate classes/functions
text
 - Locate classes/functions
new_line
- 
text
- 
analyze_imports
`analyze_imports`
inline_code
 - Dependency tracking
text
 - Dependency tracking
new_line
- 
text
- 
list_directory
`list_directory`
inline_code
 - Explore structure
text
 - Explore structure
new_line
- 
text
- 
run_tests
`run_tests`
inline_code
 - Execute test suites
text
 - Execute test suites
new_line
new_line
Improvements:
text
Improvements:
new_line
- Tool calling accuracy: 12% → 80% 
text
- Tool calling accuracy: 12% → 80% 
new_line
- Correct parameters: 8% → 87%
text
- Correct parameters: 8% → 87%
new_line
- Multi-step tasks: 3% → 78%
text
- Multi-step tasks: 3% → 78%
new_line
- End-to-end completion: 5% → 80%
text
- End-to-end completion: 5% → 80%
new_line
- Tools per task: 0.2 → 3.8
text
- Tools per task: 0.2 → 3.8
new_line
new_line
The LoRA really improves on intential tool call as an example consider the query: "Find ValueError in payment module"
text
The LoRA really improves on intential tool call as an example consider the query: "Find ValueError in payment module"
new_line
new_line
The response proceeds as follows:
text
The response proceeds as follows:
new_line
new_line
1. Calls 
text
1. Calls 
search_files
`search_files`
inline_code
 with pattern "ValueError"
text
 with pattern "ValueError"
new_line
2. Gets 4 matches across 3 files
text
2. Gets 4 matches across 3 files
new_line
3. Calls 
text
3. Calls 
read_file
`read_file`
inline_code
 on each match
text
 on each match
new_line
4. Analyzes context
text
4. Analyzes context
new_line
5. Reports: "Found 3 ValueError instances: payment/processor.py:47 for invalid amount, payment/validator.py:23 for unsupported currency..."
text
5. Reports: "Found 3 ValueError instances: payment/processor.py:47 for invalid amount, payment/validator.py:23 for unsupported currency..."
new_line
new_line
Resources:
text
Resources:
new_line
- Colab notebook 
text
- Colab notebook 
https://colab.research.google.com/github/codelion/ellora/blob/main/Ellora_Recipe_3_Enhanced_Tool_Calling_and_Code_Understanding.ipynb
https://colab.research.google.com/github/codelion/ellora/blob/main/Ellora_Recipe_3_Enhanced_Tool_Calling_and_Code_Understanding.ipynb
link
new_line
- Model - 
text
- Model - 
https://huggingface.co/codelion/Llama-3.2-1B-Instruct-tool-calling-lora
codelion/Llama-3.2-1B-Instruct-tool-calling-lora
model
resource
https://huggingface.co/codelion/Llama-3.2-1B-Instruct-tool-calling-lora
new_line
- GitHub - 
text
- GitHub - 
https://github.com/codelion/ellora
https://github.com/codelion/ellora
link
new_line
I recently worked on a LoRA that improves tool use in LLM. Thought the approach might interest folks here.

The issue I have had when trying to use some of the local LLMs with coding agents is this:

Me: "Find all API endpoints with authentication in this codebase"
LLM: "You should look for @app.route decorators and check if they have auth middleware..."

But I often want it to search the files and show me but the LLM doesn't trigger a tool use call.

To fine-tune it for tool use I combined two data sources:

1. Magpie scenarios - 5000+ diverse tasks (bug hunting, refactoring, security audits)
2. Real execution - Ran these on actual repos (FastAPI, Django, React) to get authentic tool responses

This ensures the model learns both breadth (many scenarios) and depth (real tool behavior).

Tools We Taught:
- `read_file` - Actually read file contents
- `search_files` - Regex/pattern search across codebases
- `find_definition` - Locate classes/functions
- `analyze_imports` - Dependency tracking
- `list_directory` - Explore structure
- `run_tests` - Execute test suites

Improvements:
- Tool calling accuracy: 12% → 80% 
- Correct parameters: 8% → 87%
- Multi-step tasks: 3% → 78%
- End-to-end completion: 5% → 80%
- Tools per task: 0.2 → 3.8

The LoRA really improves on intential tool call as an example consider the query: "Find ValueError in payment module"

The response proceeds as follows:

1. Calls `search_files` with pattern "ValueError"
2. Gets 4 matches across 3 files
3. Calls `read_file` on each match
4. Analyzes context
5. Reports: "Found 3 ValueError instances: payment/processor.py:47 for invalid amount, payment/validator.py:23 for unsupported currency..."

Resources:
- Colab notebook https://colab.research.google.com/github/codelion/ellora/blob/main/Ellora_Recipe_3_Enhanced_Tool_Calling_and_Code_Understanding.ipynb
- Model - https://huggingface.co/codelion/Llama-3.2-1B-Instruct-tool-calling-lora
- GitHub - https://github.com/codelion/ellora

🔥
codelion ssshadow-the-hedgehog RakshitAralimatti saattrupdan drwlf ThijsL202 tin5451 Fataj milkomeda22 dododo1234 victor AlexanderKyng robaray nightmedia sequelbox FM-1976 ronedgecomb John6666 jtvino richinseattle sdama
👍
risedangel EccoFriendly infinfin codelion Fataj Khawn2u drlee1
❤️
codelion nkpz franzipol bukit
🚀
codelion partzel Fataj
👀
codelion win10 Fataj