Can language models replace developers? #RepoCod says “Not Yet”, because GPT-4o and other LLMs have <30% accuracy/pass@1 on real-world code generation tasks. 
text
Can language models replace developers? #RepoCod says “Not Yet”, because GPT-4o and other LLMs have <30% accuracy/pass@1 on real-world code generation tasks. 
new_line
- Leaderboard 
text
- Leaderboard 
https://lt-asset.github.io/REPOCOD/
https://lt-asset.github.io/REPOCOD/
link
new_line
- Dataset: 
text
- Dataset: 
https://huggingface.co/datasets/lt-asset/REPOCOD
lt-asset/REPOCOD
dataset
resource
https://huggingface.co/datasets/lt-asset/REPOCOD
new_line
@jiang719
mention
jiang719
text
@shanchao
mention
shanchao
text
@Yiran-Hu1007
mention
Yiran-Hu1007
text
new_line
Compared to #SWEBench, RepoCod tasks are
text
Compared to #SWEBench, RepoCod tasks are
new_line
- General code generation tasks, while SWE-Bench tasks resolve pull requests from GitHub issues.
text
- General code generation tasks, while SWE-Bench tasks resolve pull requests from GitHub issues.
new_line
- With 2.6X more tests per task (313.5 compared to SWE-Bench’s 120.8).
text
- With 2.6X more tests per task (313.5 compared to SWE-Bench’s 120.8).
new_line
new_line
Compared to #HumanEval, #MBPP, #CoderEval, and #ClassEval, RepoCod has 980 instances from 11 Python projects, with
text
Compared to #HumanEval, #MBPP, #CoderEval, and #ClassEval, RepoCod has 980 instances from 11 Python projects, with
new_line
- Whole function generation
text
- Whole function generation
new_line
- Repository-level context
text
- Repository-level context
new_line
- Validation with test cases, and 
text
- Validation with test cases, and 
new_line
- Real-world complex tasks: longest average canonical solution length (331.6 tokens) and the highest average cyclomatic complexity (9.00)
text
- Real-world complex tasks: longest average canonical solution length (331.6 tokens) and the highest average cyclomatic complexity (9.00)
new_line
new_line
Introducing hashtag #RepoCod-Lite 🐟 for faster evaluations: 200 of the toughest tasks from RepoCod with:
text
Introducing hashtag #RepoCod-Lite 🐟 for faster evaluations: 200 of the toughest tasks from RepoCod with:
new_line
- 67 repository-level, 67 file-level, and 66 self-contains tasks
text
- 67 repository-level, 67 file-level, and 66 self-contains tasks
new_line
- Detailed problem descriptions (967 tokens) and long canonical solutions (918 tokens)
text
- Detailed problem descriptions (967 tokens) and long canonical solutions (918 tokens)
new_line
- GPT-4o and other LLMs have < 10% accuracy/pass@1 on RepoCod-Lite tasks. 
text
- GPT-4o and other LLMs have < 10% accuracy/pass@1 on RepoCod-Lite tasks. 
new_line
- Dataset: 
text
- Dataset: 
https://huggingface.co/datasets/lt-asset/REPOCOD_Lite
lt-asset/REPOCOD_Lite
dataset
resource
https://huggingface.co/datasets/lt-asset/REPOCOD_Lite
new_line
new_line
#LLM4code #LLM #CodeGeneration #Security
text
#LLM4code #LLM #CodeGeneration #Security
Can language models replace developers? #RepoCod says “Not Yet”, because GPT-4o and other LLMs have <30% accuracy/pass@1 on real-world code generation tasks. 
- Leaderboard https://lt-asset.github.io/REPOCOD/
- Dataset: https://huggingface.co/datasets/lt-asset/REPOCOD
@jiang719 @shanchao @Yiran-Hu1007 
Compared to #SWEBench, RepoCod tasks are
- General code generation tasks, while SWE-Bench tasks resolve pull requests from GitHub issues.
- With 2.6X more tests per task (313.5 compared to SWE-Bench’s 120.8).

Compared to #HumanEval, #MBPP, #CoderEval, and #ClassEval, RepoCod has 980 instances from 11 Python projects, with
- Whole function generation
- Repository-level context
- Validation with test cases, and 
- Real-world complex tasks: longest average canonical solution length (331.6 tokens) and the highest average cyclomatic complexity (9.00)

Introducing hashtag #RepoCod-Lite 🐟 for faster evaluations: 200 of the toughest tasks from RepoCod with:
- 67 repository-level, 67 file-level, and 66 self-contains tasks
- Detailed problem descriptions (967 tokens) and long canonical solutions (918 tokens)
- GPT-4o and other LLMs have < 10% accuracy/pass@1 on RepoCod-Lite tasks. 
- Dataset: https://huggingface.co/datasets/lt-asset/REPOCOD_Lite

#LLM4code #LLM #CodeGeneration #Security
🔥
John6666 AdinaY clem lin-tan shanchao bhadresh-savani iky1e AtAndDev
🤗
prithivMLmods lin-tan shanchao mvelic AtAndDev ai-everyday
649a54b896d5747b35e2163b
https://cdn-avatars.huggingface.co/v1/production/uploads/649a54b896d5747b35e2163b/tdZmsov6fN1VHztaE5kX9.jpeg
Vezora
Vezora
user
65831342f9c5cda913df366a
https://cdn-avatars.huggingface.co/v1/production/uploads/65831342f9c5cda913df366a/h7MLYt--shRYQj4-q5XmR.jpeg
Lin Tan
lin-tan
user