Latest post
Prompting Matters: What 18 Months of LLM Code Generation Taught Me
A long, opinionated tour through the prompting strategies that actually move the needle when you use AI agents to reason about class-level code, and the ones that look smart on paper but quietly fail.
The practical question was simple: which prompting strategies actually improve the generated code, and which ones just make the prompt look more sophisticated?
When we started measuring prompt strategies for class-level code generation, the assumption was that more context wins. It does, until it doesn't.
The interesting failures showed up in the middle of the distribution. Models generated plausible methods that compiled, passed two of three tests, and quietly hallucinated a dependency. The prompt that "looked best", a long preamble of style rules, examples, and warnings, was reliably the worst.
The study
The paper looked at class-level code generation, which is a messier problem than asking a model for one isolated method. A class has state, dependencies, naming conventions, constructor behavior, and interactions between methods. That makes it a better proxy for the kind of code developers actually ask AI tools to write.
We compared zero-shot, few-shot, chain-of-thought, and chain-of-thought few-shot prompting across GPT and Llama models. Then we evaluated the generated code with functional correctness, BLEU-3, ROUGE-L, readability, and maintainability metrics.
What moved the needle
- Naming the contract before the body. State the inputs, outputs, and invariants in the prompt body. Let the model write code against that contract.
- Showing one tight example, never three loose ones.
- Asking for the test first, then the implementation, in two passes.
- Giving enough context to remove ambiguity without turning the prompt into a policy document.
The useful prompts were not the most theatrical ones. They were specific, bounded, and boring in the best possible way.
What surprised me
Context-rich prompting improved functional correctness by up to 25%. BLEU-3 improved by 31%, and ROUGE-L improved by 50%. Those numbers matter, but the error patterns were more useful to me than the scoreboard.
Procedural logic and control-flow errors accounted for 31% of the observed failures. The model often understood the surface shape of the class but missed the sequence of decisions that made the implementation correct.
The lesson I keep using
Prompting is not a magic phrase problem. It is specification work. The better prompt is usually the one that makes the task less ambiguous for both the model and the developer reading the output afterward.
I'll publish the dataset and the per-strategy breakdown alongside the paper.