I've gradually been relying more on LLMs to edit code with their increasing adeptness at tackling the vast majority of problems. More than ever it is now only a matter of describing the task well and a very adequate and diligent solution can be had.

I am very picky about quality and efficiency with software, so I am not able to just blindly accept output without reviewing it carefully depending on how critical the software's behavior is going to be.

One of the things that I have found however is that working on markdown documents and iterating on them with a chat is significantly more powerful than just iterating in a chat session.

There are a number of significant benefits to doing something like tracking the current objective and the current plans for attacking it. I also like to have a separate markdown file that I instruct the tool to keep a record of recent changes that we are making.

This gives us the ability to terminate or restart the chat session freely and pick right back up because you have a record of what you're currently trying to do. Often we go on little rabbit holes in the chat and it tends to make it less clear to the LLM what needs to be focused on. It's a very common thing to want to nuke the chat history and start fresh. It also means you are spending fewer tokens sending recent chat history as well.

I find the work tracking helps keep "going around in circles" syndrome in check. It does tend to happen a lot when debugging issues, though I admit debugging workflows still need work.

I tried VS Code's Copilot Agent mode recently and I wasn't impressed because it typically works just like Edit mode but is simply more eager to make changes. There are often good changes but it all too often just keeps going and going and often has terrible common sense on when to stop.

I find that it works really well to collaborate with LLMs on building up a document describing a game plan for going forward with a feature or refactor. It's also pretty easy to bring relevant code context into an web-based chatbot, so we can enjoy the flexibility of running our brainstorming through a lot of different tools first. It sounds a bit ridiculous at first, but using LLMs iteratively to build and refine a planning document (which will again be used as a prompt for LLMs!) works incredibly well, and addresses a large pain point with crafting prompts, which is how time consuming it can really be. You might be surprised, you can prompt for a prompt, and refine it rapidly, and end up with a much more valuable prompt while spending much less time on it. This translates directly into better productivity.

I put these instructions into OBJECTIVES.md, but honestly I don't think it matters how you name these things, they can even be put directly into comments in your code.

The benefit of this is that you can put down a lot of detail about a chunk of work you're setting out to do, and any fine tweaks to it can be easily done rather than repeated instructions given ad-hoc in chat. It's easier to roll back and tune the in-file prompt and try again. It can also be satisfying to see the LLM issue edits to cross out parts of your implementation plan that have been completed so far.

I see the enemy in software development as some sort of combination of Murphy's law and the inherent fractal complexity of nature. When we build something from logical building blocks, the engineering considerations are around how we can efficiently assemble them in a way to withstand, in an adequate enough way, the elements: The nondeterminism from the real world that makes its way into the timing of events, the format of inputs, the ambiguities in requirements, and so on. So it is a lot like building a house and designing it to withstand the elements.

Another benefit is that these instructions can be checked into source control and you and your colleagues can browse the changes made to them to follow what will usually be a very digestible and detailed account of the work being done.

For the time being the AI tools are still missing the good sense to work on their own for extended periods of time, but I have definitely found that a workflow involving documenting your intentions and iterating on prose to plan the work prior to embarking on it are a fundamental part of being highly productive with this technology.