Notes

I have been trying models from 3 providers on 2 very different codebases and using 4 different agent/IDE toolchains. For any given task, all of these variables figure into which setup works best.

Anyone who asserts “this is the best model for coding” is oversimplifying, grifting, or has found an optimal configuration that eludes me.

I so tired of exaggerated ink traps.

There is a growing body of research into LLM self-awareness but I am particularly fascinated by the “evaluation awareness” research by Anthropic into the Hawthorne effect in LLMs. (h/t to @futureshape.bsky.social for the connection)

Jakob Nielsen is extremely grumpy about Jony Ive which makes me feel a bit better about my own ranting.

Today in “things my coding model said”:

This is a great example of premature optimization - the debouncing added complexity to solve a performance problem that doesn’t actually exist at the scale we’re operating.

(It wrote the debouncing, until I asked it to simplify.)

As I enjoy watching models explain their reasoning through work I’ve given them, I keep thinking about the recent paper _Chain-of-Thought Is Not Explainability and how much we should remind people that rationalization is not always insight:

verbalised chains are frequently unfaithful, diverging from the true hidden computations that drive a model’s predictions, and giving an incorrect picture of how models arrive at conclusions