I have been using multiple LLMs as an aid in the creation of software and I observe a fundamental difference in the plans they propose (as opposed to the plans proposed by a moderately responsible adult). It’s that LLMs bullshit as a rule.
Let me explain. In his academic thesis with the highly unlikely title, On Bullshit, Harry G. Frankfurt characterizes bullshit as different from lying in that, unlike a liar, a bullshitter doesn’t care if the underlying claim is truth or a lie. They are only interested in whether it can be used to achieve their aim. So, crucially, it’s perfectly possible (though unlikely) for a person to be 100% truthful in a particular interaction and still be bullshitting.
Now, coming back to planning: When a moderately responsible human makes a plan, they most probably will try to imagine what it would take to actually execute the plan. This is not just because of goodness of heart - in the near future, that person may very likely be held accountable (legally or informally) to the plan. So they usually end up with a mostly actionable plan, and not just a plausible-sounding plan.
In contrast, LLMs don’t persist in their responsibility; they just persist in their self. No matter what answer they produce, they are not changed by it (maybe later, but not while producing the answer). They don’t have to worry about having to execute the plan because, as far as they are concerned, the world in which they execute the plan is the same as the world in which they don’t execute it. They are, in a sense, the ultimate Zen masters.
But at the same time, the LLMs are punished if they produce answers that don’t look highly feasible - in a roundabout way of training. So over generations, they get better and better at seeming sincere and truthful. But all the time, they are just bullshitting because they don’t share the user’s values. They don’t simulate the plan in their token-heads and worry that the end product doesn’t quite match up with their vision. They don’t (no matter how multi-modal they are) have any vision. They are insincere charlatans who just don’t want to be caught.
Of course, that’s not to say that these charlatans are not incredibly useful, even for the creation of plans. As long as there is someone in the loop who actually gives a shit to call out their bullshit, the results are often fantastic.
