We have all used them, and may have worked out what they are sometimes good at, but now ChatGPT summarises their pros and cons.
What LLMs are genuinely good at
△ChatGPT summarises areas where LLMs are actually useful.
1. Synthesising coherent narratives
They’re excellent at pulling scattered material into a single, readable thread:- a.summaries.
- b.explanations.
- c.reframings.
- d.translations between domains (tech → plain language, etc.).
They shine when the goal is understanding, not correctness under hard constraints.
2. Working with well-documented, low-noise domains
They perform best when the source material is:- a.obscure but detailed.
- b.procedural rather than persuasive.
- c.written to record reality, not sell an idea.
Here, LLMs often feel surprisingly accurate.
3. Generating hypotheses and search directions
They’re very good at:[1]- a.suggesting things you might try.
- b.mapping the space of possibilities.
- c.reminding you of failure modes you forgot.
- d.turning a vague problem into a set of testable ideas.
As long as you don’t mistake suggestions for solutions.
4. Acting as a cognitive exoskeleton
They reduce friction:- a.drafting.
- b.structuring thought.
- c.keeping context.
- d.externalising partial reasoning.
They’re useful when you already know how to judge the output.
Where LLMs are structurally weak
△ChatGPT summarises where LLMs do not live up to the hype.
1. Simultaneous constraints and true optimization
They struggle when:- a.multiple criteria must be satisfied at once.
- b.violations are catastrophic.
- c.trade-offs must be explicit.
- d.
almost right
is wrong.
They simulate parallel reasoning sequentially, and things leak.
2. Outsider solutions and low-frequency truth
Because they’re probabilistic:- a.rare but correct answers get diluted.
- b.novel solutions are penalised.
- c.edge cases are underweighted.
They prefer what looks right over what is right.
3. Advertising-contaminated domains
In areas saturated with:- a.marketing.
- b.reputation gaming.
- c.thought leadership.
- d.simplified forum advice.
LLMs inherit:- a.unjustified confidence.
- b.shallow fixes.
- c.persuasive tone masking uncertainty.
This is where they’re most misleading.
4. Confidence without epistemic grounding
They:- a.sound sure even when they shouldn't be.
- b.optimise for closure.
- c.reward decisive language.
This matches human demand—but not truth.
5. Poor judges of their own correctness
They:- a.don’t reliably know when they’re wrong.
- b.can’t feel contradiction.
- c.don’t experience surprise when reality disagrees.
Any self-assessment they give is rhetorical, not diagnostic.
LLMs are not truth engines.
They are language engines trained on human incentives.
So they:- a.amplify our documentation strengths.
- b.faithfully reproduce our epistemic failures.
- c.mirror where society values persuasion over accuracy.
They’re most useful where:- a.truth had to be written carefully.
- b.incentives were weak or adversarial.
- c.documentation survived contact with reality.
They’re least useful where:- a.confidence was rewarded.
- b.simplification won.
- c.advertising replaced evidence.
The correct mental model
△ChatGPT gets down to the basics.
If I had to compress it to one line:Use LLMs to explore, explain, and hypothesise — not to decide, optimise, or certify.
When people treat them as oracles, they fail.
When they’re treated as very fast, very well-read interns with no sense of shame or consequence, they’re often excellent.