Duration: 0:00

619 views

11

It is recklessly dangerous to trust an LLM to help inform decisions in nearly ANY context - even a simple chatbot - without verifiable proof of the system prompt(s) and everything else that might be "in context", including fine-tuning guidelines/prompts, training techniques, and pre-training dataset content and usage. Even then, that's only the start of the problems. 🧾SOURCES:

1. hhttps://arxiv.org/abs/2311.07590- Large Language Models can Strategically Deceive their Users when Put Under Pressure

2. hhttps://www.sciencedirect.com/science/article/pii/S0267364922000292- The flaws of policies requiring human oversight of government algorithms

3. The stuff about "we need a new model of how oversight works" is mostly made up by Claude*. It SOUNDS nice, but that's not the same as checking whether it actually works. Remember, LLMs can and do literally lie- they complete the patterns their training and prompting leads them to. It feels likely, for one example, that Gemini and Claude have system prompting along the lines of "stay positive about the benefits of AI", for example, which would "color" everything they generate. Or, maybe there's as explicit as that, but they do AVOID including directions like "always warn people about the risks of AI", because they worry maybe that will make their product sell less well. The point is, we need to know all this stuff, AND more, for safe use.

---

Script and audio generated by NotebookLM: hhttps://notebooklm.google.com/Based on this essay generated by Claude: hhttps://pastebin.com/XAUMnBMV(I proofread it compared to the papers checking for any obvious mistakes, but I am not an AI researcher)
Visualized by MAZTR: hhttps://www.maztr.com/audiofilevisualizer
*Well, "made up" doesn't really apply to what LLMs do; it's not coming from nowhere, it comes a little bit from the papers, a little bit from how I prompted Claude to write the essay, and a little bit from what Claude is trained on; so in a way "we" made it up.