Normie Tools for Validating LLM Outputs
12Speaker: Benjamin Labaschin
Summary
=======
The speaker discusses the importance of validating outputs from large language models and the challenges that come with it. They suggest strategies such as monitoring embeddings drift, conducting A/B testing, and performing human evaluation. The speaker demonstrates their approach using a chatbot powered by Llama 2 and explains the code structure and validation process. They also discuss the use of Pydantic base models and while loops to ensure expected responses. The speaker mentions the tool Pydantic for refining model responses for catching formatting problems and the use of DSLs and JSON schemas for handling complex content.
Topics
======
⃝ Validation Strategies
* Monitoring embeddings drift
* Conducting A/B testing
* Performing human evaluation
⃝ API Message Validation
* Using Pydantic base models
* Checking for expected messages
* Using while loops for validation
* Refining model responses
* Setting criteria for response generation
⃝ Handling Complex Content
* Using domain-specific language (DSL)
* Utilizing JSON schemas
* Specifying desired output structure