Unity ML-Agents | Pretrain an LLM from Scratch with Sentence Transformers | Part 21c
0*Welcome back to our Tau LLM series! ๐*
In this episode, we're diving into our fourth training attempt, known as **Series D**. Here's what we have planned:
**Series D Training Overview**: We'll be conducting a total of **50 million training steps**. The training will be broken down as follows:
**First 10 Million Steps**: Focused on the first column of our output vector. This serves as a warm-up phase.
**Second 10 Million Steps**: Training on the first and second columns, averaged together. The averaging is based on the difference between the expected and actual values.
**Third 10 Million Steps**: Training on the first three columns, averaged together in the same manner.
**Subsequent Steps**: We'll continue this pattern, progressively including more columns and averaging them together.
**Current Progress**: We've successfully completed the first 10 million steps on the first column. Now, we'll proceed with the second run of Series D, focusing on the first and second columns averaged together.
**Training Methodology**: Our approach involves calculating the expected value minus the actual value difference for each column. This method helps us fine-tune the model's accuracy and performance.
Join us as we continue to refine our LLM with these advanced training techniques. Whether you're new to machine learning or an experienced practitioner, this episode offers valuable insights into the intricacies of training large language models.
Stay tuned and let's get started! ๐