Running 13B and 30B LLMs at Home with KoboldCPP, AutoGPTQ, LLaMA.CPP/GGML
A look at the current state of running large language models at home. Running 13B and 30B models on a PC with a 12gb NVIDIA RTX 3060.
KoboldCPP:
https://github.com/LostRuins/koboldcpp/releases
KoboldAI:
https://github.com/KoboldAI/KoboldAI-Client
Oobabooga UI:
https://github.com/oobabooga/text-generation-webui
Link to TheBloke's Huggingface page:
https://huggingface.co/TheBloke
Reinstall llama-cpp-python with CUBLAS/CUDA support:
pip uninstall -y llama-cpp-python
CMAKE_ARGS="-DLLAMA_CUBLAS=on" FORCE_CMAKE=1 pip install llama-cpp-python --no-cache-dir
Other Videos By NanoNomad
2023-08-06 | AI Null Reads Alice's Adventures in Wonderland by Lewis Carroll, Chapters 1 and 2 | joshcore |
2023-07-30 | Are Text Cleaners Making Your TTS Models Sound Bad? | TTS Model Training Tips |
2023-07-08 | .:Demo:. Tortoise TTS Expressive Speech narrating Norman Arkawy's 1955 Sci-Fi short "Selling Point" |
2023-07-03 | .::Demo::. 4 Voice Multispeaker Tortoise TTS English Fine-Tuned Model Test :: Great Dictator Speech |
2023-07-01 | Creepy Message about a 2003 Pandemic in China on found IBM PS/1 Pentium 66mhz PC |
2023-06-28 | Now for Download: YourTTS (English, French, German, Spanish) Multilingual Model with 60+ Voices |
2023-06-27 | Demo: YourTTS speaking in native French; A sampling of trained-in Voices |
2023-06-27 | Demo: YourTTS speaking in native Spanish; A sampling of trained-in Voices |
2023-06-27 | Demo: YourTTS speaking in native German; A sampling of trained-in Voices |
2023-06-27 | Demo: YourTTS speaking Norman Arkawy's 1955 Sci-Fi Story 'Selling Point'. Info in description. |
2023-06-14 | Running 13B and 30B LLMs at Home with KoboldCPP, AutoGPTQ, LLaMA.CPP/GGML |
2023-06-08 | Demo and Download: YourTTS Multi-accent, English/Spanish Multi-Voice Model 600k Checkpoint |
2023-06-05 | DEMO: YourTTS - One Voice, Many Accents. A single speaker can generate multiple accents. |
2023-06-04 | Revisiting YourTTS - Details about Training, Datasets, and experiences Voice Cloning with Coqui TTS |
2023-06-03 | DEMO: YourTTS Multi-speaker VCTK Irish-accented Dataset after 275k Steps trained using Coqui TTS |
2023-05-22 | Tortoise TTS Fine Tuning Wrap-Up |
2023-05-16 | Tortoise TTS DEMO: G-Man performs Gilbert and Sullivan's 'The Major-General's Song' |
2023-05-15 | Train Tortoise TTS in English, Spanish, French, Italian, Portuguese, German, and more? Maybe? |
2023-05-10 | DEMO: Testing French-Speaking Tortoise TTS |
2023-05-10 | DEMO: Testing German-Speaking Tortoise TTS |
2023-05-08 | DEMO: Testing Spanish Speaking Tortoise TTS |
Tags: koboldai
oobabooga
llm
chatgpt
ai text