Large Language Models and Knowledge Graphs: Merging Flexibility and Structure
178We discuss how to infuse Large Language Models (LLMs) with Knowledge Graphs (KGs)! This is a very exciting approach, as we can combine the flexibility and generalisability of LLMs with the structure and reliability of KGs, and is a first step towards neurosymbolic architectures!
I will also be going through a LangChain implementation of LLMs with knowledge graphs as inputs, demonstrate some of the limitations currently faced, and show how we can better prompt engineer KG usage with LLMs using my very own StrictJSON Framework.
~~~~~~~~~~~~~
Slides: https://github.com/tanchongmin/TensorFlow-Implementations/blob/main/Paper_Reviews/LLM%20with%20Knowledge%20Graphs.pdf
Jupyter Notebook: https://github.com/tanchongmin/TensorFlow-Implementations/blob/main/Tutorial/LLM%20with%20Knowledge%20Graphs.ipynb
StrictJSON Framework: https://github.com/tanchongmin/strictjson
LangChain Documentation: https://python.langchain.com/docs/get_started/introduction.html
Tutorial on how to use LangChain and StrictJSON Framework for Knowledge Graphs and LLMs: https://www.youtube.com/watch?v=h2RitC7DUvo
Paper on Unifying LLMs and Knowledge Graphs: https://arxiv.org/abs/2306.08302
LLMs as Graph Neural Networks / Embeddings
ERNIE: https://arxiv.org/abs/1905.07129
TransE embeddings used by ERNIE: https://proceedings.neurips.cc/paper/2013/file/1cecc7a77928ca8133fa24680a88d2f9-Paper.pdf
QA-GNN: https://arxiv.org/abs/2104.06378
FactKB: https://arxiv.org/abs/2305.08281
~~~~~~~~~~~~~
0:00 Introduction
1:55 Pros and Cons of LLMs and Knowledge Graphs (KGs)
4:55 Retrieval Augmented Generation (RAG)
8:10 Problems with LLMs and RAG
17:40 Basics of KG
26:09 Hierarchy in KG
31:13 KGs can be structurally parsed
33:17 KG can represent environmental transitions
33:58 KG as tool/memory for LLM
39:16 3 approaches to integrate KG and LLMs
40:21 Approach 1: KG-augmented LLMs
59:05 Approach 2: LLM-augmented KG
1:05:37 Approach 3: LLMs and KG two-way interaction
1:10:16 LangChain Graph QA Example
1:16:35 Strict JSON Framework Graph QA Example
1:23:00 Discussion
~~~~~~~~~~~~~
AI and ML enthusiast. Likes to think about the essences behind breakthroughs of AI and explain it in a simple and relatable way. Also, I am an avid game creator.
Discord: https://discord.gg/bzp87AHJy5
LinkedIn: https://www.linkedin.com/in/chong-min-tan-94652288/
Online AI blog: https://delvingintotech.wordpress.com/
Twitter: https://twitter.com/johntanchongmin
Try out my games here: https://simmer.io/@chongmin