Using Logic Gates as Neurons - Deep Differentiable Logic Gate Networks!

Channel:
Singapore John Tan Chong Min
Subscribers:
5,330
Published on ● Video Link: https://www.youtube.com/watch?v=VulM891DhI4


Category:
Vlog
Duration: 1:34:53
13,291 views
107

What if we could do away with all the complexities of a neuron and just model neural networks with logic gates? Fundamentally, logic gates are not differentiable, but with some modifications, we can make it differentiable. We can also make the network learn which logic gate to use using a differentiable categorical distribution. This interesting paper at NeurIPS 2022 shows that using logic gates, we can get much faster inference times and similar accuracy to that of neural networks. Scaling it up, though, is an issue, and we discuss some ways which can potentially help to scale in the next phase of improvements.


Some references:
Paper: https://arxiv.org/pdf/2210.08277
DiffLogic Code Implementation: https://github.com/Felix-Petersen/difflogic/tree/main/difflogic
Slides: https://github.com/tanchongmin/TensorFlow-Implementations/tree/main/Paper_Reviews
De Morgan's Laws: https://en.wikipedia.org/wiki/De_Morgan%27s_laws
Universal Logic Gates: https://www.electronics-tutorials.ws/logic/universal-gates.html#:~:text=Universal%20Logic%20Gates%20using%20only,it%20a%20universal%20logic%20gate.
Gated Linear Units (GLU): https://arxiv.org/abs/1908.07442
https://medium.com/deeplearningmadeeasy/glu-gated-linear-unit-21e71cd52081

~~~~~~~~~~~~~~~~~~~~~~~~~~~

0:00 Introduction
1:48 Perceptron and Logic Gates
16:08 Differences between Perceptron and Logic Gates
20:10 What Logic Gates to model?
23:26 Logic Gates Network Overall Architecture
36:02 Difficulty in training Logic Gates
37:17 Relaxation 1: Real-valued Logics
38:33 Relaxation 2: Distribution over the choice of parameter
43:55 Training Setup
45:05 Configuring output for classification
49:21 Results
59:04 Exponential Growth of Gates
59:44 Limitations
1:01:43 My thoughts on how to model biological neurons
1:08:40 Discussion

AI and ML enthusiast. Likes to think about the essences behind breakthroughs of AI and explain it in a simple and relatable way. Also, I am an avid game creator.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Discord: https://discord.gg/fXCZCPYs
Online AI blog: https://delvingintotech.wordpress.com/.
LinkedIn: https://www.linkedin.com/in/chong-min-tan-94652288/
Twitch: https://www.twitch.tv/johncm99
Twitter: https://twitter.com/johntanchongmin
Try out my games here: https://simmer.io/@chongmin



Other Videos By John Tan Chong Min


2023-04-11OpenAI Vector Embeddings - Talk to any book or document; Retrieval-Augmented Generation!
2023-04-11Tutorial #2: OpenAI Vector Embeddings and Pinecone for Retrieval-Augmented Generation
2023-04-04Creating JARVIS: ChatGPT + APIs - HuggingGPT, Memory-Augmented Context, Meta GPT structures
2023-04-02Is GPT4 capable of self-improving? Are we heading for AGI or AI doom?
2023-03-28How Visual ChatGPT works + Toolformer/Wolfram Alpha. LLMs with Tools/APIs/Plugins is the way ahead!
2023-03-21Tokenize any input, even continuous vectors! - Residual Vector Quantization - VALL-E (Part 2)
2023-03-07Using Transformers to mimic anyone's voice! - VALL-E (Part 1)
2023-02-28Learning Part-Whole Structure by Chunking - More Efficient than Deep Learning!!!
2023-02-21High-level planning with large language models - SayCan
2023-02-13Learning, Fast and Slow: Towards Fast and Adaptable Agents in Changing Environments
2023-02-07Using Logic Gates as Neurons - Deep Differentiable Logic Gate Networks!
2023-01-31Learn from External Memory, not just Weights: Large-Scale Retrieval for Reinforcement Learning
2023-01-17How ChatGPT works - From Transformers to Reinforcement Learning with Human Feedback (RLHF)
2023-01-09HyperTree Proof Search - Automated Theorem Proving with AlphaZero and Transformers!
2022-12-23CodinGame Fall Challenge 2022: A First Look (managed to get to Silver!)
2022-12-21Can ChatGPT solve CodinGame/Google Kickstart problems?
2022-12-19Reinforcement Learning Fast and Slow: Goal-Directed and Memory Retrieval Mechanism!
2022-12-12A New Framework of Memory for Learning (Part 1)
2022-11-14Hippocampal Replay for Learning (Full Length with Questions)
2022-11-14Hippocampal Replay for Learning (3 min summary)
2022-11-07AlphaTensor: Using Reinforcement Learning for Efficient Matrix Multiplication