[Classic] Word2Vec: Distributed Representations of Words and Phrases and their Compositionality

Channel:
Switzerland Yannic Kilcher
Subscribers:
284,000
Published on ● Video Link: https://www.youtube.com/watch?v=yexR53My2O4


Duration: 31:22
18,987 views
871

#ai #research #word2vec

Word vectors have been one of the most influential techniques in modern NLP to date. This paper describes Word2Vec, which the most popular technique to obtain word vectors. The paper introduces the negative sampling technique as an approximation to noise contrastive estimation and shows that this allows the training of word vectors from giant corpora on a single machine in a very short time.

OUTLINE:
0:00 - Intro & Outline
1:50 - Distributed Word Representations
5:40 - Skip-Gram Model
12:00 - Hierarchical Softmax
14:55 - Negative Sampling
22:30 - Mysterious 3/4 Power
25:50 - Frequent Words Subsampling
28:15 - Empirical Results
29:45 - Conclusion & Comments

Paper: https://arxiv.org/abs/1310.4546
Code: https://code.google.com/archive/p/word2vec/

Abstract:
The recently introduced continuous Skip-gram model is an efficient method for learning high-quality distributed vector representations that capture a large number of precise syntactic and semantic word relationships. In this paper we present several extensions that improve both the quality of the vectors and the training speed. By subsampling of the frequent words we obtain significant speedup and also learn more regular word representations. We also describe a simple alternative to the hierarchical softmax called negative sampling. An inherent limitation of word representations is their indifference to word order and their inability to represent idiomatic phrases. For example, the meanings of "Canada" and "Air" cannot be easily combined to obtain "Air Canada". Motivated by this example, we present a simple method for finding phrases in text, and show that learning good vector representations for millions of phrases is possible.

Authors: Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, Jeffrey Dean


Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n



Other Videos By Yannic Kilcher


2020-08-12Meta-Learning through Hebbian Plasticity in Random Networks (Paper Explained)
2020-08-09Hopfield Networks is All You Need (Paper Explained)
2020-08-06I TRAINED AN AI TO SOLVE 2+2 (w/ Live Coding)
2020-08-04PCGRL: Procedural Content Generation via Reinforcement Learning (Paper Explained)
2020-08-02Big Bird: Transformers for Longer Sequences (Paper Explained)
2020-07-29Self-training with Noisy Student improves ImageNet classification (Paper Explained)
2020-07-26[Classic] Playing Atari with Deep Reinforcement Learning (Paper Explained)
2020-07-23[Classic] ImageNet Classification with Deep Convolutional Neural Networks (Paper Explained)
2020-07-21Neural Architecture Search without Training (Paper Explained)
2020-07-19[Classic] Generative Adversarial Networks (Paper Explained)
2020-07-16[Classic] Word2Vec: Distributed Representations of Words and Phrases and their Compositionality
2020-07-14[Classic] Deep Residual Learning for Image Recognition (Paper Explained)
2020-07-12I'M TAKING A BREAK... (Channel Update July 2020)
2020-07-11Deep Ensembles: A Loss Landscape Perspective (Paper Explained)
2020-07-10Gradient Origin Networks (Paper Explained w/ Live Coding)
2020-07-09NVAE: A Deep Hierarchical Variational Autoencoder (Paper Explained)
2020-07-08Addendum for Supermasks in Superposition: A Closer Look (Paper Explained)
2020-07-07SupSup: Supermasks in Superposition (Paper Explained)
2020-07-06[Live Machine Learning Research] Plain Self-Ensembles (I actually DISCOVER SOMETHING) - Part 1
2020-07-05SpineNet: Learning Scale-Permuted Backbone for Recognition and Localization (Paper Explained)
2020-07-04Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention (Paper Explained)


Tags:
deep learning
machine learning
arxiv
explained
neural networks
ai
artificial intelligence
paper
jeff dean
mikolov
word2vec
word vectors
word representations
nlp
natural language processing
sentiment classification
king
queen
man
woman
arithmetic
latent space
distributed
country
capital
semantic
synonyms
skip gram
negative sampling
nce
noise contrastive estimation