Transformer XL | AISC Trending Papers

Channel:
Canada LLMs Explained - Aggregate Intellect - AI.SCIENCE
Subscribers:
22,300
Published on ● Video Link: https://www.youtube.com/watch?v=cXZ9YBqH3m0


Duration: 1:35:02
5,857 views
80

A.I. Socratic Circles - Fast Track Stream
https://aisc.a-i.science/events/2019-02-21

Discussion Lead: Florian Goebels
Facilitators: Taraneh Khazaei, Ehsan Amjadian

TRANSFORMER-XL: ATTENTIVE LANGUAGE MODELS BEYOND A FIXED-LENGTH CONTEXT

"Transformer networks have a potential of learning longer-term dependency, but are limited by a fixed-length context in the setting of language modeling. As a solution, we propose a novel neural architecture, Transformer-XL, that enables Transformer to learn dependency beyond a fixed length without disrupting temporal
coherence. Concretely, it consists of a segment-level recurrence mechanism and a novel positional encoding scheme. Our method not only enables capturing longer-term dependency, but also resolves the problem of context fragmentation. As a result, Transformer-XL learns dependency that is about 80% longer than
RNNs and 450% longer than vanilla Transformers, achieves better performance on both short and long sequences, and is up to 1,800+ times faster than vanilla Transformer during evaluation. Additionally, we improve the state-of-the-art (SoTA) results of bpc/perplexity from 1.06 to 0.99 on enwiki8, from 1.13 to 1.08 on text8, from 20.5 to 18.3 on WikiText-103, from 23.7 to 21.8 on One Billion Word, and from 55.3 to 54.5 on Penn Treebank (without finetuning). Our code, pretrained models, and hyperparameters are available in both Tensorflow and PyTorch"



Other Videos By LLMs Explained - Aggregate Intellect - AI.SCIENCE


2019-03-28[Variational Autoencoder] Auto-Encoding Variational Bayes | AISC Foundational
2019-03-25[GQN] Neural Scene Representation and Rendering | AISC
2019-03-21Towards Interpretable Deep Neural Networks by Leveraging Adversarial Examples | AISC
2019-03-18Understanding the Origins of Bias in Word Embeddings
2019-03-14[Original Style Transfer] A Neural Algorithm of Artistic Style | TDLS Foundational
2019-03-11[RecSys 2018 Challenge winner] Two-stage Model for Automatic Playlist Continuation at Scale |TDLS
2019-03-07[OpenAI GPT2] Language Models are Unsupervised Multitask Learners | TDLS Trending Paper
2019-03-04You May Not Need Attention | TDLS Code Review
2019-02-28[DDQN] Deep Reinforcement Learning with Double Q-learning | TDLS Foundational
2019-02-25[AlphaGo Zero] Mastering the game of Go without human knowledge | TDLS
2019-02-21Transformer XL | AISC Trending Papers
2019-02-19Computational prediction of diagnosis & feature selection on mesothelioma patient records | AISC
2019-02-18Support Vector Machine (original paper) | AISC Foundational
2019-02-11Tensor Field Networks | AISC
2019-02-07ACAI: Understanding and Improving Interpolation in Autoencoders via an Adversarial Regularizer
2019-02-04Code Review: Transformer - Attention Is All You Need | AISC
2019-02-04[StyleGAN] A Style-Based Generator Architecture for GANs, part2 (results and discussion) | TDLS
2019-02-04[StyleGAN] A Style-Based Generator Architecture for GANs, part 1 (algorithm review) | TDLS
2019-02-04TDLS: Learning Functional Causal Models with GANs - part 1 (algorithm review)
2019-02-04TDLS: Learning Functional Causal Models with GANs - part 2 (results and discussion)
2019-02-04Neural Ordinary Differential Equations - part 1 (algorithm review) | AISC


Tags:
machine learning
transformers
attention
natural language processing
transformer xl
transformer network
attention mechanism