文件夹: llm/2020

2020年12月01日

CPM — A Large-scale Generative Chinese Pre-trained Language Model

2020年12月01日

The Pile — An 800GB Dataset of Diverse Text for Language Modeling

2020年10月01日

mT5 — A massively multilingual pre-trained text-to-text transformer

2020年10月01日

Scaling Laws for Autoregressive Generative Modeling

2020年10月01日

An Image is Worth 16x16 Words — Transformers for Image Recognition at Scale (ViT)

2020年9月01日

Generative Language Modeling for Automated Theorem Proving (GPT-f)

2020年9月01日

Learning to summarize from human feedback

2020年9月01日

Rethinking Attention with Performers

2020年7月01日

Big Bird — Transformers for Longer Sequences

2020年6月01日

DeBERTa — Decoding-enhanced BERT with Disentangled Attention

2020年6月01日

GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

2020年6月01日

Image GPT — Generative Pretraining from Pixels

2020年6月01日

Transformers are RNNs — Fast Autoregressive Transformers with Linear Attention

2020年6月01日

Linformer — Self-Attention with Linear Complexity

2020年6月01日

OpenAI API（GPT-3 商业化 API 发布）

2020年6月01日

wav2vec 2.0 — A Framework for Self-Supervised Learning of Speech Representations

2020年5月01日

Conformer — Convolution-augmented Transformer for Speech Recognition

2020年5月01日

Language Models are Few-Shot Learners (GPT-3)

2020年5月01日

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (RAG)

2020年5月01日

Synthesizer — Rethinking Self-Attention in Transformer Models

2020年5月01日

ZeRO-2 & DeepSpeed — Shattering barriers of deep learning speed & scale

2020年4月01日

Recipes for building an open-domain chatbot (BlenderBot)

2020年4月01日

Dense Passage Retrieval for Open-Domain Question Answering (DPR)

2020年4月01日

Longformer — The Long-Document Transformer

2020年3月01日

ELECTRA — Pre-training Text Encoders as Discriminators Rather Than Generators

2020年2月01日

GLU Variants Improve Transformer

2020年2月01日

REALM — Retrieval-Augmented Language Model Pre-Training

2020年2月01日

Turing-NLG — A 17-billion-parameter language model by Microsoft

2020年2月01日

ZeRO & DeepSpeed — New system optimizations enable training models with over 100 billion parameters

2020年1月01日

Towards a Human-like Open-Domain Chatbot (Meena)

2020年1月01日

Reformer — The Efficient Transformer

2020年1月01日

AI Research 技术调研

探索

文件夹: llm/2020

CPM — A Large-scale Generative Chinese Pre-trained Language Model

The Pile — An 800GB Dataset of Diverse Text for Language Modeling

mT5 — A massively multilingual pre-trained text-to-text transformer

Scaling Laws for Autoregressive Generative Modeling

An Image is Worth 16x16 Words — Transformers for Image Recognition at Scale (ViT)

Generative Language Modeling for Automated Theorem Proving (GPT-f)

Learning to summarize from human feedback

Rethinking Attention with Performers

Big Bird — Transformers for Longer Sequences

DeBERTa — Decoding-enhanced BERT with Disentangled Attention

GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

Image GPT — Generative Pretraining from Pixels

Transformers are RNNs — Fast Autoregressive Transformers with Linear Attention

Linformer — Self-Attention with Linear Complexity

OpenAI API（GPT-3 商业化 API 发布）

wav2vec 2.0 — A Framework for Self-Supervised Learning of Speech Representations

Conformer — Convolution-augmented Transformer for Speech Recognition

Language Models are Few-Shot Learners (GPT-3)

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (RAG)

Synthesizer — Rethinking Self-Attention in Transformer Models

ZeRO-2 & DeepSpeed — Shattering barriers of deep learning speed & scale

Recipes for building an open-domain chatbot (BlenderBot)

Dense Passage Retrieval for Open-Domain Question Answering (DPR)

Longformer — The Long-Document Transformer

ELECTRA — Pre-training Text Encoders as Discriminators Rather Than Generators

GLU Variants Improve Transformer

REALM — Retrieval-Augmented Language Model Pre-Training

Turing-NLG — A 17-billion-parameter language model by Microsoft

ZeRO & DeepSpeed — New system optimizations enable training models with over 100 billion parameters

Towards a Human-like Open-Domain Chatbot (Meena)

Reformer — The Efficient Transformer

Scaling Laws for Neural Language Models