# **ML Papers of The Week** We ❤️ reading ML papers so we have created this repo to highlight the top ML papers for every week. ## Top ML Papers of the Week (Jan 9-15) | **Paper** | **Links** | | ------------- | ------------- | | 1) **Mastering Diverse Domains through World Models** -- DreamerV3 is a general algorithm to collect diamonds in Minecraft from scratch without human data or curricula, a long-standing challenge in AI. | [Paper](https://arxiv.org/abs/2301.04104v1), [Tweet](https://twitter.com/dair_ai/status/1614676677757661185?s=20&t=3GITA7PeX7pGwrqvt97bYQ)| | 2) **Tracr: Compiled Transformers as a Laboratory for Interpretability** -- DeepMind proposes Tracr, a compiler for converting RASP programs into transformer weights. This way of constructing NNs weights enables the development and evaluation of new interpretability tools. | [Paper](https://arxiv.org/abs/2301.05062), [Tweet](https://twitter.com/dair_ai/status/1614676680165187584?s=20&t=3GITA7PeX7pGwrqvt97bYQ), [Code](https://github.com/deepmind/tracr) | | 3) **Multimodal Deep Learning** -- Multimodal deep learning is a new book published on ArXiv. | [Book](https://arxiv.org/abs/2301.04856), [Tweet](https://twitter.com/dair_ai/status/1614676682555670528?s=20&t=3GITA7PeX7pGwrqvt97bYQ) | | 4) **Forecasting Potential Misuses of Language Models for Disinformation Campaigns—and How to Reduce Risk** -- OpenAI publishes new work analyzing how generative LMs could potentially be misused for disinformation and how to mitigate these types of risks. | [Paper](https://openai.com/blog/forecasting-misuse/), [Tweet](https://twitter.com/dair_ai/status/1614676684984156160?s=20&t=3GITA7PeX7pGwrqvt97bYQ) | | 5) **Why do Nearest Neighbor Language Models Work?** -- Empirically identifies reasons why retrieval-augmented LMs (specifically k-nearest neighbor LMs) perform better than standard parametric LMs. | [Paper](https://arxiv.org/abs/2301.02828), [Code](https://github.com/frankxu2004/knnlm-why), [Tweet](https://twitter.com/dair_ai/status/1614676687597469696?s=20&t=3GITA7PeX7pGwrqvt97bYQ) | | 6) **Memory Augmented Large Language Models are Computationally Universal** -- Investigates the use of existing LMs (e.g, Flan-U-PaLM 540B) combined with associative read-write memory to simulate the execution of a universal Turing machine. | [Paper](https://arxiv.org/abs/2301.04589) , [Tweet](https://twitter.com/dair_ai/status/1614676689908277252?s=20&t=3GITA7PeX7pGwrqvt97bYQ) | | 7) **A Survey on Transformers in Reinforcement Learning** -- Transformers for RL will be a fascinating research area to track. The same is true for the reverse direction (RL for Transformers)... a notable example: using RLHF to improve LLMs (e.g., ChatGPT). | [Paper](https://arxiv.org/abs/2301.03044), [Tweet](https://twitter.com/dair_ai/status/1614676692538105860?s=20&t=3GITA7PeX7pGwrqvt97bYQ) | | 8) **Scaling Laws for Generative Mixed-Modal Language Models** -- Introduces scaling laws for generative mixed-modal language models. | [Paper](https://arxiv.org/abs/2301.03728), [Tweet](https://twitter.com/dair_ai/status/1614676694920531969?s=20&t=3GITA7PeX7pGwrqvt97bYQ) | | 9) **DeepMatcher: A Deep Transformer-based Network for Robust and Accurate Local Feature Matching** -- DeepMatcher is a transformer-based network showing robust local feature matching, outperforming the state-of-the-art methods on several benchmarks. | [Paper](https://arxiv.org/abs/2301.02993), [Tweet](https://twitter.com/dair_ai/status/1614676697516752898?s=20&t=3GITA7PeX7pGwrqvt97bYQ) | | 10) **Generative Time Series Forecasting with Diffusion, Denoise, and Disentanglement** -- This work addresses the time series forecasting problem with generative modeling; involves a bidirectional VAE backbone equipped with diffusion, denoising for prediction accuracy, and disentanglement for model interpretability. | [Paper](https://arxiv.org/abs/2301.03028), [Tweet](https://twitter.com/dair_ai/status/1614676699915980804?s=20&t=3GITA7PeX7pGwrqvt97bYQ) | --- [Subscribe to our newsletter](https://nlpnews.substack.com/) to stay on top of ML research and trends. We use a combination of AI-powered tools, analytics, and human curation to build the lists of papers. ## Top ML Papers of the Week (Jan 1-8) ![My Image](pics/Week-1.png) ## Top ML Papers of the Week (Jan 1-8) | **Paper** | **Links** | | ------------- | ------------- | | 1) **Muse: Text-To-Image Generation via Masked Generative Transformers** -- GoogleAI introduces Muse, a new text-to-image generation model based on masked generative transformers; significantly more efficient than other diffusion models like Imagen and DALLE-2. | [Paper](https://arxiv.org/abs/2301.00704), [Project](https://muse-model.github.io/), [Code](https://github.com/lucidrains/muse-maskgit-pytorch), [Tweet](https://twitter.com/dair_ai/status/1612153095772938241?s=20&t=ChwZWzSmoRlZKnD54fsV6w)| | 2) **VALL-E Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers** -- Microsoft introduces VALL-E, a text-to-audio model that performs state-of-the-art zero-shot performance; the text-to-speech synthesis task is treated as a conditional language modeling task. | [Project](https://valle-demo.github.io/), [Tweet](https://twitter.com/dair_ai/status/1612153097962328067?s=20&t=ChwZWzSmoRlZKnD54fsV6w) | | 3) **Rethinking with Retrieval: Faithful Large Language Model Inference** -- A new paper shows the potential of enhancing LLMs by retrieving relevant external knowledge based on decomposed reasoning steps obtained through chain-of-thought prompting. | [Paper](https://arxiv.org/abs/2301.00303), [Tweet](https://twitter.com/dair_ai/status/1612153100114055171?s=20&t=ChwZWzSmoRlZKnD54fsV6w) | | 4) **SparseGPT: Massive Language Models Can Be Accurately Pruned In One-Shot** -- Presents a technique for compressing large language models while not sacrificing performance; "pruned to at least 50% sparsity in one-shot, without any retraining." | [Paper](https://arxiv.org/abs/2301.00774), [Tweet](https://twitter.com/dair_ai/status/1612153102513360901?s=20&t=ChwZWzSmoRlZKnD54fsV6w) | | 5) **ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders** -- ConvNeXt V2 is a performant model based on a fully convolutional masked autoencoder framework and other architectural improvements. CNNs are sticking back! | [Paper](https://arxiv.org/abs/2301.00808), [Code](https://github.com/facebookresearch/convnext-v2), [Tweet](https://twitter.com/dair_ai/status/1612153104329281538?s=20&t=ChwZWzSmoRlZKnD54fsV6w) | | 6) **Large Language Models as Corporate Lobbyists** -- With more capabilities, we are starting to see a wider range of applications with LLMs. This paper utilized large language models for conducting corporate lobbying activities. | [Paper](https://arxiv.org/abs/2301.01181) , [Code](https://github.com/JohnNay/llm-lobbyist), [Tweet](https://twitter.com/dair_ai/status/1612153106355130372?s=20&t=ChwZWzSmoRlZKnD54fsV6w) | | 7) **Superposition, Memorization, and Double Descent** -- This work aims to better understand how deep learning models overfit or memorize examples; interesting phenomena observed; important work toward a mechanistic theory of memorization. | [Paper](https://transformer-circuits.pub/2023/toy-double-descent/index.html), [Tweet](https://twitter.com/dair_ai/status/1612153108460892160?s=20&t=ChwZWzSmoRlZKnD54fsV6w) | | 8) **StitchNet: Composing Neural Networks from Pre-Trained Fragments** -- StitchNet: Interesting idea to create new coherent neural networks by reusing pretrained fragments of existing NNs. Not straightforward but there is potential in terms of efficiently reusing learned knowledge in pre-trained networks for complex tasks. | [Paper](https://arxiv.org/abs/2301.01947), [Tweet](https://twitter.com/dair_ai/status/1612153110452903936?s=20&t=ChwZWzSmoRlZKnD54fsV6w) | | 9) **Iterated Decomposition: Improving Science Q&A by Supervising Reasoning Processes** -- Proposes integrated decomposition, an approach to improve Science Q&A through a human-in-the-loop workflow for refining compositional LM programs. | [Paper](https://arxiv.org/abs/2301.01751), [Code](https://github.com/oughtinc/ice) [Tweet](https://twitter.com/dair_ai/status/1612153112638402562?s=20&t=ChwZWzSmoRlZKnD54fsV6w) | | 10) **A Succinct Summary of Reinforcement Learning** -- A nice little overview of some important ideas in RL. | [Paper](https://arxiv.org/abs/2301.01379), [Tweet](https://twitter.com/dair_ai/status/1612153114773053446?s=20&t=ChwZWzSmoRlZKnD54fsV6w) | --- [Subscribe to our newsletter](https://nlpnews.substack.com/) to stay on top of ML research and trends. We use a combination of AI-powered tools, analytics, and human curation to build the lists of papers.