r/mlscaling

: "AlphaGeometry: An Olympiad-level AI system for geometry"

: "MambaByte: Token-free Selective State Space Model"

: "Zuckerberg: "...[W]e're building massive compute infrastructure to support our future roadmap, including 350k H100s by the end of this year -- and overall almost 600k H100s equivalents of compute if you include other GPUs""

: "Minsky on abandoning DL in 1952: "I decided either this was a bad idea or it'd take thousands/millions of neurons to make it work, & I couldn’t afford to try to build a machine like that.""

: ""Estimating efficiency improvements in LLM pre-training", Daan (how much do all the improvements to GPT-3-style LLM training stack up to? >400x?)"

: "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models"

: "🦅 Eagle 7B : Soaring past Transformers with 1 Trillion Tokens Across 100+ Languages"

: "Chess-GPT, 1000x smaller than GPT-4, plays 1500 ELO chess. We can visualize its internal board state, and it accurately estimates the ELO rating of the players in a game."

: "Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model"

: ""I am a Strange Dataset: Metalinguistic Tests for Language Models", Thrush et al 2024 (only GPT-4 beats chance; parameter-scaling)"

: "Two very interesting articles by Yuxi Liu on historical resistance to connectionism and scaling"

: ""When Might AI Outsmart Us? It Depends Who You Ask", TIME"

: ""Large Language Models Struggle to Learn Long-Tail Knowledge, Kandpal et al 2022 (BLOOM models show smooth log-scaling of memorization of long-tail knowledge & larger models more sample-efficient)"

: ""What's Hidden in a Randomly Weighted Neural Network?", Ramanujan et al 2019 (even random nets contain, with increasing probability in size, an accurate sub-net)"

: ""My AI Timelines Have Sped Up (Again) [since 2020]", Alex Irpan"

: ""The Effect of Sampling Temperature on Problem Solving in Large Language Models", Renze & Guven 2024 (Johns Hopkins) (changes in temperature in the range 0.0 to 1.0 do not have a statistically significant impact on LLM performance for problem-solving tasks)"

: ""TabLib: A Dataset Of 627 Million Tables With Context", Eggert et al 2023 (69TB + 0.87t tokens descriptions)"

: "Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data - ("We highlight the value of data scaling-up of massive, cheap, and diverse unlabeled images for MDE.")"

: ""AI capabilities can be significantly improved without expensive retraining" - survey and analysis of post-training enhancements"

: "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"

: ""Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training", Hubinger et al 2024 (larger models better at hiding backdoors from safety training)"

: ""MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts", Lu et al., 2023"

: "[N] Meta open-sourced a wav2vec2 model pre-trained on 4.5M hours (Barrault et al 2023)"

: ""Are Vision Transformers More Data Hungry Than Newborn [Chick] Visual Systems?", Pandey et al 2023"

: ""TD-MPC2: Scalable, Robust World Models for Continuous Control" - Scaling behavior again noted in model-based reinforcement learning"

: ""Learning human actions on computer applications" {rabbit} - ("We share the view that the scaling law continues to permeate all aspects of neural systems research .... We hope to continue this trend with our action model ....")"

: ""Neural Network on a Commodore 64", Walker 1987"

: "Scalable Pre-training of Large Autoregressive Image Models"

: ""Large-scale Deep Unsupervised Learning using Graphics Processors", Raina et al 2009"

: ""The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers", Li et al 2022 (MLP/Transformer increasingly sparse with scale)"

: "What's the largest existing LLM that an individual can feasibly run privately?"

: "Fastest implementation of Mixtral 8x7b-32k"

: ""China's military and government acquire [a very few] Nvidia chips despite US ban""

: ""Thousands of AI Authors on the Future of AI", Grace et al 2024 {AIImpacts}"

: "What do you think about Yann Lecun's controversial opinions about ML?"