The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

during training. Every year since, researchers have reproduced this in masked-dense simulation, then hit a wall when they want the actual memory savings. PyTorch's torch.sparse_csr isn't designed for training — the backward pass is unimplemented for most ops, and the ones that exist force dense intermediates, which defeats the point. The alternative has been to write your own CSR + SIMD kernels, a six-month detour from whatever you were actually trying to study. SparseLab is that detour, packaged. Reproduced numbers (M3 Pro, all in the repo's docs/demos/): - MLP on MNIST at 90% sparsity (10% of params live): 97.45% vs 98.06% dense — 0.61pp gap, 82% memory reduction. Sparse needed 1.8x more epochs to converge. - 10M-param transformer on Tiny Shakespeare, 70% sparse attention + 90% sparse FFN, 10k steps: inference memory 15.3 MB vs 41.0 MB (37% of dense), 0.055 nats validation loss gap. - Scaling check at 40M params, 1000 steps (same architecture family, 4x larger): inference memory 55.8 MB vs 150.7 MB dense — exactly 37% of dense again. The ratio held across the scale-up. Per-step slowdown narrowed from 4.6x to 4.1x as kernel time started dominating Python overhead. - The honest caveat: on CPU we are still 4.1-4.6x slower per step than dense torch.matmul. The dW kernel is most of a step and is unvectorized in v0.1. Memory is the win, not speed. Why CPU-first is the angle. A DGX H100 has 640 GB of GPU HBM across 8 cards and costs $200-400K up front. Ten Hetzner AX102 nodes at ~€104/month each give you 1.28 TB of DDR5 — 2x the trainable memory at a fraction of the capital cost, paid monthly. For independent researchers training in the 100M-1B param range, RAM is the binding constraint, not FLOPs. Real sparse storage turns "doesn't fit in HBM" into "fits in DDR5, trains slow, but trains." DDP for wallclock recovery is on the v0.2 roadmap. API. Install with "pip install sparselab" (wheels for macOS arm64, Linux x86_64, Linux aarch64). One-line swap from nn.Linear: import sparselab layer = sparselab.SparseLinear(1536, 384, sparsity=0.9) algo = sparselab.RigL(sparsity=0.9, drop_fraction=0.3, update_freq=100) layer.apply(algo) # mutates topology during training SparsityAlgorithm is modeled on Cerebras's SparsityAlgorithm API (https://training-api.cerebras.ai/en/latest/wsc/tutorials/spa...) and credited in the docstrings. v0.1 ships Static, SET, and RigL. Help wanted. The aim is for SparseLab to become solid scaffolding for sparse-from-scratch work. Four places a contributor can own something real: 1. A new DST algorithm as a PR — Sparse Momentum, Top-KAST, GraNet. SparsityAlgorithm is ~100 lines; a new algorithm is another ~100. 2. CPU perf — dW kernel NEON/AVX-512 vectorization + parallel scheduling is the highest-leverage contribution. The 40M scaling numbers quantify exactly why. 3. CUDA port of SpMM + rewrite kernels. v0.1 is CPU-only; the layout is GPU-friendly and a CUDA port is the third contributor track. 4. Push the scaling further. We validated the memory ratio at 40M. The 100M+ regime is open territory — if you have CPU cluster time, a GPT-2 small scale-up with a real convergence budget would be the first independent reproduction above author hardware.

(HDC) Current Transformers use "dense" embeddings—e.g., a vector of 4,096 floating-point numbers (like `[0.1, -0.5, 0.03, ...]`). Every number matters. To mimic the brain, we would switch to *Hyperdimensional Vectors* (e.g., 10,000+ dimensions), but make them *binary and sparse*. * **Holographic Representation:** In HDC, concepts (like "cat") are stored as massive randomized vectors of 1s and 0s. Information is distributed "holographically" across the entire vector. You can cut the vector in half, and it still retains the information (just noisier), similar to how brain lesions don't always destroy specific memories. * **Math without Multiplication:** In this high-dimensional binary space, you don't need expensive floating-point matrix multiplication. You can use simple bitwise operations: * **Binding (Association):** XOR operations (`A ⊕ B`). * **Bundling (Superposition):** Majority rule (voting). * **Permutation:** Bit shifting. * **Simulation Benefit:** This allows a Transformer to manipulate massive "context windows" using extremely cheap binary logic gates instead of energy-hungry floating-point multipliers. ### 2\. The Architecture: "Spiking" Attention Mechanisms Standard Attention is $O(N^2)$ because it forces every token to query every other token. A "Spiking Transformer" simulates the brain's "event-driven" nature. * **Dynamic Sparsity:** Instead of a dense matrix multiplication, neurons would only "fire" (send a signal) if their activation crosses a threshold. If a token's relevance score is low, it sends *zero* spikes. The hardware performs *no* work for that connection. * **The "Winner-Take-All" Circuit:** In the brain, inhibitory neurons suppress weak signals so only the strongest "win." A simulated Sparse Transformer would replace the Softmax function (which technically keeps all values non-zero) with a **k-Winner-Take-All** function. * *Result:* The attention matrix becomes 99% empty (sparse). The system only processes the top 1% of relevant connections, similar to how you ignore the feeling of your socks until you think about them. ### 3\. The Hardware: Neuromorphic Substrate Even if you write sparse code, a standard GPU (NVIDIA H100) is bad at running it. GPUs like dense, predictable blocks of numbers. To simulate the brain efficiently, we need *Neuromorphic Hardware* (like Intel Loihi or IBM NorthPole). * **Address Event Representation (AER):** Instead of a "clock" ticking every nanosecond forcing all neurons to update, the hardware is asynchronous. It sits idle (consuming nanowatts) until a "spike" packet arrives at a specific address. * **Processing-in-Memory (PIM):** To handle the high dimensionality (e.g., 100,000-dimensional vectors), the hardware moves the logic gates *inside* the RAM arrays. This eliminates the energy cost of moving those massive vectors back and forth. ### Summary: The Hypothetical "Spiking HD-Transformer" | Feature | Standard Transformer | Simulated "Brain-Like" Transformer | | :--- | :--- | :--- | | *Dimension* | Low (\~4k), Dense, Float32 | *Ultra-High* (\~100k), Sparse, Binary | | *Operation* | Matrix Multiplication (MACs) | *Bitwise XOR / Popcount* | | *Attention* | Global Softmax ($N^2$) | *Spiking k-Winner-Take-All* (Linear) | | *Activation* | Continuous (RELU/GELU) | *Discrete Spikes* (Fire-or-Silence) | | *Hardware* | GPU (Synchronous) | *Neuromorphic* (Asynchronous) |

the model that performs either the fastest or the cheapest is going to win. For both of these cases one of the easiest ways to accomplish the goal is to use a smaller model. Specialized models are just about always smaller. Lower latency on requests and less energy usage per query are both big wins that affect the economics of the system. There are other benefits as well. It's much easier to experiment and compare specialized models, and there's less area for errors to leak in. So even if it's possible to get a model like GPT-4 to work as well as a specialized model, if you're actually putting something into production and you have the data it almost always makes sense to consider a specialized model.

proprietary: In this case, we likely have a reviewer that is unqualified to even be reviewing. 2. The reviewers know: In this case, there's an ethics violation. == Why this is a problem == There's several issues here: bias, evaluation, reproduction, ethics/scientific goals. Bias: Most of these models and datasets are well known within the community and thus we know which labs they are from. We're biased to believe that works from big labs are higher quality and more accurate. This may be true to some extent, but your purpose as a reviewer is to judge the merit of the work. We try to do so in as much isolation as possible to prevent ourselves from being influenced and just focus on the merits[note0]. For example, any work with JFT is going to be known as a work from Google. Evaluation: We'll use this work[2] as an example (it uses JFT and a lot of TPU compute: i.e. Google). Since the work only uses JFT we don't know if this scaling only works for JFT trained models, especially since it depends on the pretraining not overfitting (I'm unconvinced tbh)[note1]. At least their table compares mostly to other JFT models, but it is mostly meaningless because JFT changes size frequently and we can't assume consistent data. All we know is that this result is valid for JFT pre-trained models. Their claims are much stronger than this. Reproduction: This is rather obvious. But we also have good reason to believe that these models change as well as the datasets. So even if there was a leak we have issues with works quickly losing validity as the next work is using different settings than the previous and intimate knowledge would be required to know if this affects results or not. NLP has this issue with OpenAI changes. It is very noisy research doing things like this. Ethics and Scientific goals: Our job is to advance human knowledge. This can be done in a noisy process, but we do have a responsibility to make this as clear as possible. Our job is NOT to be ads for big companies, but closed work does so. In fact, this is a big reason big companies publish. They are great publicity. This is fine, as long as they provide utility to the community. We have other ethical issues[note2] == Proposed Solution == Submissions must utilize open models and datasets. The work is evaluated based on that. After acceptance the works can add. This way there is no deanonymizing information, we ensure reproducibility, but we also are able to gleam the potential results from the proprietary efforts. This allows for the big labs to get their publicity while ensuring that anonymity is not broken and we're not biased to big labs. This doesn't solve the arxiv issue and the dumb social media policies[note4] but it is a a simple step we can implement right away. It is also a step that YOU the reviewer, a single person, can take action against. We need to take a serious look at ourselves and ask what is the point of our science. Is it to make money or to progress human knowledge? You can do both, but what is the main goal. I'd argue that our goal is knowledge. It is fine to make profits off this knowledge, but I'm afraid we've incentivized the profits too much and our system is not aligned with the original goals. Goodhart's Law is a bitch. == Notes/Bib == [note0] There are some biases we should concern ourselves with, such as compute power, but this requires more nuanced understanding. A work that can spend months tuning hyperparameters will always have better benchmarks than works that spend a month of compute for their entire work. These can't be evaluated on benchmarks alone and doing so will cause us to miss extremely important works (see Lottery Ticket Hypothesis[1]). Reviewing should be hard and require nuance, so there should be certain roadblocks to make it difficult to be lazy. [note1] They do near de-dupe but we can't know how effective the techniques are so we can't know how valid these claims are. [note2] Conferences are a bad way to do science. Conferences are filters, journals are QC. ML/CS uses conferences. This is a zero sum environment where your competitors are given the opportunity to filter your work. You don't need active collusion to form bad actors. It pits the people evaluating the work against the work. A journal system instead has reviewers focusing on adding value to the work. This is because journaling has rounds and discussions. Conferencing is nearly zero shot with a single chance for rebuttal, often limiting to a single page to respond to 4 reviewers with vastly different concerns. We have other issues like reviewer quality (remember ACs oversee reviewers[3]), qualifications, collusion[4,5], fraud, and more. But these are for a longer discussion. [note3] Social media rules prevent authors from evangelizing their works in public, such as on Twitter. It does not prevent doing so in private or others from doing so for you. This not only leads to (actively encourages) collusion rings but worsens the problem. Big labs will always get extra publicity. People watch for their works, they have more followers, and so on. Small labs' only chance to get heard is to actively evangelize their work, and we just cut off their tools while not affecting big labs. A scratch to the giant is the loss of a limb to an ant. We shouldn't need evangelization but networks exist so it is necessary. I don't think we should stop pre-prints (they're too valuable), but we need to recognize the problems that they result in. We also shouldn't have people at the top of the ivory tower making decisions about how the handle the plights of those at the bottom without actively working with those at the bottom. [0] (Notice Carmen's review) https://openreview.net/forum?id=OpzV3lp3IMC [1] https://arxiv.org/abs/1803.03635 [2] https://arxiv.org/abs/2305.13035 [3] https://qipeng.me/blog/what-does-an-area-chair-do/ [4] https://www.reddit.com/r/MachineLearning/comments/nxbcl9/d_c... [5] https://analyticsindiamag.com/whats-wrong-with-the-paper-rev...

the case that when you have more parameters to optimize, you have more degrees of freedom and you are less likely to end up getting stuck in a local minima which is why it's actually easier to train a larger network to solve a task versus a smaller network, despite the fact that both are wildly overparameterized. [1] https://arxiv.org/abs/1803.03635

naturally uncovers subnetworks whose initializations made them capable of training effectively. Based on these results, we articulate the "lottery ticket hypothesis:" dense, randomly-initialized, feed-forward networks contain subnetworks ("winning tickets") that - when trained in isolation - reach test accuracy comparable to the original network in a similar number of iterations. The winning tickets we find have won the initialization lottery: their connections have initial weights that make training particularly effective. We present an algorithm to identify winning tickets and a series of experiments that support the lottery ticket hypothesis and the importance of these fortuitous initializations. We consistently find winning tickets that are less than 10-20% of the size of several fully-connected and convolutional feed-forward architectures for MNIST and CIFAR10. Above this size, the winning tickets that we find learn faster than the original network and reach higher test accuracy. Are these innate patterns of activity priming mammalian brains in a similar way?

translational invariance and hence it being a naturally good performer on image inputs/tasks. I think the parallel to "wildlife that is capable of certain behaviors ... before there's been a chance for significant learning to take place" is that priors are also part of biological intelligence. I.e. brain structure at birth enabling certain surivival oriented behaviors. Hence, I'm optimistic about transfer learning which could happen through _both_ better models (priors that generalize well) and pretrained weights (possibly partially pretrained, i.e. just initial feature extraction). Either could potentially provide a better starting point from the 'how many samples are necessary for good performance on a variety of tasks' perspective. The point is that either way information needs to be added for performance on tasks to increase. Doing that in a task specific way by using today's algorithms and a billion samples doesn't seem like the right approach. Finding algorithms, models or specifically perhaps neural network architectures (including training procedures, regularizers, loss function, weight tying) that generalize across tasks without needing many samples due to their informative priors seems the way forward to me. That's _not_ a naive scaling of today's algorithms to larger and larger training sets. Which was the point I was trying to make.

regularization/sparsification throughout training. The question is how best to find these lottery tickets, and encourage them from the get-go. A lot of this work is also related to training adversarially robust networks. A composition of ReLU layers corresponds to a piecewise linear function, where the # of 'pieces' is like exponential in the # of neurons. It's well known that standard training of networks results in a highly non-linear pwl that is easily fooled by adversarial examples. The robustness of a neural network against adversarial examples is typically characterized by its smoothness. One question is how to train the network or prune neurons to encourage smoothness & reduce the complexity (i.e. # of linear pieces) of the nn.