Show HN: Run 30B model in 4GB Active Memory https://ift.tt/DPTyaA5

Show HN: Run 30B model in 4GB Active Memory We have built fused operator kernels for structured contextual sparsity to avoid loading and computing activations with feed forward layer weights that eventually zero out by the activation. The result? We are seeing 5X faster MLP layer performance in transformers with 50% lesser memory consumption avoiding the sleeping nodes in every token prediction. For Llama 3.2, Feed forward layers accounted for 30% of total weights and forward pass computation resulting in 1.6-1.8x increase in throughput: Sparse LLaMA 3.2 3B vs LLaMA 3.2 3B (on HuggingFace Implementation): - Time to First Token (TTFT): 1.51× faster (1.209s → 0.803s) - Output Generation Speed: 1.79× faster (0.7 → 1.2 tokens/sec) - Total Throughput: 1.78× faster (0.7 → 1.3 tokens/sec) - Memory Usage: 26.4% reduction (6.125GB → 4.15GB) Find the operator kernels with differential weight caching open sourced at github/sparse_transformers. Lets get LLMs sprinting! https://ift.tt/rF7zt3W June 6, 2025 at 12:13AM

Komentar

Posting Komentar

Postingan populer dari blog ini

Show HN: Guish – A GUI for constructing and executing Unix pipelines https://ift.tt/HrXz5ub

Show HN: Guish – A GUI for constructing and executing Unix pipelines guish is a dynamic data pipeline UI that allows users to visually construct and execute complex command-line pipelines. It provides an intuitive interface for chaining together various Unix commands and custom plugins, making it easier to build and understand data processing workflows. It also features optional LLM support for updating pipelines with a natural language interface. https://ift.tt/2lsA0r5 August 2, 2024 at 10:00PM
Baca selengkapnya

Twin Peaks for All: Survey Results

Gambar
Twin Peaks for All: Survey Results By Benjamin Barnett Last month, the SFMTA and San Francisco Recreation and Parks Department released a survey seeking public feedback regarding  five proposed roadway options for Twin Peaks Boulevard . San Franciscans responded to the survey, ranked support for each alternative and shared opinions about how the different options met their needs.  The Twin Peaks for All survey was open for almost three weeks and received more than 1,700 responses. This number reflects the high level of interest for this San Francisco park and its future access.   Survey Background Conditions on Twin Peaks Boulevard between Portola Drive and Barnett Avenue before COVID street closures.   The Twin Peaks Boulevard gates were closed early in the pandemic to reduce crowding at the lookout point parking lot. The gates were closed on the north, at Burnett Avenue, and the south, at Portola Drive. Since then, Twin Peaks Park use by hikers, joggers and peopl...
Baca selengkapnya

Launch HN: Stacker (YC S20) – Create Apps from Airtable or Google Sheets https://ift.tt/3i3ZJso

Launch HN: Stacker (YC S20) – Create Apps from Airtable or Google Sheets We're Michael and Sam, co-founders of Stacker ( https://stacker.app/ ). We let anyone create custom software powered by data from Airtable or Google Sheets, with a nice UI, auth and rich permissions. Think Internal Tools, Custom CRMs, and Customer Portals. We've been working for ages on building something that lets non-technical people create software without code. We spent about 2 years building a really powerful and complicated drag-and-drop no-code app builder. It was really awesome, it could create social networks, SaaS, marketplaces – the works. The only problem was: nobody could use it unless they were already a developer! It turned out that even though you weren't technically writing any code… you were still actually programming, still thinking like a developer. Just with a really inefficient set of no-code tools. We (eventually!) realised that non-devs were already building systems anyway; but ...
Baca selengkapnya