Show HN: OnPair – String compression with fast random access (Rust, C++) https://ift.tt/1kuYJ2T

Show HN: OnPair – String compression with fast random access (Rust, C++) I’ve been working on a compression algorithm for fast random access to individual strings in large collections. The problem came up when working with large in-memory database columns (emails, URLs, product titles, etc.), where low-latency point queries are essential. With short strings, LZ77-based compressors don’t perform well. Block compression helps, but block size forces a trade-off between ratio and access speed. Some existing options: - BPE: good ratios, but slow and memory-heavy - FSST (discussed here: https://ift.tt/67MaOUk ): very fast, but weaker compression This solution provides an interesting balance (more details in the paper): - Compression ratio: similar to BPE - Compression speed: 100–200 MiB/s - Decompression speed: 6–7 GiB/s I’d love to hear your thoughts — whether it’s workloads you think this could help with, ideas for API improvements, or just general discussion. Always happy to chat here on HN or by email. --- Resources: - Paper: https://ift.tt/btiZufw - Rust: https://ift.tt/YjvNLgH - C++: https://ift.tt/hJEpqld https://ift.tt/YjvNLgH August 19, 2025 at 10:20PM

Komentar

Posting Komentar

Postingan populer dari blog ini

Show HN: Guish – A GUI for constructing and executing Unix pipelines https://ift.tt/HrXz5ub

Show HN: Guish – A GUI for constructing and executing Unix pipelines guish is a dynamic data pipeline UI that allows users to visually construct and execute complex command-line pipelines. It provides an intuitive interface for chaining together various Unix commands and custom plugins, making it easier to build and understand data processing workflows. It also features optional LLM support for updating pipelines with a natural language interface. https://ift.tt/2lsA0r5 August 2, 2024 at 10:00PM
Baca selengkapnya

Twin Peaks for All: Survey Results

Gambar
Twin Peaks for All: Survey Results By Benjamin Barnett Last month, the SFMTA and San Francisco Recreation and Parks Department released a survey seeking public feedback regarding  five proposed roadway options for Twin Peaks Boulevard . San Franciscans responded to the survey, ranked support for each alternative and shared opinions about how the different options met their needs.  The Twin Peaks for All survey was open for almost three weeks and received more than 1,700 responses. This number reflects the high level of interest for this San Francisco park and its future access.   Survey Background Conditions on Twin Peaks Boulevard between Portola Drive and Barnett Avenue before COVID street closures.   The Twin Peaks Boulevard gates were closed early in the pandemic to reduce crowding at the lookout point parking lot. The gates were closed on the north, at Burnett Avenue, and the south, at Portola Drive. Since then, Twin Peaks Park use by hikers, joggers and peopl...
Baca selengkapnya

History in Motion: New Photos from the 1960s to 1980s Now Online

Gambar
History in Motion: New Photos from the 1960s to 1980s Now Online By Jeremy Menzies We’ve been hard at work in the SFMTA Photo Archive and recently uploaded two large collections of photos from the early 1960s to the 1980s to our website. These photos, both color and black and white, give a glimpse into the day-to-day operations of Muni and the streets of San Francisco. Here’s a selection of shots from this brand-new batch of photos, many of which have not been seen outside the Photo Archive in years.  This shot shows a typical evening rush hour scene from the mid-1970s taken at Market and Powell Streets.  The color photos from this era are all part of a collection that was most likely used for presentations, studies and as general “stock” type photos of San Francisco, streets and Muni facilities and operations. Subjects include project documentation, copies of maps and charts, street...
Baca selengkapnya