Show HN: Version code, models, & datasets together in GitHub https://ift.tt/MTgDwtc

Show HN: Version code, models, & datasets together in GitHub Hi HN! We just launched a GitHub integration that scales your Git repos to handle 100 terabytes of files in a single repo. XetData enables data scientists and machine learning engineers to version code, models, and datasets together. Most teams have glued together clunky workflows using S3, DVC, Git, Git LFS, and other tools and make true reproducibility difficult: https://ift.tt/qB16nt5 We instead embrace and extend Git so end-users don’t need to learn a new tool and a new set of commands. Our implementation is similar to Git LFS, where we take over the .gitattributes file, push pointers to large files in GitHub, and push the raw, large files to us. We have a few distinct features that we’re proud of that improve the user experience: - Our XetData bot comments on your pull requests to provide links to useful dataset views and model diffs. We’re working on rendering these inside GitHub itself using browser extensions. - Git LFS and similar tools only implement file-level deduplication. We created a new technique called block-based deduplication (published in CIDR’23 conference) specifically for data and ML workflows. The ML lifecycle consists of making lots of iterative changes and our technique helps save storage and time spent downloading and uploading changes. - You can mount large repos to your local machine using git-xet mount for exploratory work. Individual files that are needed are streamed in just in time behind the scenes. We open sourced our implementation of mount and it was well received here on HN: https://ift.tt/4meHXgd - To give more users access to your data, just add them to your GitHub repo. This is a beta product and we would love all of your feedback. You can find all instructions to try this out here: https://ift.tt/qjn0otl While we’re in beta, our product is completely free to use. We have a Slack you can join or a GitHub issue tracker. - Slack: https://ift.tt/ZD5iubk - GitHub: https://ift.tt/01ozqsI November 17, 2023 at 01:26AM

Komentar

Posting Komentar

Postingan populer dari blog ini

Show HN: Interactive exercises for GNU grep, sed and awk https://ift.tt/OxeFwah

Show HN: Interactive exercises for GNU grep, sed and awk Hello! For the past few months, I've been using a Python framework called Textual to create TUI apps for interactive exercises. Released the app for GNU awk earlier today, so thought I'd create a post here. If you already know how to manage Python packages, you can use the following command to get all the three apps: pip install grepexercises sedexercises awkexercises `pipx` should also work, but I haven't tested it. The GitHub repo has the source code as well as more detailed installation instructions. You can use alternative CLI tools to solve these exercises as well. For example, Perl instead of GNU awk or ripgrep instead of GNU grep and so on. Hope you find these TUI apps useful. I'd highly appreciate your feedback. Happy learning :) https://ift.tt/Hp9kFEi August 17, 2023 at 05:13PM
Baca selengkapnya

Show HN: Create demos & guides just with a simple prompt https://ift.tt/HfWo3mz

Show HN: Create demos & guides just with a simple prompt Teams often create demos & guides on a daily basis. But creating them is time consuming and needs a lot of effort. From content to the branding, customizations, and many more - they are not something you would want to do on a daily basis. Enter Fable. Purpose built for teams in marketing, sales & customer success. Fable is apt for engaging visitors, closing deals faster, and onboarding your users quickly. https://ift.tt/IVCbkiG October 17, 2024 at 08:32PM
Baca selengkapnya