Show HN: Open-source observability for LLM apps https://ift.tt/fgFMVKu

Show HN: Open-source observability for LLM apps Hi HN, Hugh and Vince here. LLMonitor helps you record, trace & search your LLM queries and chatbot conversations. You can also capture user feedback on your frontend and correlate it with backend LLM queries then use that to fine-tune your own models. The project started has an internal tool in our previous (failed) AI startup. We’re aware the LLM observability space is very crowded. Apart from being open-source, we differentiate with: - Model-agnostic and minimal lock-in (no MITM of requests). - High focus on DX and dashboard clarity. - Support for complex scenarios: e.g. a chatbot that invokes multiple agents & tools before giving an answer. - Targeting early-stage startups + indie developers. Two of our big focuses in the near future are: - Tests (evaluations) written in either code or English (with GPT-4) to know when queries go wrong. - OpenTelemetry support to expand traces to the rest of the stack. (Both of those have turned out to be much more complicated than anticipated.) One decision we made is to stick with Postgres for everything instead of going OLAP with something like Clickhouse. As we are writing & updating many rows concurrently for the traces, Postgres seems to be the best tool for that. It has been sufficient for our analytics and search needs at the moment. We hope that also makes it easier for people to self-host and contribute. The entire codebase is licensed under Apache 2.0 and we’re fully bootstrapped. Being open-source has already begun to foster a community, e.g. with people building integrations for tooling we wouldn't have been able to support otherwise. To be honest we're still quite embarrassed with the state of the product, but I guess we’d be doing something wrong if that wasn’t the case. We’d love to hear your feedback - thank you! Landing page: https://llmonitor.com Docs: https://ift.tt/C1N5hAO Repo: https://ift.tt/bLxHOcB https://ift.tt/bLxHOcB October 28, 2023 at 07:58PM

Komentar

Posting Komentar

Postingan populer dari blog ini

Show HN: Interactive exercises for GNU grep, sed and awk https://ift.tt/OxeFwah

Show HN: Interactive exercises for GNU grep, sed and awk Hello! For the past few months, I've been using a Python framework called Textual to create TUI apps for interactive exercises. Released the app for GNU awk earlier today, so thought I'd create a post here. If you already know how to manage Python packages, you can use the following command to get all the three apps: pip install grepexercises sedexercises awkexercises `pipx` should also work, but I haven't tested it. The GitHub repo has the source code as well as more detailed installation instructions. You can use alternative CLI tools to solve these exercises as well. For example, Perl instead of GNU awk or ripgrep instead of GNU grep and so on. Hope you find these TUI apps useful. I'd highly appreciate your feedback. Happy learning :) https://ift.tt/Hp9kFEi August 17, 2023 at 05:13PM
Baca selengkapnya

Show HN: Create demos & guides just with a simple prompt https://ift.tt/HfWo3mz

Show HN: Create demos & guides just with a simple prompt Teams often create demos & guides on a daily basis. But creating them is time consuming and needs a lot of effort. From content to the branding, customizations, and many more - they are not something you would want to do on a daily basis. Enter Fable. Purpose built for teams in marketing, sales & customer success. Fable is apt for engaging visitors, closing deals faster, and onboarding your users quickly. https://ift.tt/IVCbkiG October 17, 2024 at 08:32PM
Baca selengkapnya