Show HN: Phospho – Text Analytics for LLM Apps (Posthog for Prompts) https://ift.tt/cjsEBKO

Show HN: Phospho – Text Analytics for LLM Apps (Posthog for Prompts) Hello HN! Pierre and Paul here. We are building an open source text analytics tool for user inputs and LLM app outputs The repo is https://ift.tt/Lji76TO and landing is https://phospho.ai Most people building with LLMs today don’t have quantified evaluation and usage metrics on the interactions between users and their product. The only solution is to read every message (or a sample) to get a sense of what is going on. You can't improve your product without understanding who your users are and how they are using it. Nobody would launch a website without standard analytics today; the same principle should apply to LLM products. We made phospho to analyze the large amounts of text from user inputs and LLM app outputs, and give you quantified and actionable insights. You first log messages and set up semantic events. Eg: “user is talking about sports”, “assistant didn’t quote the source”. We then run asynchronous jobs to detect if events are present in the text or not. To do so, we use GPT-4 for the first few events, and then downsize to smaller fine-tuned models (cheaper & faster). It works with any LLM provider (OpenAI, Mistral, Ollama…). No proxy, no monkey patch, and no OpenAI key needed. You can link phospho to your users’ feedback, and even use the platform to annotate some messages yourself. This helps you design step by step a custom evaluation pipeline that runs automatically, fits your needs, and enables you to iterate. Results are available in dashboards, as dataframes, or via API. You can also directly leverage the events in your app to trigger actions in real-time with the API or via webhooks. Deploy everything with Docker, or use the hosted cloud version. We have Python/Javascript SDK and an API. License is Apache 2.0. Give it a spin and see where we’re at: https://ift.tt/Lji76TO We’re interested in both feature requests and roasts. Let us know what you think! https://ift.tt/Lji76TO March 13, 2024 at 10:14PM

Komentar

Posting Komentar

Postingan populer dari blog ini

Show HN: Interactive exercises for GNU grep, sed and awk https://ift.tt/OxeFwah

Show HN: Interactive exercises for GNU grep, sed and awk Hello! For the past few months, I've been using a Python framework called Textual to create TUI apps for interactive exercises. Released the app for GNU awk earlier today, so thought I'd create a post here. If you already know how to manage Python packages, you can use the following command to get all the three apps: pip install grepexercises sedexercises awkexercises `pipx` should also work, but I haven't tested it. The GitHub repo has the source code as well as more detailed installation instructions. You can use alternative CLI tools to solve these exercises as well. For example, Perl instead of GNU awk or ripgrep instead of GNU grep and so on. Hope you find these TUI apps useful. I'd highly appreciate your feedback. Happy learning :) https://ift.tt/Hp9kFEi August 17, 2023 at 05:13PM
Baca selengkapnya

Show HN: My Book Bulletproof TLS and PKI (Second Edition) Is Out https://ift.tt/5PZ9mxF

Show HN: My Book Bulletproof TLS and PKI (Second Edition) Is Out Hello HN. I am excited to share with you my new book, Bulletproof TLS and PKI. I've worked in this space since the very early days (think SSLv2), always frustrated with the fact that the field is vast but the documentation poor. That first led me to create SSL Labs (which ended up being very popular) and then the first edition of my book (in 2014), where I aimed to cover everything a curious person needed to know about SSL/TLS and PKI. Most importantly, it's a very practical book that you can use to just learn what you need at that moment. The second edition (just out) adds coverage of TLS 1.3. I publish two chapters as a separate (and free) OpenSSL Cookbook. There's another free sample chapter as well. The best part of Bulletproof TLS and PKI is that it's a living book. There's nothing worse than obsolete documentation! Because none of the traditional publishers were interested in that sort of thing,
Baca selengkapnya