Show HN: LLMs suck at writing integration code… for now https://ift.tt/YzV3SfJ

Show HN: LLMs suck at writing integration code… for now Hi HN! Stefan here from superglue and today I’d like to share a new benchmark we’ve just open sourced: an Agent-API Benchmark, in which we test how well LLMs handle APIs. We gave LLMs API documentation and asked them to write code that makes actual API calls. Things like "create a Stripe customer" or "send a Slack message". We're not testing if they can use SDKs; we're testing if they can write raw HTTP requests (with proper auth, headers, body formatting) that actually work when executed against real API endpoints and can extract relevant information from that response. tl:dr: LLMs suck at writing code to use APIs. We ran 630 integration tests across 21 common APIs (Stripe, Slack, GitHub, etc.) using 6 different LLMs. Here are our key findings: - Best general LLM: 68% success rate. That's 1 in 3 API calls failing, which most would agree isn’t viable in production - Our integration layer scored a 91% success rate, showing us that just throwing bigger/better LLMs at the problem won't solve it. - Only 6 out of 21 APIs worked 100% of the time, every other API had failures. - Anthropic’s models are significantly better at building API integrations than other providers. Here is the results chart: https://ift.tt/YaWhQlS What made LLMs fail: - Lack of context (LLMs are just not great at understanding what API endpoints exist and what they do, even if you give them documentation which we did) - Multi-step workflows (chaining API calls) - Complex API design: APIs like Square, PostHog, Asana (Forcing project selection among other things trips llms over) We've open-sourced the benchmark so you can test any API and see where it ranks: https://ift.tt/zTMO7qF... Check out the repo, consider giving it a star, or see the full ranking at https://ift.tt/A3CFl74 . If you're building agents that need reliable API access, we'd love to hear your approach, or you can try our integration layer at superglue.ai. Next up: benchmarking MCP. https://ift.tt/lfgjY15 July 24, 2025 at 10:28PM

Komentar

Posting Komentar

Postingan populer dari blog ini

Show HN: Guish – A GUI for constructing and executing Unix pipelines https://ift.tt/HrXz5ub

Show HN: Guish – A GUI for constructing and executing Unix pipelines guish is a dynamic data pipeline UI that allows users to visually construct and execute complex command-line pipelines. It provides an intuitive interface for chaining together various Unix commands and custom plugins, making it easier to build and understand data processing workflows. It also features optional LLM support for updating pipelines with a natural language interface. https://ift.tt/2lsA0r5 August 2, 2024 at 10:00PM
Baca selengkapnya

Twin Peaks for All: Survey Results

Gambar
Twin Peaks for All: Survey Results By Benjamin Barnett Last month, the SFMTA and San Francisco Recreation and Parks Department released a survey seeking public feedback regarding  five proposed roadway options for Twin Peaks Boulevard . San Franciscans responded to the survey, ranked support for each alternative and shared opinions about how the different options met their needs.  The Twin Peaks for All survey was open for almost three weeks and received more than 1,700 responses. This number reflects the high level of interest for this San Francisco park and its future access.   Survey Background Conditions on Twin Peaks Boulevard between Portola Drive and Barnett Avenue before COVID street closures.   The Twin Peaks Boulevard gates were closed early in the pandemic to reduce crowding at the lookout point parking lot. The gates were closed on the north, at Burnett Avenue, and the south, at Portola Drive. Since then, Twin Peaks Park use by hikers, joggers and peopl...
Baca selengkapnya

History in Motion: New Photos from the 1960s to 1980s Now Online

Gambar
History in Motion: New Photos from the 1960s to 1980s Now Online By Jeremy Menzies We’ve been hard at work in the SFMTA Photo Archive and recently uploaded two large collections of photos from the early 1960s to the 1980s to our website. These photos, both color and black and white, give a glimpse into the day-to-day operations of Muni and the streets of San Francisco. Here’s a selection of shots from this brand-new batch of photos, many of which have not been seen outside the Photo Archive in years.  This shot shows a typical evening rush hour scene from the mid-1970s taken at Market and Powell Streets.  The color photos from this era are all part of a collection that was most likely used for presentations, studies and as general “stock” type photos of San Francisco, streets and Muni facilities and operations. Subjects include project documentation, copies of maps and charts, street...
Baca selengkapnya